SHIELD Integrations

Integration System Architecture

This page documents the integration system architecture for processing various expense and tachograph files in the SHIELD application.

API Integrations

TODO

File Import

This is for all integrations where type is email.

Overview

The file import system is responsible for:

• Accepting uploaded files through various channels (web UI on the tenant, web UI on central, email ingress from anyone that knows the correct tag)
• Validating and identifying file formats
• Processing files according to their specific integration requirements
• Converting file data into standardized formats
• Storing and tracking import progress

Key Components

1. Import Service Layer

ImportDocumentService - Central service that orchestrates the file import process:

• Handles file uploads from multiple sources
• Manages document validation and storage
• Coordinates job scheduling for processing and whether other jobs need to run before an import
• Tracks import progress and reports errors to a given email

2. Integration Architecture

Base Classes

BaseImportIntegration - Abstract base for all import integrations

• Provides common file identification logic
• Handles file validation using Laravel validation rules
• Manages error grouping and reporting
• Converts raw data to standardized formats

ExpenseIntegration - Extends BaseImportIntegration for expense-related imports

• Specialized for expense data processing
• Handles vehicle and driver mapping
• Implements product categorization

Other modules like insurance or tachograph that have email integrations currently extends BaseImportIntegration and have independant logic. A InsuranceIntegration or TachographIntegration could be made in the future to centralise logic.

Integration Pattern

Each email integrations follow this pattern:

class ExampleIntegration extends ExpenseIntegration / BaseImportIntegration implements ExpenseImportIntegration / ClaimsImportIntegration / TachographImportIntegration
{
    public function getAvailableImports(): array
    {
        return [
            'Import Name' => ImportClass::class
        ];
    }

    // Optional: File preprocessing
    public function mutateFileBeforeImport(string $file_path): string
    {
        // Modify file contents before processing
        return $file_path;
    }
}

3. Import Classes

IntegrationImport - Abstract base for file format handlers

All import classes extend this base which provides:

• Data mapping functionality
• Validation rule definitions
• Header requirements

The logic will fail if the import class does not extend this class. This allows a unified structure.

Example Import Class Structure

class ExampleImport extends IntegrationImport
{
    // This maps the file headers to something that can be converted to what we want for the integration.
    // The secondary mapping is completed in the integration file.
    public function map($row): array
    {
        return [
            'timestamp' => parseImportTimestamp($row['date']),
            'product' => $row['product_name'],
            'quantity' => $row['amount'],
            // ... other mappings
        ];
    }

    // This is used to validate that the correct file is being used for the correct import.
    // Mainly required for Holman which has 4 different files.
    public function requiredHeaders(): array
    {
        return ['date', 'product_name', 'amount'];
    }

    // The rules have been added recently. They run after the file is converted to an array and the headers are mapped but before the map function. This function is the main cause of reports being generated.
    public function rules(): array
    {
        return [
            '*.date' => ['required', new ImportDateRule],
            '*.product_name' => 'required|string',
            '*.amount' => 'required|numeric|min:0',
        ];
    }

    // Optional: only required if the heading is not on the first row
    public function headingRow(): int
    {
        return 6;
    }
}

4. File Processing Flow

1. File Upload

   • Files received via web UI or email ingress
   • Stored in integration-specific directories
   • Original filenames preserved in the database

2. Format Identification

   • The integration is determined from the web route or email tag
   • The correct import is identified by checking the required headers against file contents
   • This acts as the initial validation as we don’t need to process the entire file to see if it has the required data
   • Returns appropriate import class or throws ImportFileException which generated a report document

3. File Preprocessing (Optional)

   • Some integrations modify files before processing
   • Example: FuelWorks removes HTML and fixes formatting
   • FuelWorks is the only file that can sometimes produce HTML tags so far but the logic could be abstracted if needed

4. Initial Processing

   • Laravel Excel converts the contenst of the file to an array
   • The WithHeadings means Laravel Excel automatically maps the headings to the data

5. Data Validation

   • Validation rules applied to each row
   • Errors grouped and formatted for user feedback

6. Data Transformation

   • Raw data mapped to standardized format in the import file
   • Data is then mapped further in the the integration file
   • Vehicle/driver references resolved
   • Product categorization applied

7. Data added to the database

   • The data is provided to the different repositories
   • It is either matched to existing data or added to the database

5. Error Handling

ImportFileException

Custom exception for import-related errors:

• Contains grouped error messages
• Provides user-friendly feedback
• Tracks validation failures by row ranges
• Is the expected exception for the emailed report

Error Grouping

• Similar errors consolidated
• Row numbers converted to ranges (e.g., “1-5, 10, 15-20”)
• Detailed error descriptions provided

6. Validation System

Helper Functions

• parseImportTimestamp() - Handles various date formats
• excelDateToCarbon() - Converts Excel date numbers
• validDateTimeFormats() - Returns supported date formats

Custom Validation Rules

• ImportDateRule - Validates date strings
• ImportExcelDateRule - Validates Excel date numbers

File Processing Examples

FuelWorks Integration

// File preprocessing to fix formatting issues
public function mutateFileBeforeImport(string $file_path): string
{
    $contents = file_get_contents($file_path);

    // Remove HTML tags
    $clean = strip_tags($contents);

    // Fix header formatting
    $clean = preg_replace('/TrNum\s+(\d{1,2} [A-Za-z]{3} \d{4})/', 'TrNum,$1', $clean, 1);

    // Add missing commas in data rows
    $clean = preg_replace('/(\d+)\s+(\d{1,2} [A-Za-z]{3} \d{4})/', '$1,$2', $clean);

    file_put_contents($file_path, $clean);
    return $file_path;
}