Universal Product Import from CSV, Excel, XML, JSON

Implementing Product Import from Supplier Files (CSV/Excel/XML/JSON)

We often encounter suppliers sending price lists in whatever format is convenient for them: some in Excel, some in XML, some in CSV with non-standard delimiters. We solve the challenge of building a universal product import that works with any format through a single interface without writing separate code for each supplier. On one project, we integrated price lists from 15 different suppliers, each with its own column structure. Up to 500,000 rows were processed per shift — manual loading was infeasible. The solution was a universal parser with a factory and field mapping that cut development time by 60% and allowed onboarding a new supplier in 2 hours. Proper import architecture pays off by the third supplier.

Why a Universal Parser Interface is Key to Scalability

A single FileParserInterface enables adding new formats without altering the import logic. A factory selects the parser based on extension or MIME type. For example, for CSV we use a configurable delimiter and encoding; for XML we use streaming XMLReader, which saves 10x memory compared to SimpleXML. Adding a new format boils down to writing one class and registering it in the factory.

interface FileParserInterface { /** @return iterable<array<string, mixed>> */ public function parse(string $filePath): iterable; public function supports(string $mimeType, string $extension): bool; } 

class FileParserFactory { private array $parsers; public function make(string $filePath): FileParserInterface { $ext = strtolower(pathinfo($filePath, PATHINFO_EXTENSION)); $mime = mime_content_type($filePath); foreach ($this->parsers as $parser) { if ($parser->supports($mime, $ext)) return $parser; } throw new \RuntimeException("No parser for: {$ext} / {$mime}"); } } 

How to Handle CSV with Non-standard Delimiters and Encodings

CSV is the most unpredictable format. Delimiters: comma, semicolon, tab. Encodings: UTF-8 with or without BOM, Windows-1251. Our parser is configurable per source: we auto-detect BOM, convert Windows-1251 to UTF-8, and handle non-standard delimiters via settings. Example:

class CsvParser implements FileParserInterface { public function __construct( private string $delimiter = ',', private string $enclosure = '"', private bool $hasHeader = true, ) {} public function parse(string $filePath): iterable { $handle = fopen($filePath, 'r'); $bom = fread($handle, 3); fclose($handle); if ($bom === "\xEF\xBB\xBF") { $filePath = $this->removeBom($filePath); } $handle = fopen($filePath, 'r'); $headers = $this->hasHeader ? fgetcsv($handle, 0, $this->delimiter, $this->enclosure) : null; while ($row = fgetcsv($handle, 0, $this->delimiter, $this->enclosure)) { if (!array_filter($row)) continue; yield $headers ? array_combine($headers, $row) : $row; } fclose($handle); } public function supports(string $mimeType, string $extension): bool { return in_array($extension, ['csv', 'txt']) || str_contains($mimeType, 'csv'); } } 

For files over 10 MB we use streaming, reducing memory consumption by 3–5 times. As noted in the PhpSpreadsheet documentation, streaming reduces memory usage by up to 70%. An Excel file of 200 MB with setReadDataOnly(true) is processed in 40 seconds on a typical VPS.

Processing Large Excel and XML Files

For Excel we use PhpSpreadsheet with memory-saving options. For XML we use streaming XMLReader. Comparison:

Streaming XML parser processes files 10 times faster than loading the entire document with SimpleXML. This is critical when a supplier sends a price list with 1 million items. In a test with a 500 MB file, the streaming parser processed 1 million records in 90 seconds.

Ensuring Data Integrity

Each row is validated before writing to the database. Invalid rows (missing required fields, incorrect SKU) are logged and do not interrupt the import. Re-imports do not create duplicates — we use upsert by SKU key. For integrity control, we store a row hash and last update date.

Column Mapping Eliminates Manual Work

Every supplier uses their own column names. Mapping configuration is stored in the database and editable via the admin panel:

{ "sku": "Артикул", "name": "Наименование", "price": "Цена руб.", "qty": "Кол-во", "description": "Описание", "category": "Раздел" } 

The transformer applies the mapping before passing data to the importer. Thanks to DB-stored configuration, setting up a new supplier takes 30 minutes instead of 4 hours. Different mappings are supported for different suppliers.

What's Included in the Work

• Development of parsers for all formats (CSV, Excel, XML, JSON).
• Configuration of column mapping for each supplier.
• Creation of a UI for managing mapping and viewing logs.
• Testing with real files (up to 1 million rows).
• Documentation on architecture and adding new formats.
• Operator training on the admin panel.
• Technical support during integration.

Implementation Process

1. Analysis of supplier file formats and mapping requirements.
2. Development of base parsers and import pipeline.
3. Configuration of mapping and streaming processing.
4. Integration testing with supplier exports.
5. Deployment to production server and go-live.

Implementation Timeline

Total: from 3 to 4 days for basic integration. Timeline varies based on number of formats and mapping complexity.

Contact us for a project assessment — we'll prepare a commercial proposal within one day. Order a universal import implementation and forget about manual price list loading.