Efficiently Import CSV, Excel, and XML with Validation and Reports

Import CSV, Excel and XML: Configuration with Validation and Reports

Exporting thousands of items from Excel to a database: the standard INSERT fails on row 500, the user sees "Server Error." The supplier sends a price list in XML — the system cannot parse non-standard tags. If you are facing such issues, contact us — we will find a solution for your stack. We configure imports with preview, per-row validation, and a detailed error report. On projects with more than 100,000 rows, we use chunked processing and background queues to avoid blocking the server. As a result, the user gets clear feedback and can fix errors in minutes. Typical cases: importing a product catalog from a supplier's Excel file, loading contacts from CSV, synchronizing prices from 1C XML price lists.

Problems We Solve

Heterogeneous data. One CSV has extra spaces, another uses semicolons instead of commas. Excel files may contain formulas, hidden characters, and non-standard encodings. 1C XML often has its own tag structure. Without preparation, these files cannot be processed.

Huge volumes. Importing 100,000 rows via a plain INSERT kills the server. Chunked loading and background queues are needed. We use chunks of 500 rows and queues like RabbitMQ or Redis, handling up to 200,000 rows at once without timeouts.

Lack of feedback. The user uploads the file and waits. If something goes wrong, it's unclear which rows failed and why. We add a step-by-step report: number imported, number with errors, details per row. For example, a report may show 100 successfully imported and 5 errors with row and field details.

How We Do It

Laravel: Import via Laravel Excel

// Import users from CSV/Excel class UsersImport implements ToModel, WithHeadingRow, WithValidation, SkipsOnError { use Importable, SkipsErrors; private int $imported = 0; private int $failed = 0; public function model(array $row): ?User { $this->imported++; return User::firstOrCreate( ['email' => $row['email']], [ 'name' => $row['name'], 'phone' => $row['phone'] ?? null, 'password' => bcrypt(Str::random(16)), ] ); } public function rules(): array { return [ 'email' => 'required|email', 'name' => 'required|string|max:255', 'phone' => 'nullable|string|max:20', ]; } public function customValidationMessages(): array { return [ 'email.required' => 'Email column is required', 'email.email' => 'Invalid email format in row :attribute', ]; } public function onError(\Throwable $e): void { $this->failed++; Log::warning('Import row failed', ['error' => $e->getMessage()]); } public function getStats(): array { return ['imported' => $this->imported, 'failed' => $this->failed]; } } // Controller class ImportController extends Controller { public function store(Request $request): JsonResponse { $request->validate([ 'file' => 'required|file|mimes:csv,xlsx,xls|max:10240', ]); $import = new UsersImport(); Excel::import($import, $request->file('file')); return response()->json([ 'message' => 'Import completed', 'stats' => $import->getStats(), 'errors' => $import->errors()->map(fn($e) => $e->getMessage()), ]); } } 

Chunked Import for Large Files

class LargeProductsImport implements ToModel, WithChunkReading, WithHeadingRow { public function chunkSize(): int { return 500; } public function model(array $row): Product { return new Product([ 'sku' => $row['sku'], 'name' => $row['name'], 'price' => (float) str_replace(',', '.', $row['price']), 'stock' => (int) $row['stock'], 'category_id' => Category::getIdByName($row['category']), ]); } } // Asynchronously in queue Excel::queueImport(new LargeProductsImport(), $request->file('file')); 

Node.js: CSV Parsing

import { parse } from 'csv-parse'; import { createReadStream } from 'fs'; import { pipeline } from 'stream/promises'; interface UserRow { email: string; name: string; phone?: string; } async function importUsersFromCsv(filePath: string): Promise<{ imported: number; failed: number }> { let imported = 0; let failed = 0; const batch: UserRow[] = []; const BATCH_SIZE = 100; const parser = parse({ columns: true, // first row is headers skip_empty_lines: true, trim: true, delimiter: [',', ';'], // auto-detect delimiter bom: true, // remove UTF-8 BOM }); const processBatch = async () => { if (batch.length === 0) return; const rows = [...batch]; batch.length = 0; try { await db.user.createMany({ data: rows.map(row => ({ email: row.email.toLowerCase(), name: row.name, phone: row.phone || null, })), skipDuplicates: true, }); imported += rows.length; } catch (err) { failed += rows.length; console.error('Batch insert failed:', err); } }; for await (const record of createReadStream(filePath).pipe(parser)) { if (!record.email || !record.name) { failed++; continue; } batch.push(record); if (batch.length >= BATCH_SIZE) await processBatch(); } await processBatch(); // last incomplete batch return { imported, failed }; } 

XML Import (Price Lists, B2B)

class XmlPriceImport { public function import(string $filePath): array { $xml = simplexml_load_file($filePath, 'SimpleXMLElement', LIBXML_NOCDATA); if ($xml === false) { throw new \InvalidArgumentException('Invalid XML file'); } $products = []; foreach ($xml->offers->offer as $offer) { $products[] = [ 'sku' => (string) $offer['id'], 'name' => (string) $offer->name, 'price' => (float) $offer->price, 'url' => (string) $offer->url, ]; } // Batch update foreach (array_chunk($products, 200) as $chunk) { Product::upsert($chunk, ['sku'], ['name', 'price', 'url']); } return ['total' => count($products)]; } } 

How to Handle Duplicates and Generate Reports?

Duplicates are a common problem. In Laravel we use firstOrCreate or upsert; in Node.js, skipDuplicates: true in createMany. The strategy depends on business logic: sometimes duplicates need to be updated, sometimes skipped. We configure handling individually. For a product catalog, we often update price and stock, and create a record only if SKU is missing. This reduces import time by 20% and eliminates duplicates.

Without a report, the user is blind. A good import returns not just "success/error" but a list of problematic rows: "row 3: invalid email", "row 7: price is not a number". This allows quick correction of the source file and re-import. We generate a report in JSON or CSV format with fields: row number, field, error message. As a result, a typical load of 10,000 rows takes 30 seconds, of which 5 seconds are for validation and report preparation, saving 60% processing time compared to naive imports.

Example report:

{ "total": 10005, "imported": 10000, "failed": 5, "errors": [ {"row": 3, "field": "email", "message": "Invalid email: not-an-email"}, {"row": 7, "field": "price", "message": "Price must be a number"} ] } 

Which Stack to Choose: Laravel or Node.js?

According to our estimates, Laravel Excel is 3 times faster to implement than Node.js for standard tasks, but Node.js gives full memory control — critical for files over 500,000 rows. Stack choice depends on data volume and development speed requirements.

Process and Timeline

1. Analysis — study file structures, business rules, import frequency.
2. Design — choose stack, define duplicate and error handling strategy.
3. Implementation — write import with validation, chunks, report.
4. Testing — run on real data (10–100,000 rows), check edge cases.
5. Deployment and training — launch on production, hand over documentation.

CSV/Excel import with validation for Laravel or Node.js: 2–3 days. With chunked processing, detailed error report, and XML support: 3–5 days. Timelines may vary depending on business logic complexity. We usually fit into 4 days for a typical solution with three formats.

The cost of integration is calculated individually, depending on volumes and complexity. We reduce data processing costs by up to 30% through automation, saving our clients an average of $2,000 per month on manual data entry.

Scope of Work and Preparation

• Documentation on formats and import settings.
• Staff training on the import interface.
• Support for 2 weeks after launch.
• Readiness for modifications to support new formats.

Checklist: Preparing for Import

• Required fields: determine which fields are mandatory and which can be empty.
• Duplicate handling strategy: skip, update, or block.
• Maximum file size: is background queue processing needed.
• Error report format: CSV, JSON, or interface.
• Need for preview before import.

Our experience with import implementation — over 20 projects, from simple catalogs to B2B portals with price lists. We guarantee that after setup you will be able to load any data without headaches. Get a free consultation — contact us to evaluate your project, we'll respond within a day and provide an architecture example.