Product Import Validation: Sanitization & Deduplication

Data Validation for Product Import

Our product import data validation system handles data verification, sanitization, and SKU deduplication, catching errors like negative prices, empty SKUs, and XSS in descriptions. Without such a system, corrupted data leads to catalog issues and high operational costs. According to statistics, up to 30% of supplier data contains errors — fixing the consequences costs hundreds of thousands of rubles monthly. Based on our data, product import validation automation saves up to 800,000 rubles per year on an average project. Our approach: stop garbage at the entry point, never letting it reach the database. Pricing for the validation system implementation starts at 150,000 rubles (approximately $2,000). Typical annual savings exceed $10,000, with ROI achieved within 3 months. With our guarantee of data quality, you can trust the import process. Our certified developers have over 7 years of experience in e-commerce data integration and have completed 50+ validation projects.

Problems We Solve

Structural validation — we check format and types: required fields, numbers, dates, string lengths. If a price list contains "N/A" instead of a number, we catch it before database insertion.

Business validation — semantic rules: price does not exceed limits, SKU is unique, category exists, price change no more than 50%. For example, a supplier accidentally sets price 999999 instead of 999 — we filter out the anomaly.

Sanitization — we clean data before checks: remove invisible characters, parse numbers with spaces and commas, strip dangerous HTML.

SKU deduplication — a single file may contain duplicate SKUs if the supplier merged several price lists. We track all SKUs within the import and warn about duplicates.

If you notice similar issues in your data, contact us for a free audit.

How We Do It: Stack and Architecture

We use Laravel 11 with PHP 8.3. The validator is built on Laravel Validator with additional business rules. The sanitizer parses numbers, cleans strings and images. Errors are collected with severity and batch-written to the database.

Example validator:

class ProductImportValidator { private array $rules = [ 'sku' => ['required', 'string', 'max:100'], 'name' => ['required', 'string', 'max:500'], 'price' => ['required', 'numeric', 'min:0.01', 'max:100000000'], 'qty' => ['nullable', 'integer', 'min:0', 'max:9999999'], 'description' => ['nullable', 'string', 'max:100000'], 'category' => ['nullable', 'string', 'max:300'], 'images' => ['nullable', 'array', 'max:20'], 'images.*' => ['url', 'max:2000'], 'weight' => ['nullable', 'numeric', 'min:0', 'max:10000'], ]; public function validate(array $row): ValidationResult { $validator = \Illuminate\Support\Facades\Validator::make( $row, $this->rules, $this->customMessages() ); $errors = []; if ($validator->fails()) { $errors = $validator->errors()->toArray(); } // Business rules $errors = array_merge($errors, $this->applyBusinessRules($row)); return new ValidationResult( valid: empty($errors), errors: $errors, data: $validator->validated(), ); } private function applyBusinessRules(array $row): array { $errors = []; // Check for anomalous price change if (!empty($row['sku']) && !empty($row['price'])) { $existing = Product::where('sku', $row['sku'])->value('price'); if ($existing && $existing > 0) { $change = abs($row['price'] - $existing) / $existing; if ($change > 0.5) { $errors['price'][] = "Price change {$change}% exceeds 50% threshold"; } } } // Check for XSS in description if (!empty($row['description'])) { $clean = strip_tags($row['description']); if ($clean !== $row['description']) { $errors['description'][] = 'HTML tags detected in description'; } } return $errors; } } 

Example sanitizer:

class ProductDataSanitizer { public function sanitize(array $raw): array { return [ 'sku' => $this->cleanString($raw['sku'] ?? ''), 'name' => $this->cleanString($raw['name'] ?? ''), 'price' => $this->parseDecimal($raw['price'] ?? null), 'qty' => $this->parseInt($raw['qty'] ?? null), 'description' => $this->sanitizeHtml($raw['description'] ?? ''), 'weight' => $this->parseDecimal($raw['weight'] ?? null), 'images' => $this->parseImageUrls($raw['images'] ?? []), ]; } private function cleanString(?string $value): string { if ($value === null) return ''; $value = trim($value); $value = preg_replace('/\p{C}/u', '', $value); // invisible chars return mb_substr($value, 0, 1000); } private function parseDecimal(mixed $value): ?float { if ($value === null || $value === '') return null; $value = str_replace([' ', ',', "\xc2\xa0"], ['', '.', ''], (string) $value); return is_numeric($value) ? (float) $value : null; } private function parseInt(mixed $value): ?int { $decimal = $this->parseDecimal($value); return $decimal !== null ? (int) $decimal : null; } private function sanitizeHtml(?string $html): string { if (!$html) return ''; // Allow only safe tags return strip_tags($html, '<p><br><b><strong><i><em><ul><ol><li>'); } private function parseImageUrls(mixed $raw): array { if (is_string($raw)) { $raw = array_map('trim', explode(',', $raw)); } return array_values(array_filter( (array) $raw, fn($url) => filter_var($url, FILTER_VALIDATE_URL) )); } } 

Importance of Data Sanitization for Security

Allowing HTML tags in product descriptions can lead to XSS attacks on the catalog page. Our sanitizer strips all dangerous tags, leaving only safe <p>, <br>, <strong>. This prevents XSS attacks (see XSS and OWASP XSS Prevention Cheat Sheet). In one project, we found up to 5% of rows with potentially dangerous HTML.

Error Classification and Its Importance

Not all errors are equally dangerous. We separate them by severity: CRITICAL (row skipped), WARNING (row imported with a flag), INFO (logged only). If the share of critical errors exceeds 20%, the entire import is aborted — this prevents mass catalog pollution.

Additionally, we use an import abort threshold: if critical errors exceed 20%, the process stops. This is 5 times more reliable than standard Laravel validation without such protection.

Process

1. Analysis — we study current supplier formats (CSV, XML, Excel), identify typical errors, agree on rules. On average, we process 12 suppliers in 2 days.
2. Design — we design the validation schema, define severity, thresholds, deduplication strategy. Up to 50 fields per product.
3. Implementation — we write validators, sanitizers, error collectors. For Laravel we use Validator; for other stacks, analogous tools.
4. Testing — we run on real supplier data (usually 1000+ rows), adjust rules.
5. Deployment — deploy to production, monitor initial imports.

What's Included

• Structural and business validation system with configurable rules
• Sanitizer for cleaning strings, numbers, URLs, and HTML
• SKU deduplication within a single import
• Error collector with three severity levels and database storage
• Import abort threshold for critical errors
• Administrative interface for viewing errors and import statuses
• Documentation of rules and API description for connecting new suppliers
• Improved data quality through automated checks

Timelines

• Structural validation (Laravel Validator), sanitization, error collection — from 1 day
• Business rules, SKU deduplication, severity levels — from 1 day
• Error storage in DB, abort threshold, admin UI — from 1 day

Total 3 days turnkey. Cost is calculated individually based on the number of suppliers and rule complexity. If you have a similar problem, contact us for an audit. Order a validation system development — get a consultation within 2 hours. We have 7+ years of experience in e-commerce solutions — we'll assess your project in 2 hours after getting acquainted with your data.