Automated product image download and import

Automated product image download and import

During product import from a supplier price list, images are the main bottleneck. Without automation, you spend hours downloading, optimizing, and uploading each picture. And if the supplier changes links or adds new items, the process repeats. Our team, with 7 years of import automation experience, has processed over 5 million images for dozens of online stores. We automate this pipeline: parsing links from YML, CSV, or API, downloading with retries, validating MIME types, converting to WebP, and saving to storage. The result is a store with ready-to-use images without manual labor. Time savings reach 90% on the first bulk import.

Why automation is critical for an online store

With a catalog of 1,000+ products, manually handling each picture takes days. Operator errors—wrong size, wrong file, broken link—lead to incomplete product cards and lower conversion rates. Our solution guarantees that 100% of images are downloaded, optimized, and attached to products in minutes. According to Intervention Image documentation, WebP conversion reduces file sizes by 30-50% without visible quality loss.

How to avoid duplicates during import

Deduplication is implemented on two levels. Before downloading, we check source_url in the product_images table. If the URL already exists and the file is on disk, the task is skipped. For extra protection, we compute a SHA-256 hash of the first 4 KB of content: identical files under different URLs are not downloaded twice. This is critical when dealing with suppliers that may re-send the same product with different URLs.

How we do it: step-by-step plan

1. Analyze the supplier's price list format (CSV, YML, API, FTP).
2. Develop a link parser tailored to the data structure.
3. Configure an ImageDownloadJob queue with retries and timeouts.
4. Implement MIME type validation and size limits.
5. Perform WebP conversion with multiple size generation.
6. Add deduplication by URL and content hash.
7. Set up dead-link monitoring and reporting.
8. Conduct load testing and train administrators.

Technical implementation: download, validation, and optimization

Under the hood, we use Laravel Queue with a dedicated connection for images. The ImageDownloadJob class downloads the file via an HTTP client, checks the size (max 20 MB) and MIME type (JPEG, PNG, WebP, GIF). Then, using Intervention Image (v3) and Spatie Image Optimizer, we resize and convert to WebP in three sizes: full (1200x1200), catalog (400x400), thumbnail (100x100).

class ImageProcessor { private const ALLOWED_MIME = ['image/jpeg', 'image/png', 'image/webp', 'image/gif']; private const MAX_SIZE = 20 * 1024 * 1024; public function download(string $url): string { $response = $this->client->get($url, ['timeout' => 30, 'stream' => true]); $tmpPath = tempnam(sys_get_temp_dir(), 'img_'); $body = $response->getBody(); $size = 0; $fp = fopen($tmpPath, 'wb'); while (!$body->eof()) { $chunk = $body->read(8192); $size += strlen($chunk); if ($size > self::MAX_SIZE) { fclose($fp); unlink($tmpPath); throw new \RuntimeException("Image too large: {$url}"); } fwrite($fp, $chunk); } fclose($fp); $mime = mime_content_type($tmpPath); if (!in_array($mime, self::ALLOWED_MIME)) { unlink($tmpPath); throw new \RuntimeException("Invalid MIME type: {$mime} for {$url}"); } return $tmpPath; } public function processAndStore(string $tmpPath, int $productId, int $sort): string { $manager = new \Intervention\Image\ImageManager( new \Intervention\Image\Drivers\Gd\Driver() ); $image = $manager->read($tmpPath); $variants = ['full' => [1200, 1200], 'catalog' => [400,400], 'thumbnail' => [100,100]]; $paths = []; foreach ($variants as $name => [$w, $h]) { $resized = clone $image; $resized->coverDown($w, $h); $filename = "products/{$productId}/{$sort}_{$name}.webp"; $encoded = $resized->toWebp(quality: 85); Storage::disk('public')->put($filename, $encoded); $paths[$name] = $filename; } unlink($tmpPath); return json_encode($paths); } } 

The coverDown method crops the image from the center while preserving proportions—the standard for catalog photos. WebP conversion reduces file sizes by 30-50% without quality loss.

Error handling and broken links

On a 404 or broken link, the task retries 3 times with a 30-second interval. If all attempts fail, the link is marked as dead. Once a week, the administrator receives a report suggesting manual upload. Already saved images are not deleted, so existing product cards remain intact.

Parallelism and performance

With 10,000 images and 4 workers, the full cycle takes 20–40 minutes (depends on the supplier's server speed). The images queue does not block other processes.

Comparison: manual import vs our service

Automation pays for itself after the first mass import. Contact us—our engineers will evaluate your project for free and propose the optimal solution.

What is included in the work

• Parsing links from CSV, YML, API, FTP.
• Developing ImageDownloadJob with retries and timeouts.
• MIME and size validation.
• WebP conversion with configurable sizes.
• Deduplication by URL and hash.
• Dead-link monitoring and reporting.
• Operation documentation.
• Administrator training.
• 3-month warranty support.

Implementation timelines

• Basic integration (HTTP, validation, WebP, storage) — 2 days.
• Multiple size generation + deduplication — +1-2 days.
• FTP source + parallel queue + progress dashboard — +1 day.

Exact timelines depend on the supplier's data format. Order automated image import and eliminate routine—submit a request, and we will prepare a custom proposal.