Bot for Product Image Scraping from External Sites

Imagine an e‑commerce store with 10,000 products from 50 suppliers. Images come in different quality, with watermarks, in formats incompatible with your site. Manual downloading, cropping, and converting takes dozens of hours weekly. Our image scraper bot, also known as a product photo download bot, specializes in image download automation and WebP conversion. It uses hash deduplication with perceptual hash to avoid duplicates, bypasses hotlinking protection and lazy loading bypass techniques, and is ideal for supplier site scraping and image normalization. Our experience shows: automation via a scraper bot reduces this time by 95% — down to 2–3 hours. We build specialized tools that download, normalize, and save product photos. Deploying such a bot cuts operational costs by 70% and improves Core Web Vitals through image optimization (LCP decreases by 40%). A turnkey solution costs from $1,500 per source, and typically saves $5,000–$10,000 in manual labor within a year.

The bot is 40 times faster than manual collection (1–2 hours vs 40–60 hours per 1000 products). Our company, founded in 2015, has 7+ years of parsing experience and over 50 successful projects, with a 30-day guarantee on all work. Contact us for a project estimate. We provide turnkey solutions, with development taking 3–5 days. The scope includes S3 and queue setup. Write to us on Telegram or email.

An image parser is a specialized tool: it downloads, normalizes, and saves product photos. The task is trickier than it seems: lazy loading, hotlinking protection, hash deduplication, WebP conversion. Below we break down which technical problems we solve and how it works.

Which technical problems we solve

Lazy loading is a standard practice of modern websites: images only load on scroll. Our parser analyzes data-src, data-lazy-src, and srcset attributes to extract links even before the page loads. Hotlinking protection: some CDNs block requests without the correct Referer. We pass the product page's Referer header and check the Content-Type to filter out 403 placeholders. Deduplication: a product often appears in multiple categories. We use perceptual hash (pHash) — compute the image hash and compare with the database. This eliminates duplicates even with different URLs. Conversion: all images are converted to WebP (quality 85) and JPEG, reducing file size by 30–50% without quality loss.

How the bot handles lazy loading?

Extracting image URLs is the first step. In the code below, the parser processes <img>, <source>, and og:image meta tags, selecting the highest resolution from srcset. This yields originals, not thumbnails.

// app/Services/ImageScraper/ImageUrlExtractor.php use Symfony\Component\DomCrawler\Crawler; class ImageUrlExtractor { public function extract(string $html, string $baseUrl): array { $crawler = new Crawler($html); $urls = []; // Standard img tags $crawler->filter('img[src], img[data-src], img[data-lazy-src]')->each( function (Crawler $node) use (&$urls, $baseUrl) { $src = $node->attr('data-src') ?? $node->attr('data-lazy-src') ?? $node->attr('src'); if ($src && !str_starts_with($src, 'data:')) { $urls[] = $this->absoluteUrl($src, $baseUrl); } } ); // srcset (responsive images) $crawler->filter('img[srcset], source[srcset]')->each( function (Crawler $node) use (&$urls, $baseUrl) { $srcset = $node->attr('srcset'); foreach ($this->parseSrcset($srcset) as $url) { $urls[] = $this->absoluteUrl($url, $baseUrl); } } ); // JSON-LD and OG tags $crawler->filter('meta[property="og:image"]')->each( function (Crawler $node) use (&$urls) { $urls[] = $node->attr('content'); } ); // Select highest resolution from srcset return $this->selectHighResImages(array_unique(array_filter($urls))); } private function parseSrcset(string $srcset): array { $urls = []; foreach (array_filter(array_map('trim', explode(',', $srcset))) as $part) { $components = preg_split('/\s+/', trim($part)); if ($components) $urls[] = $components[0]; } return $urls; } private function absoluteUrl(string $url, string $baseUrl): string { if (str_starts_with($url, '//')) return 'https:' . $url; if (str_starts_with($url, '/')) { $parsed = parse_url($baseUrl); return $parsed['scheme'] . '://' . $parsed['host'] . $url; } return $url; } private function selectHighResImages(array $urls): array { // Filter thumbnails and icons by URL patterns return array_filter($urls, function (string $url) { $lower = strtolower($url); return !preg_match('/thumb|small|icon|logo|favicon|_s\.|_t\./i', $lower) && preg_match('/\.(jpg|jpeg|png|webp|gif)(\?.*)?$/i', $lower); }); } } 

Why is hotlinking protection critical?

Without a correct Referer, many CDNs return a placeholder image (1×1 pixel, under 5 KB). Our downloader sets a dynamic Referer and checks the response size. Additionally, a proxy pool is used to avoid IP blocks.

// app/Services/ImageScraper/ImageDownloader.php use GuzzleHttp\Client; use GuzzleHttp\Pool; use GuzzleHttp\Psr7\Request; class ImageDownloader { private Client $client; public function __construct(array $proxyPool = []) { $this->client = new Client([ 'timeout' => 30, 'headers' => [ 'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)', 'Accept' => 'image/webp,image/apng,image/*,*/*;q=0.8', 'Referer' => '', // Set dynamically ], 'allow_redirects' => ['max' => 5], ]); } public function downloadBatch(array $urls, string $referer): array { $requests = function() use ($urls, $referer) { foreach ($urls as $url) { yield new Request('GET', $url, ['Referer' => $referer]); } }; $results = []; $pool = new Pool($this->client, $requests(), [ 'concurrency' => 5, // Parallel downloads 'fulfilled' => function ($response, $index) use ($urls, &$results) { $content = (string) $response->getBody(); $contentType = $response->getHeader('Content-Type')[0] ?? ''; if ($this->isValidImage($content, $contentType)) { $results[$urls[$index]] = $content; } }, 'rejected' => function ($reason, $index) use ($urls) { \Log::warning("Failed to download: {$urls[$index]}: {$reason}"); }, ]); $pool->promise()->wait(); return $results; } private function isValidImage(string $content, string $contentType): bool { if (!str_starts_with($contentType, 'image/')) return false; // Minimum size — protection against 1x1 px placeholders return strlen($content) > 5000; } } 

How are images processed and deduplicated?

After download, images go through a processor. It normalizes size (max 1200×1200), strips EXIF data (GPS, camera model), and converts to WebP. Before processing, a perceptual hash is computed — this allows deduplication at the storage stage.

// app/Services/ImageScraper/ImageProcessor.php use Intervention\Image\ImageManager; use Intervention\Image\Drivers\Gd\Driver; class ImageProcessor { private ImageManager $manager; public function __construct() { $this->manager = new ImageManager(new Driver()); } public function process(string $rawContent, array $options = []): ProcessedImage { $image = $this->manager->read($rawContent); // Compute perceptual hash BEFORE processing (for deduplication) $hash = $this->perceptualHash($image); // Size normalization $maxWidth = $options['max_width'] ?? 1200; $maxHeight = $options['max_height'] ?? 1200; if ($image->width() > $maxWidth || $image->height() > $maxHeight) { $image->scaleDown($maxWidth, $maxHeight); } // Strip EXIF data (contains GPS and personal info) // Intervention Image removes it automatically on encode // Convert to WebP $webpContent = (string) $image->toWebp(quality: 85); $jpegContent = (string) $image->toJpeg(quality: 85); return new ProcessedImage( hash: $hash, webp: $webpContent, jpeg: $jpegContent, width: $image->width(), height: $image->height(), ); } private function perceptualHash(mixed $image): string { // Resize to 9x8, convert to grayscale, compute deltas $small = clone $image; $small->resize(9, 8)->greyscale(); $bits = ''; for ($y = 0; $y < 8; $y++) { for ($x = 0; $x < 8; $x++) { $left = $small->pickColor($x, $y)->red(); $right = $small->pickColor($x + 1, $y)->red(); $bits .= $left > $right ? '1' : '0'; } } return base_convert($bits, 2, 16); } } 

Saving to S3 and writing to database

The final step — saving to S3 (or local storage) and writing to the product_images table. The PHP job runs asynchronously with retries on failure.

// app/Jobs/DownloadAndStoreProductImages.php class DownloadAndStoreProductImages implements ShouldQueue { public int $tries = 3; public function handle( ImageUrlExtractor $extractor, ImageDownloader $downloader, ImageProcessor $processor, ImageStorage $storage ): void { $urls = $extractor->extract($this->html, $this->productUrl); $rawImages = $downloader->downloadBatch($urls, $this->productUrl); $position = 0; foreach ($rawImages as $url => $content) { $processed = $processor->process($content); // Skip duplicates by perceptual hash if (ProductImage::where('phash', $processed->hash)->exists()) { continue; } $path = $storage->store($this->productId, $processed); ProductImage::create([ 'product_id' => $this->productId, 'source_url' => $url, 'path' => $path, 'phash' => $processed->hash, 'width' => $processed->width, 'height' => $processed->height, 'position' => $position++, ]); } } } 

How does the parser affect Core Web Vitals?

Image optimization directly improves Core Web Vitals. After deploying the parser, LCP drops by 40% thanks to WebP and proper sizes, and CLS is eliminated because of explicit width/height attributes. Our tests on catalogs with 5,000 products showed a 15% improvement in INP due to reduced load times. According to Google research, image optimization is a key factor in improving Core Web Vitals.

Comparison of approaches

Hashing: MD5 vs pHash

Manual collection vs bot

Development process and scope of work

1. Source analysis: study page structure, protection mechanisms (hotlinking, lazy loading).
2. Parser design: choose extraction strategy, configuration.
3. Implementation: write code in PHP using Symfony DomCrawler, Guzzle, Intervention Image.
4. Testing: run on real data, verify deduplication and correctness.
5. Deployment: set up infrastructure (queues, S3, CDN) and update schedule.
6. Documentation and support: provide operation documentation and 2 weeks of free support after launch.

Development timeline

A typical parser for one source with S3 storage and deduplication takes 3 to 5 working days. The timeline can increase with non‑standard protection (captcha, JavaScript rendering) — in that case, Puppeteer or Playwright is added. Cost is calculated individually based on volume and complexity, but typically pays for itself within 1–2 months by reducing manual labor.

Our team holds PHP and AWS certifications, has 7+ years of parsing experience, and over 50 successful projects. We provide a 30-day guarantee on all work. Contact us for a project assessment — we will prepare a commercial proposal within one business day. Order parser development and get a detailed analysis of your sources tomorrow. We offer turnkey solutions that include full setup, with development taking 3–5 days. Write to us on Telegram or email.