Supplier Product Parser Bot: Automated Data Collection

Managers spend up to 3 hours daily copying data from supplier dashboards. Manual entry leads to error rates of 15%, and outdated prices cause direct losses. The result: lost deals and dissatisfied customers. We automate the collection of products, prices, and stock without human intervention. In our time in the market, we have delivered 30+ projects, reducing data collection time for clients by 80%. For example, an online store with 5,000 SKUs spent 40 hours monthly updating prices; after implementing the parser, the process takes 4 hours autonomously. Budget savings of up to 30%, which in monetary terms could, for example, amount to $2.2k–3.1k per year.

According to the definition, web scraping is the automatic extraction of data from the internet. Here's how it works in practice.

Why manual data collection is costly?

80% of time goes to monotonous tasks. 15% of records contain errors due to human factors. Customers are lost due to outdated prices or out-of-stock items. Automation reduces costs by 30% and completely eliminates input errors.

What data we collect

How the parser works and how we bypass protection?

Typical solution architecture:

1. Scheduler (cron / Laravel Horizon) runs tasks on schedule.
2. HTTP client (Guzzle / Playwright) fetches supplier pages.
3. Parser extracts data using CSS selectors or XPath.
4. Normalizer brings prices, dates, and currencies to a unified format.
5. Deduplicator checks products by SKU and updates or creates records.
6. Notifications for errors or significant price changes.

Modern suppliers use Cloudflare, CAPTCHAs, and dynamic loading. We apply residential proxy rotation, Playwright/Puppeteer browser emulation with random mouse movements, delays from 500 ms to 2 s, and User-Agent switching. If the site uses JavaScript, Playwright loads the page like a real browser and returns the ready DOM.

What is better: Guzzle or Playwright?

Guzzle processes static pages 2 times faster than Playwright. Combined approach — Guzzle for static, Playwright for SPA — yields the best result.

Technical implementation

Basic parser in PHP

// app/Services/Scrapers/SupplierScraper.php
use GuzzleHttp\Client;
use Symfony\Component\DomCrawler\Crawler;

class SupplierScraper
{
    private Client $client;

    public function __construct(
        private string $baseUrl,
        private array $proxyPool = []
    ) {
        $this->client = new Client([
            'timeout' => 15,
            'connect_timeout' => 5,
            'headers' => [
                'User-Agent' => $this->randomUserAgent(),
                'Accept-Language' => 'ru-RU,ru;q=0.9',
                'Accept' => 'text/html,application/xhtml+xml',
            ],
        ]);
    }

    public function scrapeProductList(string $categoryUrl): array
    {
        $html = $this->fetchWithRetry($categoryUrl);
        $crawler = new Crawler($html);
        return $crawler->filter('.product-card')->each(function (Crawler $node) {
            return [
                'url' => $node->filter('a.product-link')->attr('href'),
                'title' => trim($node->filter('.product-title')->text()),
                'price' => $this->parsePrice($node->filter('.price')->text()),
                'sku' => $node->filter('[data-sku]')->attr('data-sku'),
            ];
        });
    }

    public function scrapeProductDetail(string $productUrl): array
    {
        $html = $this->fetchWithRetry($this->baseUrl . $productUrl);
        $crawler = new Crawler($html);
        return [
            'title' => $crawler->filter('h1.product-name')->text(),
            'description' => $crawler->filter('.description')->html(),
            'price' => $this->parsePrice($crawler->filter('.current-price')->text()),
            'images' => $crawler->filter('.gallery img')->each(
                fn(Crawler $img) => $img->attr('src')
            ),
            'specs' => $this->extractSpecs($crawler),
            'in_stock' => $crawler->filter('.in-stock')->count() > 0,
            'sku' => $crawler->filter('[itemprop="sku"]')->text(''),
        ];
    }

    private function extractSpecs(Crawler $crawler): array
    {
        $specs = [];
        $crawler->filter('.specs-table tr')->each(function (Crawler $row) use (&$specs) {
            $key = trim($row->filter('td:first-child')->text(''));
            $val = trim($row->filter('td:last-child')->text(''));
            if ($key && $val) {
                $specs[$key] = $val;
            }
        });
        return $specs;
    }

    private function fetchWithRetry(string $url, int $attempts = 3): string
    {
        $proxy = $this->proxyPool ? $this->randomProxy() : null;
        for ($i = 0; $i < $attempts; $i++) {
            try {
                $options = $proxy ? ['proxy' => $proxy] : [];
                $response = $this->client->get($url, $options);
                return (string) $response->getBody();
            } catch (\Exception $e) {
                if ($i === $attempts - 1) throw $e;
                sleep(rand(2, 5));
            }
        }
    }

    private function parsePrice(string $text): float
    {
        return (float) preg_replace('/[^\d.,]/', '', str_replace(',', '.', $text));
    }

    private function randomUserAgent(): string
    {
        $agents = [
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120.0',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15',
        ];
        return $agents[array_rand($agents)];
    }
}

Job for background processing

// app/Jobs/ScrapeSupplierProducts.php
class ScrapeSupplierProducts implements ShouldQueue
{
    use Queueable;

    public int $tries = 2;
    public int $timeout = 300;

    public function __construct(
        private int $supplierId,
        private string $categoryUrl
    ) {}

    public function handle(SupplierScraper $scraper, ProductImportService $importer): void
    {
        $products = $scraper->scrapeProductList($this->categoryUrl);

        foreach ($products as $productPreview) {
            ScrapeSupplierProductDetail::dispatch(
                $this->supplierId,
                $productPreview['url']
            )->onQueue('scraper-detail');

            usleep(rand(500000, 1500000));
        }
    }
}

Playwright for JS-heavy sites

// scraper/playwright-worker.js
const { chromium } = require('playwright');

async function scrapeProduct(url) {
  const browser = await chromium.launch({ headless: true });
  const context = await browser.newContext({
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)...',
    viewport: { width: 1366, height: 768 },
  });
  const page = await context.newPage();
  await page.goto(url, { waitUntil: 'networkidle' });
  const product = await page.evaluate(() => ({
    title: document.querySelector('h1')?.textContent,
    price: document.querySelector('.price')?.textContent,
    images: [...document.querySelectorAll('.gallery img')].map(i => i.src),
  }));
  await browser.close();
  return product;
}

PHP calls the Node process via proc_open or an HTTP microservice.

Deduplication and update

// app/Services/ProductImportService.php
class ProductImportService {
    public function upsert(int $supplierId, array $data): void {
        $product = SupplierProduct::updateOrCreate(
            [
                'supplier_id' => $supplierId,
                'supplier_sku' => $data['sku'],
            ],
            [
                'title' => $data['title'],
                'price' => $data['price'],
                'in_stock' => $data['in_stock'],
                'description' => $data['description'],
                'images' => json_encode($data['images']),
                'specs' => json_encode($data['specs']),
                'scraped_at' => now(),
            ]
        );

        if ($product->wasChanged('price')) {
            $change = abs($product->price - $product->getOriginal('price'));
            if ($change / $product->getOriginal('price') > 0.05) {
                PriceChangedNotification::dispatch($product);
            }
        }
    }
}

Schedule configuration

// app/Console/Kernel.php
protected function schedule(Schedule $schedule): void
{
    $schedule->command('scraper:supplier --supplier=1')
        ->dailyAt('03:00')
        ->withoutOverlapping();

    $schedule->command('scraper:supplier --supplier=1 --prices-only')
        ->everyFourHours()
        ->withoutOverlapping();
}

Development process and timelines

1. Analysis (1-2 days) — study the supplier site structure, identify protection types, agree on fields.
2. Design (1 day) — choose the stack, design architecture (queues, schedule, data schema).
3. Implementation (3-7 days) — write parsers, normalizers, deduplication, integration with your system.
4. Testing (1-2 days) — run on real data, verify correctness and robustness.
5. Deployment (1 day) — configure environment, monitoring, notifications, hand over documentation.

Estimated timelines

• Basic parser for one supplier (static HTML, 5-10 fields): from 3 to 5 business days.
• Parser with complex protection or JS rendering: from 7 to 14 days.
• Integration with multiple suppliers into a single pipeline: from 10 business days.

Cost is calculated individually after auditing the target sites.

What's included in the result

• Parser with open-source code (PHP + Laravel or Node.js).
• Integration with your database via API or direct import.
• Schedule and monitoring configuration.
• Documentation for launch and maintenance.
• 30-day warranty on stable operation.

Typical mistakes in parser development

• Ignoring rate limit — the site blocks your IP after 100 requests. Solution: add random delays and rotation.
• Hardcoding HTML structure — the slightest design change breaks the parser. Solution: use data-attributes or XPath.
• Lack of error handling — one supplier failure crashes the entire queue. Solution: isolate tasks and add retries.
• Storing all data in a single table without normalization — duplicates and gaps. Solution: use SKU as the unique key.

Contact us for a consultation — we'll evaluate your project and offer a turnkey solution. Order your parser development and get a ready solution that saves your time and money.