Competitor Price Monitoring Bot with Reports

Developing a Bot for Competitor Price Monitoring With Reports

Competitor price monitoring is not a one-time export but a continuous data collection process with subsequent analytics. The bot should track specific products on specific websites, store price history, and alert on significant changes. Without automation, this work is done manually, consuming hours daily and providing stale data.

Architecture

Scheduler (cron) → Scraper Workers → Price DB → Analytics → Reports/Alerts

Key components:

• Scraper — fetches HTML or JSON from competitor site
• Parser — extracts price from content
• Storage — stores change history
• Analyzer — calculates deltas, trends
• Notifier — sends reports to Telegram/Email

Data Schema

CREATE TABLE monitored_products (
    id              BIGSERIAL PRIMARY KEY,
    our_product_id  BIGINT REFERENCES products(id),
    competitor_id   INT REFERENCES competitors(id),
    url             TEXT NOT NULL,
    selector        VARCHAR(500),          -- CSS selector for price
    last_price      NUMERIC(12,2),
    last_checked_at TIMESTAMP,
    is_active       BOOLEAN DEFAULT TRUE,
    UNIQUE(competitor_id, url)
);

CREATE TABLE price_snapshots (
    id              BIGSERIAL PRIMARY KEY,
    monitored_id    BIGINT REFERENCES monitored_products(id),
    price           NUMERIC(12,2),
    in_stock        BOOLEAN,
    raw_text        VARCHAR(100),          -- "29 990 rubles" before parsing
    captured_at     TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_snapshots_monitored_captured
    ON price_snapshots(monitored_id, captured_at DESC);

Scraper With User-Agent and Proxy Rotation

class CompetitorScraper
{
    private array $userAgents = [
        'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...',
        'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15...',
        'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36...',
    ];

    public function fetch(string $url): ?string
    {
        $response = Http::withHeaders([
            'User-Agent'      => $this->userAgents[array_rand($this->userAgents)],
            'Accept-Language' => 'en-US,en;q=0.9',
            'Accept-Encoding' => 'gzip, deflate, br',
        ])
        ->timeout(15)
        ->retry(3, 2000, fn($e) => $e instanceof ConnectionException)
        ->get($url);

        if ($response->status() === 429) {
            // Rate limit — delay before retry
            sleep(rand(30, 60));
            return null;
        }

        if (!$response->successful()) {
            Log::warning("Scraper failed: {$url}", ['status' => $response->status()]);
            return null;
        }

        return $response->body();
    }
}

For JavaScript-rendered sites, use Playwright API or Browserless.io:

class PlaywrightScraper
{
    public function fetch(string $url): ?string
    {
        $response = Http::post(config('scraper.browserless_url') . '/content', [
            'url'     => $url,
            'waitFor' => '.price, [data-price], .product-price',  // wait for element
            'options' => ['timeout' => 20000],
        ]);

        return $response->successful() ? $response->body() : null;
    }
}

Price Parser

class PriceParser
{
    public function parse(string $html, MonitoredProduct $config): ?ParsedPrice
    {
        $crawler = new Symfony\Component\DomCrawler\Crawler($html);

        // Try CSS selector from config
        if ($config->selector) {
            try {
                $text = $crawler->filter($config->selector)->first()->text();
                return $this->extractPrice($text);
            } catch (\Exception $e) {
                // Selector didn't work — fallback to heuristics
            }
        }

        // Heuristic search by typical patterns
        $priceSelectors = [
            '[itemprop="price"]',
            '.price__current',
            '.product-price',
            '[data-price]',
            '.js-price',
        ];

        foreach ($priceSelectors as $selector) {
            try {
                $node = $crawler->filter($selector)->first();
                if ($node->count()) {
                    // Try data-price attribute first
                    $dataPrice = $node->attr('data-price') ?? $node->attr('content');
                    if ($dataPrice && is_numeric($dataPrice)) {
                        return new ParsedPrice(price: (float) $dataPrice, rawText: $dataPrice);
                    }
                    return $this->extractPrice($node->text());
                }
            } catch (\Exception $e) {
                continue;
            }
        }

        return null;
    }

    private function extractPrice(string $text): ?ParsedPrice
    {
        // Remove spaces, replace comma with period
        $normalized = preg_replace('/[^\d,.]/', '', $text);
        $normalized = str_replace(',', '.', $normalized);

        // Remove thousands separators: "29.990" → "29990" (if no decimals)
        if (preg_match('/^\d{1,3}[.]\d{3}$/', $normalized)) {
            $normalized = str_replace('.', '', $normalized);
        }

        if (!is_numeric($normalized) || (float)$normalized <= 0) {
            return null;
        }

        return new ParsedPrice(price: (float)$normalized, rawText: $text);
    }
}

Job for Checking Price

class CheckCompetitorPriceJob implements ShouldQueue
{
    public int $timeout = 60;

    public function handle(
        CompetitorScraper $scraper,
        PriceParser       $parser,
        PriceAlertService $alerts,
    ): void {
        $monitored = MonitoredProduct::findOrFail($this->monitoredId);

        $html   = $scraper->fetch($monitored->url);
        if (!$html) return;

        $parsed = $parser->parse($html, $monitored);
        if (!$parsed) {
            Log::warning("Price parse failed", ['url' => $monitored->url]);
            return;
        }

        PriceSnapshot::create([
            'monitored_id' => $monitored->id,
            'price'        => $parsed->price,
            'in_stock'     => $parsed->inStock,
            'raw_text'     => $parsed->rawText,
        ]);

        // Alert on significant change
        if ($monitored->last_price) {
            $changePct = abs($parsed->price - $monitored->last_price) / $monitored->last_price * 100;
            if ($changePct >= 5) {
                $alerts->priceChanged($monitored, $monitored->last_price, $parsed->price);
            }
        }

        $monitored->update([
            'last_price'      => $parsed->price,
            'last_checked_at' => now(),
        ]);
    }
}

Reports

Daily report is generated by schedule and sent to Telegram group:

class DailyPriceReportJob implements ShouldQueue
{
    public function handle(): void
    {
        $report = $this->buildReport();
        $this->telegram->sendMessage([
            'chat_id'    => config('telegram.price_alerts_chat'),
            'text'       => $this->formatMarkdown($report),
            'parse_mode' => 'Markdown',
        ]);
    }

    private function buildReport(): array
    {
        return MonitoredProduct::with(['ourProduct', 'competitor'])
            ->whereHas('snapshots', fn($q) => $q->where('captured_at', '>=', now()->subDay()))
            ->get()
            ->map(function ($m) {
                $yesterday = $m->snapshots()->where('captured_at', '>=', now()->subDays(2))
                    ->where('captured_at', '<', now()->subDay())->avg('price');
                $today = $m->last_price;

                return [
                    'product'    => $m->ourProduct->name,
                    'competitor' => $m->competitor->name,
                    'price_now'  => $today,
                    'price_was'  => $yesterday,
                    'delta_pct'  => $yesterday ? round(($today - $yesterday) / $yesterday * 100, 1) : null,
                    'our_price'  => $m->ourProduct->price,
                    'position'   => $today < $m->ourProduct->price ? 'cheaper' : 'expensive',
                ];
            })
            ->toArray();
    }
}

Check Schedule

// High-priority competitors — every 2 hours
$schedule->command('monitor:check --priority=high')->everyTwoHours();

// Others — every 6 hours
$schedule->command('monitor:check --priority=normal')->everySixHours();

// Report — 9:00 AM daily
$schedule->job(new DailyPriceReportJob)->dailyAt('09:00');

Timeline

• Scraper + PriceParser + data schema: 1–2 days
• Playwright adapter for JS sites: +1 day
• Job system + scheduling: 0.5 days
• Telegram + Email reports: 1 day
• Competitor list management UI in admin: 1 day

Total: 4–5 business days.