Competitor Price Scraper Development for Monitoring

We develop competitor price monitoring systems that solve a specific task: knowing when and by how much a competitor changed a price, before customers notice. Without monitoring automation, you risk missing a price drop on a key product. Manual checking of tens of thousands of items is unrealistic, and a one-time audit gives only a snapshot. Over our work, we have launched more than 30 such projects for e-commerce—from small stores to large marketplaces. Our experience guarantees stable operation of the scraper even when site structure changes. If you need a reliable tool, contact us—we will evaluate your project and offer the optimal solution.

Monitoring System Architecture

The key difference from a one-time scraper is product prioritization. Popular items should be checked every hour, the long tail of the catalog once a day. This reduces load on the source and speeds up reaction to important changes. The task queue is distributed by priority, allowing processing up to 10,000 products per minute without overload.

[Scheduler] ├── High priority queue (top products, every hour) └── Low priority queue (the rest, once a day) ↓ [Fetcher] → [Parser] → [Change Detector] → [Alert Engine] ↓ [price_history table] 

The Change Detector compares the new price with the last record in history. If changed—record in price_history and event in alert queue. If unchanged—only update last_checked_at to avoid bloating history. This approach allows storing up to 5 years of history with minimal data volume.

Why JSON-LD Is the Best Source for Price Parsing?

Prices on websites are represented differently. In HTML—CSS selector .product-price or attribute data-price. In Schema.org—reliable, does not break during redesign. Via XHR API—intercepting network requests with Playwright. Dynamically via JS after loading—needs headless browser. JSON-LD is the most stable source: many SEO-optimized stores add microdata for search bots. Error rate when parsing via HTML can reach 10%, while via JSON-LD it's less than 0.5%.

import * as cheerio from 'cheerio'; interface PriceData { price: number; priceSale?: number; currency: string; inStock: boolean; } function extractPriceFromJsonLd(html: string): PriceData | null { const $ = cheerio.load(html); for (const scriptEl of $('script[type="application/ld+json"]').toArray()) { try { const data = JSON.parse($(scriptEl).html() ?? '{}'); const product = data['@type'] === 'Product' ? data : (Array.isArray(data['@graph']) ? data['@graph'].find((n: { '@type': string }) => n['@type'] === 'Product') : null); if (product?.offers) { const offer = Array.isArray(product.offers) ? product.offers[0] : product.offers; return { price: parseFloat(offer.price), currency: offer.priceCurrency ?? 'USD', inStock: offer.availability?.includes('InStock') ?? true, }; } } catch { continue; } } return null; } 

Handling price formats in text: "$12–17", "$12.99", "€ 9,90"—normalization via regex. Store as DECIMAL(10,2) with separate currency field. Track three levels: price without discount (price_original), price with discount (price_sale), loyalty card price (often a third hidden price).

Change Detector Mechanism

CREATE TABLE monitored_products ( id SERIAL PRIMARY KEY, source VARCHAR(100) NOT NULL, external_id VARCHAR(255) NOT NULL, title TEXT, url TEXT NOT NULL, priority SMALLINT DEFAULT 5, -- 1=highest, 10=lowest check_interval INT DEFAULT 360, -- minutes last_checked_at TIMESTAMPTZ, UNIQUE(source, external_id) ); CREATE TABLE price_history ( id BIGSERIAL PRIMARY KEY, product_id INT REFERENCES monitored_products(id), price DECIMAL(10,2), price_original DECIMAL(10,2), in_stock BOOLEAN, currency VARCHAR(3) DEFAULT 'USD', recorded_at TIMESTAMPTZ DEFAULT NOW() ); CREATE INDEX ON price_history(product_id, recorded_at DESC); -- Fast access to current price without JOIN with history ALTER TABLE monitored_products ADD COLUMN current_price DECIMAL(10,2), ADD COLUMN current_in_stock BOOLEAN; 

async function processNewPrice( productId: number, newPrice: number, newInStock: boolean ): Promise<{ changed: boolean; delta?: number }> { const product = await db.monitoredProducts.findById(productId); const priceChanged = product.currentPrice !== newPrice; const stockChanged = product.currentInStock !== newInStock; if (!priceChanged && !stockChanged) { // Only update check time await db.monitoredProducts.update(productId, { lastCheckedAt: new Date() }); return { changed: false }; } // Record in history await db.priceHistory.create({ productId, price: newPrice, inStock: newInStock, recordedAt: new Date(), }); // Update current values await db.monitoredProducts.update(productId, { currentPrice: newPrice, currentInStock: newInStock, lastCheckedAt: new Date(), }); const delta = product.currentPrice ? ((newPrice - product.currentPrice) / product.currentPrice) * 100 : 0; return { changed: true, delta }; } 

Configuring Alerts and Thresholds

Configurable trigger rules:

• Price dropped more than X% (e.g., 5% or 10%)
• Price fell below your price for a similar product
• Product appeared or disappeared from stock
• Price changed for N+ competitors simultaneously (sign of market shift)
• Price reached historical minimum over the last 90 days

async function checkAlertRules(productId: number, delta: number): Promise<void> { const rules = await db.alertRules.findAll({ productId, active: true }); for (const rule of rules) { const triggered = (rule.type === 'price_drop_percent' && delta < -rule.threshold) || (rule.type === 'below_my_price' && await isPriceBelowMyPrice(productId)) || (rule.type === 'out_of_stock' && newInStock === false); if (triggered) { await sendAlert(rule, productId, delta); } } } 

Delivery: Telegram bot (instant via Bot API), email digest (once a day), webhook to price management system (for automatic reaction). Thresholds can be changed in real-time via dashboard.

What's Inside the Analytics Dashboard?

Minimum necessary screens:

Monitoring Table — all tracked products with current competitor price, your price, percentage difference, and trend (up/down arrow).

Price Chart — competitor price vs your price over selected period. Recharts LineChart with two lines and change markers.

Alert Feed — last 50 changes with filtering by source and change type.

Quick dashboard implementation — Metabase connected to PostgreSQL. Custom React interface with Recharts is needed if the dashboard is embedded into an existing assortment management system.

Development Process

1. Analytics — study target sites, determine product priorities, agree on alerts and dashboard.
2. Design — design scraper architecture, database, alert rules.
3. Implementation — write code, configure queues, integrations.
4. Testing — run on real data, check accuracy and stability.
5. Deployment and support — deploy on your server or cloud, hand over documentation, train.

What's Included

• Scraper source code with open documentation
• Dashboard and API access
• Operation manual
• 3-month stability guarantee
• Support for site structure changes (up to 5 adaptations per month)

Timelines and Scale

For 100k+ products with a year of history, ClickHouse instead of PostgreSQL for storing price_history: analytical queries (aggregation over period, finding minimum) work an order of magnitude faster on large volumes. PostgreSQL remains for operational data and configuration.

Contact us for a consultation—we will evaluate your project and offer the optimal solution. Get a reliable price scraper turnkey with a stability guarantee.