Competitor Price Monitoring Parser Development

Development of Competitor Price Monitoring Parser

Price monitoring solves one specific task: know when and by how much a competitor changed price before customers notice. For this, you need not just a parser, but a system with change history, analytics, and real-time alerts.

System Architecture for Monitoring

Key difference from one-time parser—product prioritization. Popular positions need hourly checks, long tail of catalog—once daily. This reduces source load and speeds up reaction to important changes.

[Scheduler]
  ├── High priority queue (top products, hourly)
  └── Low priority queue  (others, daily)
        ↓
[Fetcher] → [Parser] → [Change Detector] → [Alert Engine]
                              ↓
                       [price_history table]

Change Detector compares new price with last history entry. On change—record in price_history and event in alert queue. Without change—only update last_checked_at to avoid inflating history.

Price Parsing: Technical Details

Prices on sites are represented differently:

• In HTML (simple case)—CSS selector .product-price or attribute data-price
• In JSON-LD (schema.org Product)—reliable, doesn't break on redesign
• Via XHR API—intercept network requests via Playwright
• Dynamically via JS after load—need headless browser

JSON-LD is the most stable source for price scraping. Many SEO-optimized shops add schema.org markup for search bots:

import * as cheerio from 'cheerio';

interface PriceData {
  price: number;
  priceSale?: number;
  currency: string;
  inStock: boolean;
}

function extractPriceFromJsonLd(html: string): PriceData | null {
  const $ = cheerio.load(html);

  for (const scriptEl of $('script[type="application/ld+json"]').toArray()) {
    try {
      const data = JSON.parse($(scriptEl).html() ?? '{}');
      const product = data['@type'] === 'Product' ? data :
        (Array.isArray(data['@graph'])
          ? data['@graph'].find((n: { '@type': string }) => n['@type'] === 'Product')
          : null);

      if (product?.offers) {
        const offer = Array.isArray(product.offers) ? product.offers[0] : product.offers;
        return {
          price: parseFloat(offer.price),
          currency: offer.priceCurrency ?? 'RUB',
          inStock: offer.availability?.includes('InStock') ?? true,
        };
      }
    } catch { continue; }
  }

  return null;
}

Handle text price formats: "1 299,00 ₽", "$12.99", "€ 9,90"—normalize via regex. Store as DECIMAL(10,2) with separate currency field. Track three levels: original price (price_original), sale price (price_sale), loyalty card price (often hidden third price).

Database

CREATE TABLE monitored_products (
  id              SERIAL PRIMARY KEY,
  source          VARCHAR(100) NOT NULL,
  external_id     VARCHAR(255) NOT NULL,
  title           TEXT,
  url             TEXT NOT NULL,
  priority        SMALLINT DEFAULT 5,  -- 1=highest, 10=lowest
  check_interval  INT DEFAULT 360,     -- minutes
  last_checked_at TIMESTAMPTZ,
  UNIQUE(source, external_id)
);

CREATE TABLE price_history (
  id             BIGSERIAL PRIMARY KEY,
  product_id     INT REFERENCES monitored_products(id),
  price          DECIMAL(10,2),
  price_original DECIMAL(10,2),
  in_stock       BOOLEAN,
  currency       VARCHAR(3) DEFAULT 'RUB',
  recorded_at    TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX ON price_history(product_id, recorded_at DESC);

-- Quick access to current price without JOIN with history
ALTER TABLE monitored_products
  ADD COLUMN current_price DECIMAL(10,2),
  ADD COLUMN current_in_stock BOOLEAN;

Change Detector

async function processNewPrice(
  productId: number,
  newPrice: number,
  newInStock: boolean
): Promise<{ changed: boolean; delta?: number }> {
  const product = await db.monitoredProducts.findById(productId);

  const priceChanged = product.currentPrice !== newPrice;
  const stockChanged = product.currentInStock !== newInStock;

  if (!priceChanged && !stockChanged) {
    // Only update check time
    await db.monitoredProducts.update(productId, { lastCheckedAt: new Date() });
    return { changed: false };
  }

  // Record in history
  await db.priceHistory.create({
    productId,
    price: newPrice,
    inStock: newInStock,
    recordedAt: new Date(),
  });

  // Update current values
  await db.monitoredProducts.update(productId, {
    currentPrice: newPrice,
    currentInStock: newInStock,
    lastCheckedAt: new Date(),
  });

  const delta = product.currentPrice
    ? ((newPrice - product.currentPrice) / product.currentPrice) * 100
    : 0;

  return { changed: true, delta };
}

Alerts and Thresholds

Configurable trigger rules:

• Price dropped more than X% (e.g., 5% or 10%)
• Price dropped below your price for similar product
• Product appeared or disappeared from stock
• Price changed at N+ competitors simultaneously (market shift sign)
• Price reached historical minimum in last 90 days

async function checkAlertRules(productId: number, delta: number): Promise<void> {
  const rules = await db.alertRules.findAll({ productId, active: true });

  for (const rule of rules) {
    const triggered =
      (rule.type === 'price_drop_percent' && delta < -rule.threshold) ||
      (rule.type === 'below_my_price' && await isPriceBelowMyPrice(productId)) ||
      (rule.type === 'out_of_stock' && newInStock === false);

    if (triggered) {
      await sendAlert(rule, productId, delta);
    }
  }
}

Delivery: Telegram bot (instant via Bot API), email digest (once daily), webhook to price management system (for automatic reaction).

Analytics Dashboard

Minimally necessary screens:

Monitoring table—all tracked products with competitor price, your price, difference %, and trend (up/down arrow).

Price chart—competitor price vs your price for selected period. Recharts LineChart with two lines and change markers.

Alert feed—last 50 changes with filtering by source and change type.

Quick dashboard—Metabase connected to PostgreSQL. Custom React interface with Recharts if dashboard is embedded in existing assortment management system.

Timeline and Scale

For 100k+ products with year history—ClickHouse instead of PostgreSQL for price_history storage: analytics queries (aggregation over period, minimum search) on large volumes much faster. PostgreSQL remains for operational data and configuration.