You visit a website, type "Samsung Galaxy S24", and it shows prices from 20 stores, sorts by cheapest, and plots a price history graph. We build such price aggregators—platforms that automatically collect prices, match products, and show users the best offers. Technically, this is a complex task: parsing heterogeneous sources (YML, API, HTML), data normalization, fuzzy matching, and continuous updates. Each step is full of pitfalls: from rate limiting to differences in product names. With over 10 years of experience, we have built fault-tolerant collection and matching systems processing up to 1 million products per day. A matching error—and users see incorrect prices or duplicates. Web scraping without control—blocks and penalties. We ensure data stability and accuracy through thoughtful architecture, helping our clients save up to 40% of their budget by eliminating manual checks.
How to organize data collection from different sources?
Data Sources
Product and price data comes in three ways:
- Price lists and feeds — the store provides a YML, XML, or CSV file with the current assortment. The most reliable source: structured data, official partnership, no risk of bans. Yandex.Market YML is the de facto standard for the Russian-language market.
- Partner APIs — some stores provide REST APIs. Documentation is often weak, request limits are strict. Per official Yandex documentation, partner links require prior approval.
- Web scraping — for stores without feeds. High risk: CAPTCHA, rate limiting, layout changes, IP blocking. Requires constant maintenance.
At the start of an aggregator, it is better to work only with feeds and APIs—they are more stable. We selectively add scraping for key sources.
Data Collector Architecture
Scheduler (Celery Beat / Laravel Scheduler) ↓ every N hours FeedFetcher workers (one per source) ↓ RawData storage (S3 or local FS) ↓ Parser workers (XML/CSV/JSON → normalized objects) ↓ Normalizer (unit conversion, text cleanup) ↓ Matcher (match against products in DB) ↓ PriceHistory (record in timeseries) ↓ ElasticsearchIndexer (update index) Task queue: Celery + Redis for Python stack, Laravel Horizon + Redis for PHP stack. Each feed is processed independently; an error in one source does not block others.
Parsing Yandex.Market YML
YML is an XML with a strict schema. Critical fields:
<offer id="12345" available="true"> <url>https://shop.example.com/product/12345</url> <price>4990</price> <currencyId>RUB</currencyId> <categoryId>14</categoryId> <name>Samsung Galaxy A55 128GB</name> <vendor>Samsung</vendor> <model>Galaxy A55</model> <barcode>8806095076783</barcode> <param name="Color">Blue</param> <param name="Memory">128 GB</param> </offer> The barcode (barcode) is the best key for matching. GTIN/EAN is unique for each product variation. If barcodes are present for most suppliers, matching becomes trivial. According to Yandex.Market documentation, barcode is a recommended element for precise matching.
Why is product matching the key stage?
This is the most complex part of the aggregator. The task: determine that Samsung Galaxy A55 128GB Blue from store A and Smartphone Samsung Galaxy A55 (SM-A556B) 128 Gb Blue from store B are the same product.
Deterministic Methods
- GTIN/EAN matching: if both products have a barcode—unambiguous match.
- Manufacturer part number (MPN): SM-A556B is unique within the brand.
- URL canonicalization: some stores include GTIN in the URL.
Fuzzy Matching
from rapidfuzz import fuzz def match_score(title_a: str, title_b: str, brand_a: str, brand_b: str) -> float: if brand_a.lower() != brand_b.lower(): return 0.0 title_similarity = fuzz.token_sort_ratio(title_a, title_b) return title_similarity / 100 Match threshold: 0.85+ considered automatic match, 0.65–0.85 sent for manual review, below—new product.
ML Approach
Product name embeddings (sentence-transformers, ruBERT) + cosine similarity. Significantly more accurate than fuzzy, especially for different phrasings of the same product. The model is trained on historically confirmed matches.
Storage structure:
canonical_products (id, gtin, mpn, brand, name, category_id, attrs JSONB) source_offers (id, source_id, external_id, canonical_product_id, price, url, in_stock, updated_at) match_candidates (offer_id, canonical_id, score, status) -- pending | approved | rejected | Method | Accuracy | Speed | Implementation Complexity |
|---|---|---|---|
| Deterministic (GTIN) | 100% | High | Low |
| Fuzzy | 80–90% | Medium | Medium |
| ML (embeddings) | 95%+ | Low (requires GPU) | High |
How are prices updated and history stored?
Price History
The core value of an aggregator is not only the current price but also the change history. Each price change is recorded, not overwritten.
price_history ( id BIGSERIAL, source_offer_id BIGINT, price NUMERIC(12,2), in_stock BOOLEAN, recorded_at TIMESTAMPTZ DEFAULT NOW() ) For timeseries storage we use TimescaleDB—a PostgreSQL extension that partitions the table by time. Alternatives: InfluxDB or ClickHouse for high loads (up to 10,000 inserts per second).
A price history chart is a standard component on the product page. We use Chart.js or Recharts, aggregating data by day: SELECT date_trunc('day', recorded_at), min(price) FROM price_history WHERE ....
Update Frequency
| Source Type | Update Interval |
|---|---|
| Large store YML feed | Every 2–4 hours |
| Rate-limited API | 1–6 times per day |
| Scraped page | 1–2 times per day |
| Real-time API (rare) | On change via webhook |
When a price changes—page cache invalidation and recalculation of minimum price in the index.
How to set up data collection in 4 steps
- Connect sources: obtain YML feeds from stores or set up API access. One source = one config.
- Configure parsing: for feeds—use ready YML/XML parser; for APIs—write an adapter per documentation.
- Run matching: define rule set—GTIN priority, then fuzzy, then ML. Start with manual validation.
- Monitor and update: set up Celery or Horizon scheduler, add alerts for source failures.
What's included in aggregator development
As a result, you receive:
- Documentation for source integration (format descriptions, examples).
- Deployed infrastructure (Docker, CI/CD, monitoring).
- Admin panel for editing matching and viewing statistics.
- Training of the client's team on system operation.
- Technical support for 3 months after launch.
- Guarantee on matching correctness and timely price updates.
Want to discuss your project? Contact us for a free estimate. Get an engineer's consultation with no obligations.
Our experience
Over 10 years on the market, 40+ implemented projects. We don't just write code—we design architecture that doesn't fall under load and scales easily. We'll evaluate your project for free. Get in touch—we'll propose architecture, timelines, and turnkey pricing.