Why price comparison breaks on real catalogs
Shops and marketplaces provide catalogs in different forms: CSV exports, XML price lists, JSON via API, and some sources have to be parsed directly from page markup.
For the same product in two catalogs, word order in the name, delimiters in SKUs, units of measure, and set of attributes differ.
Exact string matching on heterogeneous catalogs misses in tens of percent of cases, and that is before accounting for the human factor in the data itself.
Then a matching error turns into a wrong price in the card: different models merge into one item, a case sits next to a phone, and the selection keeps offers from stores that no longer sell the product.
The user does not figure out whose failure it is — they see that the number did not match reality and leave for a competitor.
A separate story is outdated exports. Sources update at different frequencies: one provides data hourly, another once a day, a third only on request.
While the cache and update schedule do not account for this spread, during a sale the selection contains a two-hour-old price, next to an already changed one: price comparison loses meaning at the first mismatch with the checkout.
Product data quality finishes it off: empty attributes, different designations of color and volume, brand sometimes in the name, sometimes in a separate field.
Therefore matching is built not on one string but on a set of features — SKU and global identifier, brand, key attributes, name similarity — and normalization rules and match thresholds are configured for a specific catalog.
Below we break down how this is covered: from receiving exports to merge rules and freshness control.
What accurate price comparison delivers: coverage, freshness, trust
Price comparison works not when the catalog has many stores, but when for each item you can verify three values: the share of products with a correct price, the time from a price change at the source to its appearance in the app, and the number of support requests about “price did not match.” These three metrics we include in acceptance — otherwise “comparison” rests on feeling, not verifiable data.
Catalog coverage and the share of discrepancies are calculated by reconciliation: the source export is compared with what the buyer sees, and discrepancies are broken down by category and store.
This shows where the failure is systemic — outdated price list format, mismatch of units or bundle contents — and where it is a one-off error. Popular categories are checked more often, the long tail less often: effort goes where price actually influences choice.
Price freshness rests on the update schedule. Fast categories are polled frequently and in small batches, slow ones less often, on schedule; instead of a full catalog reload, only changed items are updated, so price reaches the app in minutes, not days.
Behavior when the source is unavailable is configured separately: the last confirmed price is preserved with a timestamp, not replaced by a guess.
Support requests are the most honest metric: a buyer writes when they saw one thing in the app and another at the checkout.
The reduction comes not only from data accuracy but also from screen behavior on discrepancy: a “price confirmed on date” label, a short explanation of the cause, and one-tap feedback.
This turns a complaint into a signal for reconciliation, and the share of discrepancies becomes a manageable value, not background noise.
Data acquisition options: regular exports and storefront scraping
A price comparison catalog is always a set of heterogeneous sources. Stores provide price lists as files on schedule, affiliate programs as a stream via API, and some sellers are available only through a public storefront.
The chosen mode affects price freshness, load on sources, and the amount of manual work with discrepancies in product matching.
| Option | When it fits | Limitations |
|---|---|---|
| Regular exports from stores (CSV, XML, YML on schedule) | The store is ready to provide the full catalog, prices change no more than several times a day | Delay between updates; different set of fields and encodings; normalization and matching by barcode and name required |
| Affiliate channels | There is an agreement with a chain or aggregator, accurate stock and prices are needed | Source limits on request count; dependence on one data provider; break when terms change |
| Storefront scraping | The seller has neither an export nor affiliate access, but the catalog is open | Fragility to layout changes, anti-bot protections, legal restrictions, noticeable load on the source |
| Hybrid scheme | The market is heterogeneous: some sources provide files, some are available only via storefront | Harder operation — a unified normalization layer and duplicate control between channels are required |
A hybrid scheme almost always wins on coverage: exports give accuracy and cheap updates, affiliate channels give real-time stock, and storefront scraping covers the long tail of stores.
The key here is a unified product model and a common data ingestion pipeline: then a source can be added, rate-limited, or disabled without rebuilding the app.
Work stages: from data analysis to launch and handover
The order of stages is arranged so data reconciles before the first app screen appears. First we analyze sources and agree on which attribute makes a product the same, then we build matching, and only then we enable updates.
An error in the data model here costs hours of work; in a finished app, it costs rework of logic and interface.
- Source analysis. We look at how stores provide catalogs: exports, open APIs, page markup. We record update frequency, availability of SKU and barcode, and load limits on the source.
- Matching schema. We design rules for consolidating a product into one card: exact identifiers, then the “brand + model” link, then name parsing. We describe categories and units separately — otherwise price comparison produces false pairs.
- Setting up updates. Prices are recalculated more often, the catalog less often. We configure the request queue, retries when the source is unavailable, and price history accumulation — without it, dynamics cannot be shown.
- Testing on real data. We run the stream for a period and compare: share of matched cards, false merges, missed items. Acceptance is on live categories, not a prepared sample.
- Launch. We bring updates into the production loop, enable alerts for source outages and slow responses, and check app behavior with incomplete data.
- Project handover. We provide documentation on sources and rules, update regulations, access, and train the customer’s team to work with the markup.
Each stage is closed with a verifiable artifact: source description, matching rules, regulations, reconciliation report — work is accepted by fact, not by feeling.
Deliverables: data schemas, update regulations, access
Delivery is closed not by the release date but by a package of artifacts that lets the app be developed and maintained without us. Below is what remains with your team along with the source code.
- Data schema. Entities of product, store, offer, and price history, their relationships and keys — the schema immediately shows where the source of truth for each card is stored and how the output for comparison is built.
- Source description. List of price ingestion channels (scheduled export, file, store’s open API), their format, frequency, and limits on the number of requests.
- Matching rules. Normalization of names, models, and attributes, match thresholds, and the queue for manual review of disputed pairs — so the same product does not split into multiple cards.
- Update regulations. Schedule for price rebuilds, actions when the source is unavailable, and the rule by which an outdated offer is hidden rather than shown with the previous price.
- Access handover. Service and dashboard accounts, integration keys, permissions to publish builds, contacts of responsible parties on the store side.
- Internal documentation. Service map, deployment scheme, release rollback procedure, and typical support operations with expected result description.
- Collection metrics. Dashboard with the share of successfully ingested sources, percentage of matched items, and delay between a price change and its display to the user.
Such a kit allows the project to be handed to another team or run in-house: data boundaries, rules, and responsibilities are described, not left in chat.
Case study: reduced data discrepancies and update time
Where it broke before
A price comparison service collected offers from several hundred stores: price exports arrived in different formats, and item matching was by name string.
“Smartphone X 128 GB black” and “Smartphone X, 128GB, Black” remained different products, while neighboring generations of the same line were merged.
The share of incorrect items in the output stayed at 12–14%, the price on the storefront lagged behind the store site by tens of minutes, and support handled a stream of identical requests “your price is wrong.”
What we changed and what resulted
We moved matching to a stable key: manufacturer SKU and barcode where available, plus normalization of name and attributes for other cards.
Reconciliation with the reference catalog Google Merchant Center was made regular, and discrepancies visible: a pair receives a confidence score, and disputed matches go to manual review rather than output.
Price updates were moved to processing a stream of changes — only affected items and their neighbors in comparison are recalculated, not the entire catalog.
The share of incorrect items fell to 0.6–0.9%, and reaction time to a price change decreased from tens of minutes to 1–3 minutes on the hot stream.
Support load dropped by about six times: manual catalog edits were reduced to resolving ambiguous cards, and typical requests closed themselves.
Before/after metrics are placed on the monitoring dashboard, so matching degradation is visible on the day it appears, not via complaints — regression is caught before the buyer notices it.
What to do if sources are already connected but data disagrees?
Discrepancies almost never live in the app — they live in the product matching rules between sources. If catalogs are connected and prices for the same product diverge, then normalization of name, unit of measure, SKU, or currency did not match. Rewriting the app is pointless — new code will inherit the same errors.
An audit starts with a sample, not code. We export discrepancies for the period and break them into classes: the product did not find a pair, found a false pair, matched by name but diverged in attributes.
The share of each class sets the order of work — usually one or two classes produce the bulk of the noise.
Then rules are moved into a matching layer separate from the output logic.
The primary key is barcode or manufacturer SKU; the fallback rule is brand plus normalized name with a similarity threshold; disputed pairs go to a manual confirmation table that the catalog update does not overwrite.
Units of measure and weight are recorded separately: price per kilogram or liter diverges most often — this also happens in exports like Google Shopping.
Phased refinement is cheaper than a full rework. The normalization layer changes step by step, the output contract stays the same, so app stability does not suffer: price comparison, favorites, and notifications work on the same data.
The effect is measurable after each stage — the share of matched items grows, and false pairs decrease.
The readiness criterion is not “no discrepancies” but a manageable share and clear rollback: a rule that worsened the output is removed by configuration without a release. Describe which sources are connected and where they diverge — we will return with an audit plan.
Common technical concerns: matching and data freshness
Product matching is where trust in the output most often collapses: the same product comes from sources with different names, SKUs, and units. Therefore we compare not strings but a set of features — model, attributes, category — and reduce them to a canonical card.
Below are questions usually asked at acceptance: source limits, behavior without a network connection, adding new data providers.
How are products matched if sources have no common SKUs?
A canonical product and rules for reducing to it sit between the source and the output. A match is confirmed by several features at once, and disputed pairs go to review, not results. The trigger threshold is configured by category: strict for electronics, softer for clothing.
What does a person see without a network connection?
The last received output and prices are stored locally, and the screen remains functional. When the connection returns, data updates, and outdated values are marked — it is visible that the price could have changed. This way the list does not become empty in the subway or an underground parking garage.
How is a new data provider connected?
The source is added as a separate module: we describe its output structure, bring fields to the common schema, and include it in the common stream. Other sources are not touched.
If the provider changes the output structure, edits remain inside its module, and the app continues working.
Who is responsible for price accuracy?
We are responsible for receiving, matching, and displaying data; the source is responsible for what it provides. Next to the price, it is visible when it was updated and where it came from, and discrepancies are recorded in the log. This removes the dispute “the app showed a wrong price” before a support request.
Let’s discuss the task: what’s needed for an estimate and work plan
To return with a timeline estimate and work plan, rather than general wording, we need inputs on the product. Below is the minimum that removes the main uncertainty; we will clarify the rest in conversation and record it in the technical description.
- Price sources: affiliate store APIs, catalog exports, open pages — what is available and on what terms.
- Geography and chains: countries and cities, whether regional prices are needed and in what currency we display the price.
- Product matching: whether barcodes and supplier SKUs are available, by which attributes we consider two offers one product.
- User scenarios: search, comparison list, price history, price drop notification, favorites, account login.
- Load: how many products and stores, how often prices are recalculated, how fresh data is needed.
- Access and rules: agreements with stores, restrictions on use of their data, personal data requirements.
- Analytics and publishing: what we consider the target action, in which app stores we publish, and who owns the accounts.
Based on these inputs, we return with a work plan: stages, team composition — mobile development, backend, data normalization, testing — and a timeline estimate with release order. Describe the task and send what you already have: we will return with an estimate and plan.