Development of a bot parser for competitor product availability

Product Availability Parser: Where Data Collection Breaks

A product availability parser solves one task — regularly delivering an up-to-date picture of someone else's catalog: what's in stock, what's gone, what's back. As long as the sample is clean, the buyer works from facts.

As soon as a mismatch appears in the chain "source → parsing → storage → report," the data becomes more dangerous than its absence: decisions are made based on it.

The most common failure is outdated stock in the catalog. The export is pulled in full and overwritten once a day, although the source updates more often; some pages are served from cache, and availability data collection records yesterday's state as current. The buyer sees "in stock," but the item was sold out that morning.

The second class is empty cards after markup changes. Parsing relies on specific page elements: when the price or stock block is redesigned, the node disappears, and the collector silently returns "out of stock" instead of a parsing error. The report looks valid, but half the catalog in it is false zeros.

The third is a discrepancy between warehouse and storefront. The product is in the accounting system (1C) but not published, or vice versa: the card is open, but the stock is not posted.

Without reconciling these two sources, competitor catalog monitoring shows what the customer sees, not what the supplier actually sells.

Then manual work kicks in: the buyer opens dozens of pages and rechecks disputed items because there is no trust in the export. Let's break down how this is closed — from freshness control and parsing integrity checks to discrepancy signals.

What Continuous Collection of Availability and Price Data Provides

While checks are done by hand, a window of a day or two remains between catalog crawls, and a stockout lives in it unnoticed.

The buyer spends time on mechanical crawling instead of assortment decisions, and the discrepancy between manual tables and reality accumulates until the next reconciliation.

Continuous collection removes most of this work: the share of manual checks drops to handling exceptions — usually a few percent of the previous volume.

The rest is handled by scheduled price and availability monitoring: unchanged items don't distract, and changes immediately enter the report with a timestamp.

The speed of reaction to an item going out of stock no longer depends on when the buyer gets to the card.

The signal arrives at the moment of change, so the number of missed stockouts drops to zero, and a backup sourcing decision is made the same day, not after the supplier has sold out the remaining stock.

A separate class of errors disappears entirely — an outdated price in the purchasing sheet, a forgotten card, two versions of the same table.

The change history shows when an item went out and came back, how long the stockout lasted, and how the price behaved among competitors: this is a basis for negotiations and for adjusting the assortment, not a guess.

This is exactly how procurement automation changes the human role: they handle deviations rather than search for them.

Data Collection Methods: Comparison and Limitations

The choice of collection method determines not only launch speed but also how much time will go into maintenance six months later.

Let's break down four options that are realistically chosen between: exchanging data with a website through a programmatic interface, crawling catalog pages, a ready-made export from the supplier, and a third-party monitoring service.

In practice, methods are almost always combined: where there is an interface, we work through it, the rest is covered by catalog crawling, and the supplier export goes as a backup channel in case of unavailability.

Data from all sources is brought to a single schema: SKU, platform, stock, measurement time. Then a failure of one channel doesn't bring down collection, and discrepancies between sources are visible immediately.

Development Stages: From Task Analysis to Launch

Our bot development stages follow a fixed order: first we agree on what exactly counts as availability, then we collect data, and only then put it on a schedule. This removes the main risk — a bot that formally works but shows something other than what procurement needs.

1. Data source audit. We analyze each website: whether there is an open programmatic interface or only card markup, how pagination works, whether crawling regional subdomains is needed. The output is a list of sources with update frequency and product matching rules.
2. Collection schema and matching key. We take a stable identifier — SKU or barcode — so that renaming a card doesn't produce a false "out of stock." We describe behavior on markup changes in advance: an error signal instead of a silent empty export.
3. Building the core on one or two sources. We test storage and price comparison on simple websites before connecting the rest. This way logic errors surface before a dozen sources appear in the schema.
4. Update schedule. We select the interval for the task, spread runs over time, limit the number of simultaneous requests, and set retries on network failures so that other people's servers don't see us as load.
5. Test run. We compare the result with a manual sample across several dozen cards, calculate the share of discrepancies, and see what the bot does with items removed from sale or hidden from the catalog.
6. Acceptance. We check on a real dataset: completeness for each source, data freshness by the time of the report, number of false positives. We agree on criteria before launch so that acceptance is a formality, not an argument.
7. Handover and training. We provide configurations, documentation, and instructions for connecting a new source, and show how to read the error log. After that, the customer's team runs the bot update cycle themselves.

Delivery Package: Code, Schemas, Update Regulations

After handover, the customer is left not with a service on our side but with a full package: code, bot documentation, and regulations by which the system can be developed in-house or handed to another team without reverse engineering. We fix the list in the acceptance act — each item is checked during acceptance.

• Source code — a repository with change history, divided into modules: source crawling, page parsing, product matching, notification sending.

• Data collection schema — a list of sources and fields for each, crawl frequency, signs of product absence from stock, pause rules under source-side restrictions.

• Matching rules — by what attributes a product is considered the same across different sellers, how duplicates and renames are handled, an exceptions table.

• Bot documentation — instructions for launching from scratch, environment composition, where access keys are stored and how to change them without editing code.

• Update regulation — frequency of source revision, procedure for connecting a new store, actions when page markup changes at the source.

• Access handover — accounts, roles, key rotation procedure when an employee leaves, who each access is assigned to.

• History storage — retention depth for availability and price snapshots, cleanup schedule, export format for analytics.

• Support regulation — response time to collection failure, recovery procedure, and control under which bot silence is not read as "item is in stock."

Case: Assortment Control in a Retail Chain

A retail chain with an assortment of several thousand items kept control of competitors' availability manually: category managers went through cards on marketplaces and supplier websites, checked stock in a shared table, and made edits by hand.

A full reconciliation took a team of three people almost two working days, so changes were detected with a delay — after the item had already gone out of stock, and the purchasing decision was made blindly.

What Was Changed

We built a parser that on a schedule crawls a given list of sources — from supplier websites to Yandex Market — and compares availability and price with the previous snapshot. We store state for each item: latest value, change time, source.

We describe markup with selector configuration separately from crawl logic, so a layout change on the source side doesn't bring down collection — one description is fixed, without code changes.

Retries of failed requests use increasing delay, and discrepancies between sources are marked as suspicious rather than as an availability change.

What Changed

Reconciliation takes 15–20 minutes instead of two days, and the summary arrives before the shift opens. The share of changes caught in the first 24 hours grew from 40% to 95% — procurement reacts while the competitor hasn't yet sold out its warehouse.

Manual operations decreased severalfold: the manager handles only suspicious discrepancies, not the whole list. When markup changed twice on one source, collection was restored in 20 minutes by fixing selectors, without rewriting the logic.

How Does Collection via a Programmatic Interface Differ from Page Crawling?

If a store has a programmatic interface, availability comes in ready fields: stock by warehouse, "in stock" or "on order" status, delivery time.

We pull dozens and hundreds of items in one request and get a set of values that doesn't depend on how the card is rendered on the website at that moment. Collection speed and stability are maximal here, as long as the agreement with the source doesn't change.

Page crawling is used where there is no interface or it doesn't provide everything. Catalog markup is a one-sided agreement: a change in site structure, a card template change, or moving the availability block breaks the sample.

Therefore collection stability here rests on rules tied to the meaning of the block and to neighboring elements, not to specific class names, and on regular comparison with reference cards.

A programmatic interface or page crawling is not an either-or choice. More often the main channel is the interface, and crawling works as insurance: it confirms availability at points of sale that the interface doesn't cover and provides backup when responses arrive with a delay.

It works the other way too: without an interface, crawling covers everything, and the interface is connected later if the store opens it.

The difference is visible in support. An interface changes rarely but breaks entirely — you need response format control and deviation alerts. Markup changes often but pointwise — you need reference histories and an automatic signal when the share of empty fields grows. The choice is dictated by coverage of the required fields, not by development convenience.

Concerns About Load, Blocks, and Support

Collecting data from someone else's catalog is a matter of pace and rules: we set request frequency, number of simultaneous sessions, and page list in settings, rather than leaving it to chance.

We keep the request flow below what a normal search robot generates, so the load on a competitor's website remains unnoticeable to its owner.

Below are the concerns most often voiced before the start: request blocking, legal boundaries of collection, false positives, and post-launch support.

What are the legal boundaries of data collection?

We work only with open information about availability and price — without authorization, personal accounts, or personal data. For each snapshot we save the source address and acquisition time: this is the basis for reconciliation and analysis of disputed cases.

How do you filter out false positives?

A change is confirmed by a repeat request after a few minutes and comparison with the previous snapshot. If availability returns to the original state, the event is marked as noise and doesn't go into the notification — only stable changes enter the report.

Who is responsible for operation after launch?

Markup on third-party pages changes, so after handover a regulation remains: monitoring the share of successful snapshots, alerts about drops and structure changes, scheduled updates of parsing rules. We provide field documentation and the procedure for independent launch.

Let's Discuss the Task: Collection Schema and Launch Plan

To come back with an estimate and plan rather than clarifying questions, a minimum of source data is needed. Below is what to send for task discussion: this is enough to calculate the scope of work, build a collection schema, and give timelines.

• Source list — platforms and sections where availability needs to be tracked: catalog addresses, categories, product pages.
• Product lists — SKUs, names, or card links by which we reconcile items; if there are several lists, indicate how they differ.
• Required fields — what exactly we record in the export: availability status, stock, delivery time, card address, measurement time.
• Update frequency — once a day, once an hour, or by event; the crawl schema and history storage volume depend on this.
• Export format — table, file, or record into your accounting system, and whether change history is needed and for what depth.
• Access and restrictions — where authorization is required, whether there are limits on the number of requests, and whether work through your proxy perimeter is required.
• Reaction rules — to whom and how to report a change: email, messenger notification, or a separate summary.

With this set, we come back with a work plan: collection schema for each source, stages, team composition, procedure for checking data on a test sample, and post-launch support terms. Describe the task — we'll send an estimate and plan.