Why a marketplace recommendation system loses accuracy
A classic failure of a mature marketplace is that recommendations collapse into bestsellers.
The model shows what is already being bought, gets clicks, and confirms its own hypothesis, while the long tail of the catalog sinks: a product without impressions never accumulates statistics, and without statistics it never becomes a candidate.
A new item with good margin never reaches working positions—even though formally the algorithm is trained and reporting metrics look fine.
The second reason is fragmented behavioral data. Views, add-to-cart events, search queries, cancellations, and offline sales live in different systems with incompatible event schemas, and some visits are not stitched into a single profile.
The model sees fragments of sessions and cannot reconstruct intent: a user who searched for a specific model gets a selection based on general popularity.
The third reason is declining recommendation accuracy as the catalog grows.
Offline validation on an old sample shows stable quality, but in production the share of queries without a relevant answer grows and coverage falls: impressions concentrate on a few percent of the assortment.
Scheduled retraining cannot keep up with the arrival of new products, and usually only clicks are monitored—without diversity, novelty, or the share of the catalog receiving impressions.
All three failures are treated not by tweaking weights but by rebuilding the loop: stitching profiles, training with novelty in mind, and separate coverage monitoring. Let’s break down how this is addressed.
What changes after implementation: conversion, coverage, repeat orders
The effect of a recommendation system is measured by three counters: the share of orders from the recommendation block, coverage of products outside the top sellers, and the share of repeat orders.
We output them in a separate report and compare against a control group of users—otherwise seasonal growth can easily be mistaken for the model’s performance.
Long-tail catalog coverage grows because candidates are selected from the entire assortment, not from the first pages of results. Ranking is then trimmed by business rules: stock on hand, delivery time, share of new items, stop lists. Low-demand positions receive impressions where they are relevant and stop being dead weight.
Controllable recommendations are not only personalization but also quotas: how many impressions a new seller gets, which positions we promote, which we exclude.
We keep ranking on CatBoost—the model reproducibly retrains on fresh data, and quality does not depend on which engineer assembled the storefront.
Repeat purchases grow differently—through the buyer’s own history: recurring categories, intervals between orders, behavior of similar users.
Here it is important not to push toward a purchase with intrusive blocks, but to return what is relevant: a buyer who came for consumables or a refill places an order without additional traffic.
The growth in conversion to order is visible in the first weeks, while long-tail coverage and customer returns are visible over months. That is why we measure in stages and record exactly which recommendation block produced the shift.
Approaches to building recommendations: comparison of options
The approach is chosen not by how “modern” the model is, but by the data: how many active profiles there are, whether there is order history, how quickly interest in a product fades.
We compare ranking approaches along three axes—quality on cold objects, controllability of recommendations, and maintenance effort.
| Option | When it fits | Limitations |
|---|---|---|
| Manual selection rules | Catalog up to several thousand items, little behavioral data—predictable control over recommendations is needed | Rules become more numerous than products; they conflict with each other and do not account for the user profile |
| Models on behavioral data | There is a history of views, carts, and orders sufficient for training | Cold start for new products and users, impression bias, regular quality control needed |
| Hybrid recommendation scheme | Both personalization and business priorities are needed: availability, promotions, margin | Two loops—rules and model—must be reconciled, tested, and explained to the team |
A pure model rarely goes into production as is: it forms a pool of candidates, and rules fill in the constraints—remove out-of-stock positions, hold back promotions, even out impressions for new items.
Offline metrics on a holdout sample filter out weak options, but the decision is made by comparison on live traffic: conversion of the recommendation block, share of impressions for new products, speed of reaction to fresh events.
Such a hybrid provides both personalization and the predictability that the commercial team demands.
Implementation stages: from data audit to handover to operations
Implementing recommendations is not about plugging in a ready-made library, but a sequence of steps where each next step relies on the result of the previous one.
If you skip the data audit and go straight to the algorithm, it will learn from incomplete events and produce random suggestions. The order of work is as follows.
-
Data audit. We examine which user actions are recorded: views, search, add-to-cart, purchases, returns. We check completeness, delivery latency, and the share of noise—the output is a list of signals suitable for training.
-
Event flow diagram. We describe where events enter the storage, where they are lost and duplicated. The diagram shows which gaps must be closed before starting recommendations; otherwise recommendations will be built on an outdated picture.
-
Approach selection. We determine what will produce the effect: models based on the behavior of similar users, content features of the product, recommendation rules, or a combination. We make the decision based on the volume of history and purchase density, not on fashion.
-
Implementation of recommendations. We assemble training and serving, connect the block to the catalog, product page, and cart, and set up recalculation and logging.
Recommendations must be assembled in fractions of a second; otherwise the recommendation block will slow down the page.
-
Validation on a control group. We split traffic: some users see the new recommendations, some see the previous ones. We compare behavior and browsing depth, and roll back changes if the uplift is not confirmed.
-
Handover to operations. We hand over the code, configurations, flow diagram, and retraining procedure. The client’s team changes parameters and updates the model without our involvement.
What we hand over: configurations, flow diagrams, procedures
Handover is not an archive of files but a working system: the client’s engineers reproduce recommendations from our artifacts, change rules, and launch retraining themselves.
Each document answers a specific question—what to configure, where the data comes from, who has access, and at which metrics a release is considered safe.
- Recommendation configurations: weights and rules for blending signals, the balance of relevance, popularity, and diversity, and restrictions on repeats in the feed.
- Data flow diagrams: the path from view and cart events to the feature store and recalculation of product vector representations, indicating latencies and bottlenecks.
- Feature selection rules: the list of signals used, aggregation windows, and separate cold-start logic for new products and sellers.
- Model documentation: metrics on a holdout sample, applicability boundaries, known weak segments, and the reasons they are weak.
- Update procedure: frequency and triggers for retraining, the approval procedure for rollout, and the set of metrics under which a new version is automatically rejected.
- Data access policies: roles, anonymization of user events, segmentation of rights, and logging of access to personal fields.
- Monitoring and rollback procedure: control of distribution shift and recommendation degradation, and the procedure for reverting to the previous version in one operation.
- Ownership matrix: which changes the client’s team makes independently, which require our engineers, and by what procedure.
The logic is simple: every item on the list should be such that six months later, without us, it can be used not from memory but from the document.
Case study: share of orders from the recommendation block on a marketplace tripled
A catalog with several hundred thousand items, and in the recommendation block on the first screen there was almost the same top sellers by category.
New items settled at the bottom of the recommendations, personal differences between buyers almost did not affect impressions, and the share of orders from the recommendation block stayed around 8%.
| Metric | Before | After |
|---|---|---|
| Catalog coverage in recommendations | 24% of items | 71% |
| Share of orders from the recommendation block | 8% | 25% |
| Share of empty impressions | 31% | 6% |
| Reaction to a new item | 3 days | 2 hours |
We switched to a two-stage scheme: fast candidate retrieval by vector similarity, then ranking taking into account viewing history, cart contents, and the behavior of similar buyers.
Features and event aggregates are recalculated in ClickHouse, behavioral signals are updated as a stream, so a new item enters the selection a couple of hours after appearing in the catalog, not on the third day.
Seasonal demand spikes proved to be the main test: on peak days the load on recommendations grew severalfold, and the previous scheme degraded—the block began serving precomputed selections instead of personalized ones.
Recommendations were moved to automatic scaling with priority for fresh online features, and we run load tests on historical peaks before rollout. The share of empty impressions fell from 31% to 6%: recommendations stopped “staying silent” on rare categories and new positions.
How rules and manual selections differ from trainable models
Rules are explicit conditions: the product is in stock, delivery to the region, the brand is not on the stop list, the price segment fits the task. They produce results immediately, while there is no interaction history yet, and every decision is explainable.
The limit is personalization: all buyers see the same list, and the share of transitions to order stops growing.
The model works with two signal sources. Product features—category, attributes, description text, seasonality—allow recommending even positions that have no history.
Behavior—views, add-to-cart events, repeat orders, co-purchases—is a stronger signal, but for rare products it is sparse, and without features the model is useless for them.
In practice, a hybrid wins. Two-stage candidate selection scheme: first fast retrieval—co-purchases for “hot” positions, feature similarity for new items—then ranking for a specific buyer.
On top—rules as insurance: availability, region, assortment restrictions, and protection against the popularity loop when the model shows the same thing.
Order also matters: rules that trigger before ranking cut off infeasible options and do not spend computation on them; some are applied afterward—for business priorities.
Where to stop is decided by the volume of history and the share of new items: with a large flow of new positions, product features and rules work first, and the model is connected as data accumulates.
Technical concerns: cold start, labeling, data quality
Quality recommendations are determined not by the model but by the data beneath it. Cold start: a new product and a new buyer have no history, and the system either shows something random or lumps everyone into popular items.
Data labeling with an error in the target action turns optimization in the wrong direction—if click is set as the goal, the share of clicks grows while purchases fall.
Event quality sets the ceiling for recommendations more strongly than architecture. Duplicates in logs, gaps from undelivered events, bots, and test accounts inflate the sample and skew weights.
Therefore, before starting work we calculate the share of anomalous events and compare it with business reports: a 10–15% discrepancy means we first fix collection and then build the model.
What to do about cold start if there is no history at all?
We rely on content features of the product, the behavior of similar buyers, and the request context—three signal sources instead of history. In the first weeks, part of the impressions goes to exploration, so recommendation quality grows as events accumulate, not from day one.
How do you check labeling and event quality before starting work?
We take an export for a period and look at completeness, duplicates, delivery delays, and the share of bots. We reconcile the number of orders in the events with the accounting system and localize the discrepancy by segment. The output is a list of events that can be used as targets and those that require fixing.
How do you track recommendation quality after launch?
We keep recommendation quality monitoring on business metrics: share of orders from recommendations, catalog coverage, repeat purchases. Technical signals only explain a drop.
We fix thresholds and the rollback procedure before launch—degradation is visible within hours, not at the end of the month.
Let’s discuss the task: what catalog and what event flow do you have now
To make the conversation specific rather than a general overview, it is worth gathering a few things.
We are not asking for a ready-made technical specification—raw data and answers to questions about how your catalog is structured and what is already happening with traffic are enough. The more specific the input, the more accurate the estimate of the scope of work.
- Catalog export—product structure, categories, attributes, stock, and prices in the form in which they live in the accounting system: from it we determine what can be used as features and what will have to be enriched.
- Event log—views, search queries, add-to-cart events, orders, and returns: we are interested in the period and export format to understand whether the history is sufficient for training and quality validation.
- Access and data—which storages and services you are ready to grant access to, whether it is possible to work on production data, and whether there is a separate environment for experiments.
- Current recommendations—what is already being shown now: rules, manual selections, blocks on the homepage. This is the baseline for comparison; without it there is nothing to compare the result against.
- Display points—which places in the interface we cover first: homepage, product page, cart, emails, and push notifications.
- Constraints—requirements for response speed, personal data rights, and products that cannot be recommended under the platform’s rules.
Describe the task in free form—we will come back with an estimate of the scope of work and an implementation plan: stages, team composition, timelines, and checkpoints at which you can validate the result before rollout to all buyers.
