Why a content recommendation system loses accuracy
A content recommendation system breaks not at launch, but after several weeks of operation, when recommendations stabilize and stop responding to changes. Formally, the recommendation works: blocks are filled, there are no errors, yet the target action rate slowly declines. The cause is usually not load, but that features diverge from actual audience behavior.
The catalog is described using editorial labels — section, topic, author, format. The user chooses differently: by viewing context, by what they opened before that, by where they stopped.
When recommendation accuracy relies only on item description, the system confidently suggests things similar by labels, but not by interest, and each subsequent impression leads away from the goal rather than closer to it.
A separate failure point is catalog cold start. A new item has no interaction history, so it receives no impressions, and without impressions history will never appear.
The same positions remain in recommendations, fresh sections gain no engagement, and the recommendation gradually stops reflecting what is actually in the catalog.
The third cause is recommendation monotony. A recommendation tuned to an average profile offers items with similar descriptions: the user sees five similar cards and leaves.
The loop closes: the system reads such a response as confirmation of its hypotheses and narrows recommendations even further. We explain how this is addressed — from rebuilding features to managing diversity.
What changes after implementation: measurable effects
Interest hit rate
The user profile is built from features of the content they actually consumed: topic, author, format, duration. Recommendations rely on matching properties, not overall popularity, so the target action rate — clicks, read-throughs, saves — grows even in narrow topics.
Speed of new item appearance
A new item enters impressions immediately after publication: its features are compared with profiles, and it finds its first readers with virtually no delay.
There is no need to wait for interaction statistics to accumulate — content does not “hang” in the catalog while the algorithm gathers data.
Catalog coverage completeness
Impressions are distributed across the entire catalog, rather than concentrated on a few popular positions. The share of items that received at least one impression during the period grows, and the long tail stops being dead weight — including archived publications.
Repeat visits
When a user regularly receives relevant content, returning becomes a habit: they know they will find what they need without searching. The share of repeat visits and session depth grow — with the same catalog and without manual tuning for each segment.
Reduction of manual edits
Manual boosts are the most expensive way to support recommendations, and they scale poorly. After launch, the share of manual edits falls to isolated cases: seasonal collections, promo blocks, rights restrictions on an item.
Recommendation approaches: how the system selects impressions
The signal source differs between approaches: the content of the item itself, the behavior of users with similar interests, or a combination of the two. What to choose depends on the volume of accumulated statistics and how quickly the catalog updates.
| Approach | When it fits | Limitations |
|---|---|---|
| Content-based filtering | Catalog grows faster than impression statistics accumulate | Interest profile narrows, unfamiliar items are shown rarely |
| Matching by behavior of similar users | There is action history over several weeks | New items and new users fall out of recommendations |
| Hybrid recommendation | Catalog is mixed: both fresh and long-lived content | Weight tuning is needed, otherwise one signal overrides the others |
Interest profile and feature weights
The interest profile is item features with weights: topic, author, format, freshness, language, source. Each action changes the weight: read-through and saving raise it, a quick return to the list lowers it. This keeps the profile fluid and reflects a change of interest, rather than an all-time average.
Weights are normalized among themselves — otherwise one active feature overrides the others and recommendations collapse into a single topic.
Features are stored in the Elasticsearch search index: rebuilding the feed for the current profile happens on the fly, without recalculating the entire item base.
How we implement recommendations: engineering stages
We start with data, not a formula: until the “content — interest” relationships are restored, recommendations produce random cards with any careful mathematics. Then we proceed through six stages.
- Data review. Inventory of cards, texts, tags, view and rating history; we note which features are missing.
- Success criteria. We define what we consider good recommendations: top-10 hit rate, read-through, catalog coverage. Without this, quality cannot be measured.
- Feature schema. We describe how content turns into a profile: topic, author, tags, description keywords, and the weight of each feature.
- Recommendation tuning. We build the user interest profile, calculate proximity to content, and cut off weak matches with a threshold.
- Recommendation quality validation. We calculate metrics on a holdout sample, then compare against a control group of the live audience.
- Solution handover. We deploy the service and provide the client’s team with the configuration, feature schema, and retraining procedure.
Data preparation
We normalize texts, collapse duplicates, and restore missing tags from descriptions. The output is a comparable vector for each item instead of free-form fields.
Recommendation quality validation
Offline metrics are not enough: they do not show how recommendations behave on new items and unfamiliar users. We look at top hits, recommendation diversity, and interest distribution — so the recommendation does not trap a person in a narrow topic.
Deliverables: models, schemas, and procedures
The client receives not only working recommendations, but also everything needed to support them independently. The documentation package records which features participate in ranking, where they come from, and how their quality is validated.
What we hand over as part of the work
- Feature schema: list of fields, types, sources, recalculation frequency, and missing value handling rules.
- Update procedure: how often model weights are recalculated and which metrics serve as approval for publication.
- Data access policies: roles, rights to interaction history, anonymized profiles, and service logs.
- Editorial instructions: how to add content and which fields are required for an item to enter recommendations.
- Change validation procedure: test environment, acceptance criteria, and rollback procedure to the previous version.
- Feature catalog with a description of the storage where they reside — in our case, PostgreSQL.
- Experiment log: which hypotheses were tested and why some of them were rejected.
Support procedure
After handover, we stay in touch: we investigate recommendation quality incidents, help recalculate features when the catalog changes, and advise the editorial team on disputed items.
Scheduled recalculations follow the update procedure’s schedule; unscheduled ones go only through the test environment with a control sample run. If metrics fall below the threshold, we return to the previous set of weights without stopping recommendations.
Case study: improved recommendation accuracy in an online store
Initial situation
The online store’s catalog on 1C-Bitrix was assembled manually: a category manager selected products once a week based on past sales and placed them in blocks on the homepage and in product cards.
About one fifth of the catalog was shown, a new item waited about two weeks for its first impression, and the interest hit rate remained at 11% — the visitor opened a product card and left.
The root of the problem is that manual selection relies on sales, and only the top items have them. Items that did not make it into blocks receive no impressions and gain no statistics — the loop closes, and the catalog does not exit it on its own.
What content-based filtering delivered
Manual blocks were replaced with attribute-based recommendations: the interest profile is built from the name, description, category, materials, and properties of a specific product, then compared with the visitor’s viewing history.
Recalculation happens as new items appear, so a new item does not need to wait for sales to accumulate.
The interest hit rate grew from 11% to 34%, catalog coverage — from 20% to 88%, and the first impression of a new item takes less than a day instead of two weeks. Manual selections remained only for promotions: the manager sets conditions rather than sifting through products.
Metrics were calculated on a holdout period, then confirmed on the live audience with a comparison group.
How content-based filtering differs from collaborative filtering
Both approaches solve one task — showing the user what they are highly likely to open — but rely on different signal sources. Content-based filtering describes the object itself: topic, author, tags, text and vector features.
Collaborative filtering looks not at the object, but at the interaction matrix and infers connections from the behavior of similar users.
| Approach | What it works on | Where it is strong | Limitation |
|---|---|---|---|
| Content-based filtering | Features of the objects themselves | New items, narrow catalogs | Locks into what is already interesting |
| Collaborative filtering | Interaction history | Dense traffic, unexpected discoveries | Cold start, data sparsity |
| Hybrid | Both signal sources | Mixed demand, updating catalog | Needs tuning of each contribution |
The difference is visible at the edges. Content-based filtering starts without behavioral data, but poorly leads beyond a familiar topic. Collaborative filtering provides unexpected connections, but runs into the cold start of a new user and data sparsity in narrow sections.
The choice of approach is determined by what the system has more of — object descriptions or behavior. With a daily updating catalog and a small audience, relying on content delivers results immediately; with dense traffic, behavioral connections work more accurately.
In practice, a hybrid more often wins: content covers cold start and fresh items, while collaborative mode refines recommendations on accumulated data. We build both loops and tune their contribution based on measurements on live traffic.
Common technical questions about implementation
Most often, questions at the start concern not the algorithm itself, but how the system behaves on real data: a short interaction history and a catalog that changes every week. Below is what we review with the client’s technical team before work starts.
The Content-Based Filtering approach relies on features of the content itself and the user interest profile, rather than on mass interaction history, so meaningful recommendations are possible even with sparse data.
What should we do if there is almost no data on user behavior?
The interest profile is built from explicit signals — selected items, queries, ratings — and features of the content itself. As history accumulates, the weight of behavioral signals grows, and the system transitions from content rules to personalization without a restart.
The catalog updates every week — will everything need retraining?
No. New items are indexed by features and enter recommendations immediately, without a full recalculation. A full recalculation of profiles runs on schedule and does not block system operation.
Can content and behavioral features be combined?
Yes, mixed recommendations usually deliver the best result: content features provide coverage of the entire catalog, including new items, while behavioral features provide ranking accuracy. The balance is set by the amount of history for a specific user.
How do we verify that recommendations have actually improved?
We compare offline metrics on a holdout sample with an online snapshot: top hit rate, recommendation diversity, and the share of users with a non-empty recommendation. After launch, we monitor feature drift — this is part of support.
Let's discuss the task and return with an assessment and plan
To discuss the task concretely and return with an assessment and plan based on facts rather than assumptions, we need a short review with your product and technical context.
In our experience, the effort required for a content recommendation system is determined not by algorithms, but by source data and constraints on it — so some questions should be resolved before the meeting.
What will be useful from the client side:
- Source data: content catalog with metadata, interaction history (views, read-throughs, saves, skips), texts, and topic labeling, if available.
- Volume and quality of history: over what period actions have accumulated, what share comes from new users, and whether there are duplicates and records without features.
- Data usage constraints: what can be used for training, what requires anonymization, and what retention periods and regional requirements apply.
- Success criteria: what we consider improvement — relevant impression rate, session depth, retention, catalog coverage. Without agreed metrics, tuning turns into a debate about tastes.
- Constraints and environment: where the system runs, what response time is needed during peak hours, how it integrates into the current recommendation layer, and what must not be broken.
- Participants and decisions: who is responsible for data export, who accepts the result, and who approves changes in product logic.
Describe the task — we will return with an assessment and plan: stages, scope of work, data requirements, and checkpoints.
