Why news feed personalization breaks at scale
On small traffic, news feed personalization looks workable: the model ranks by freshness and clickability, metrics grow.
At scale, the same logic starts to destroy itself — the behavior of tens of thousands of sessions per day is unevenly distributed, features are recalculated from the event stream (we collect via Apache Kafka), and any skew becomes systemic rather than a random error.
The model quickly finds the dominant interest and pushes it to the limit: the monotony of the feed grows, new materials gain no impressions and do not enter the training sample — recommendation looping becomes statistically entrenched.
Session depth falls: the user scrolls through three or four cards and leaves because there is nothing unexpected in the feed anymore.
A separate failure is user cold start: without history there are no signals, and the person gets a random set in the first sessions.
Manual weight edits, which are usually used to fix such skews, work up to a certain point, and then conflict with each other, and it is no longer possible to understand which rule broke the feed without analyzing feature contribution.
We break down how this is closed: where an exploration share is added to ranking, how diversity becomes a separate goal alongside relevance, and how to limit manual interventions so they do not cancel each other out.
What managed personalization delivers: effect in metrics
Managed personalization is measured not by clicks in the moment, but by behavior over distance.
The basic set of metrics: share of readers who returned on day 7 and day 30, session depth, share of the feed accounted for by one topic, and share of topics that a person has not opened even once in a month.
The first two show that the feed has not been abandoned, the latter show that it has not been narrowed to a single category.
Feed diversity is set by a rule, not by hoping for the model.
Some slots are reserved for topics that are absent from the reader's history, and the share of similar materials in a row is limited — the share of repeats in the feed falls, topic coverage grows without retraining the ranking. The rules themselves live in configuration and change separately from the model.
For new users, the feed behaves predictably: while there is no history, explicit cold start works — topical and editorial collections, geo, time of day. Rare topics are not washed out of the feed because they have a guaranteed minimum of impressions over the window.
Public breakdowns by Netflix and other services have repeatedly shown: optimization only for instant response raises clicks and sinks long-term retention. That is why we evaluate feed rules over a long window and on a holdout sample, not on a single day.
Bottom line: recommendation system metrics become manageable quantities. A rule is rolled out to part of the traffic, compared with control, and either stays or is rolled back — without rebuilding the entire model.
Feed modes: rules, model, and hybrid scheme
The choice of ranking scheme is determined not by fashion, but by traffic density and catalog volume.
While there are thousands of materials and a narrow audience, transparent logic is enough; when interactions number in the hundreds of thousands, without a trainable model the feed stops hitting the interest. Below are working options with their applicability and limitations.
A separate axis is session context: a one-off visit and a deep series of consecutive reads require different serving, otherwise the feed either scatters attention or locks it onto one topic.
| Option | When it fits | Limitations |
|---|---|---|
| Ranking rules | Small catalog, clear priorities: freshness, category, author | Each new condition conflicts with the previous one, the effect of an edit is unpredictable |
| Collaborative filtering | Dense traffic, many repeat visits, accumulated interaction history | New material and a new reader remain without recommendations |
| Content similarity | Sparse data, fast-growing catalog, niche topics | Locks the feed into one topic, diversity falls |
| Hybrid feed scheme | Mixed traffic, different audience segments | Harder to configure: needs control of each signal's contribution and regular validation |
The hybrid gives the best result, but requires discipline: the diversity share is set explicitly, and metrics are calculated separately by segment, otherwise a strong group masks a failure in a weak one.
Before launch, the scheme is run on historical data and compared with the current feed by session time, completion rate, and repeat returns. This shows reader behavior, not the beauty of a report.
Implementation stages: from data audit to launch on users
Feed personalization is not a single model but a loop: event collection, features, ranking, and diversity control. The order of work here is not a formality — each stage relies on the previous one and removes some risk from the next.
Below is what implementation of recommendations looks like from data audit to release to live users.
- Data and event audit. We look at which signals are already collected, where there are gaps, duplicates, and arrival delay. The output is an event map and a list of features that are calculated reliably.
- Scheme design. We determine how relevance is calculated, where diversity is enabled, and which rules prevent the feed from collapsing into one topic. Here we also fix success metrics.
- Prototype on historical data. Training and validation on past traffic, without risk to users. Separately, model quality is checked: cold start, new authors, rare topics.
- Control comparison of options. The new feed runs against the current one on equal traffic shares. We look not only at average growth but also at segment drops.
- Gradual rollout to traffic. The user share grows in steps, each with stop conditions: falling completion rates, rising complaints, decreasing diversity.
- Rollback of changes. A switch returns the previous ranking scheme in minutes, without rebuilding: the model version and configuration are preserved.
- Handover and team training. Documentation on the scheme, metric dashboards, retraining regulations, and incident review.
After launch, the loop remains with your team along with metrics and regulations — development and retraining proceed without us.
What we hand over to the client's team: artifacts and regulations
Handing over a system is not handing over code, but handing over control. After launch, the client's team must independently change feed rules, add signals, and explain to the editorial team why the feed behaves this way — without contacting us for every change.
- Ranking scheme — a list of signals, their weights, and application order, with a note on which parameters are safe to edit and which require feature recalculation.
- Mixing rules — how the diversity share is set, the depth of anti-repeat by topic and source, what happens to new categories that do not yet have history.
- Access separation — editor, analyst, and administrator roles: who changes weights, who only views metrics, who approves release to the production feed.
- System documentation — architecture diagram, integration points with the news source, signal dictionary, and change log from the first launch.
- Update regulations — how often weights are revised, on which sample the hypothesis is tested, and at what point rollback to the previous configuration is enabled.
- Editorial instructions — step-by-step scenarios: raise a category in priority, exclude a source, pin a topic manually, check the effect of an edit after a day.
- Measurement methodology — which metrics are considered target, how to separate seasonal fluctuations from the impact of changes, and how to record the experiment result.
Acceptance is based on one criterion: the client's team independently changes feed rules, measures the effect, and returns the previous version if metrics drop.
After that, we stay in touch for support and improvements — but the control point is already inside your team, not with us.
Case study: how the feed stopped looping on one interest
The project is a regional media outlet Городские Новости with a daily agenda: the home feed was assembled from viewing history and material freshness.
A month after launching personalization, regular readers began to receive almost the same set of categories — the model confidently converged on a narrow interest zone and showed almost nothing beyond it.
What the metrics showed before restructuring
The share of materials new to the reader in the feed stayed at 6–8%: the feed worked as a repeat of what had already been read. Materials outside the top categories lost reach, and authors outside the audience core received noticeably fewer impressions.
Every week the editorial team manually edited weights to pull up individual publications — this took hours, and the result rolled back after a few days.
What was changed in ranking
Diversity was made an explicit part of the formula: in addition to click probability, we calculated topic novelty for a specific reader and category saturation — when skewed, the weight of familiar categories decreased automatically.
Candidates were selected with a guaranteed share of unfamiliar content, and for new readers we relied on editorial collections rather than accumulated impression statistics.
Implementation results
The share of new materials in the feed grew from 7% to 23% while maintaining viewing depth — readers did not scroll less. Repeat visits during the week increased among the audience that previously opened only its own category.
Manual weight edits were removed entirely: the editorial team returned to content, and changes to the ranking scheme are rolled out via settings, without code changes and without downtime.
How filtering differs from ranking and what to choose
Filtering and ranking solve different tasks and work on different data. Filtering is a binary decision for each material: show or not. Ranking is the order of materials within an already admitted set, where each of them has the right to be shown.
Everything that should not reach the feed goes into the filter: duplicates and nearly identical reprints, materials outside the relevance window, already read items, unavailable sources, empty cards.
Such rules are deterministic, covered by tests, and explained in one line — this noticeably simplifies complaint review and rollback of changes.
Ranking is responsible for how materials line up on the first screens. If the filter fails, the order goes to junk: reprints rise to the top, and diversity falls — several similar materials occupy the entire feed.
| Option | When it fits | Limitations |
|---|---|---|
| Filtering only | Small catalog, rare updates, chronological feed | Order does not account for interest, relevance does not grow |
| Ranking only | Catalog already cleaned, behavior history accumulated | Duplicates remain in the set, quality depends on data completeness |
| Filtering and ranking | Large publication flow, high share of reprints | Both parts need to be maintained and validated together |
The choice of approach depends on the flow: with dozens of publications per day, cutoff rules are enough; with thousands, without ordering the feed loses relevance.
The guideline is simple: if after filtering duplicates still appear on the first screens, the filter is configured incompletely, and ranking will not fix this.
Doubts before starting: addressing technical questions
Doubts before starting are usually the same: where reader data goes, what to do with rare categories, and how to notice degradation rather than write it off as “the feed has changed” — typical questions in news feed personalization. Below are short answers to each point.
Where is reader behavior data stored?
Depersonalized events — views, completions, transitions — remain in your environment or in a dedicated segment under our management. Only aggregates and quality metrics go outside. Access and retention periods are fixed by regulations.
Will rare categories drop?
This is an expected risk: the fewer impressions a topic has, the less often the model suggests it, and the category goes into a loop.
We maintain an impression quota and a separate weight for the long tail — relevance grows, and the feed does not collapse into three popular topics.
How to understand that it got worse, not “different”?
Before launch, we fix a baseline: completions, session depth, returns, exits after the first screen. We compare on the same sample — part of the traffic goes by the old logic. A drop beyond the threshold is rolled back with one switch, without edits to the feed code.
Can it be built on top of the current system?
Yes, if the feed keeps an impression log and allows an external ranking layer. We add this layer and a switch between the old and new order without rewriting the system. A full replacement is needed only if impressions are not stored anywhere.
Let's discuss the task and come back with an estimate and plan
To discuss the task substantively, and not in the format of “we want AI in the feed,” a few inputs are enough — they replace a week of correspondence.
From them we build a picture of the current feed and understand exactly where the balance between relevance and diversity is lost.
- Description of the current feed scheme: how the feed is formed today — rules, manual collections, ranking models — and at what point in the pipeline the order of cards is built.
- Data composition: which events are recorded (impressions, clicks, completions, skips, unsubscribes), with what delay they are available, and in what form they can be provided for analysis.
- Content volume and structure: how many publications are released per day, how many sources and categories, whether there is topic tagging, and how fresh material without accumulated statistics behaves.
- Dimensions important for personalization: new and regular reader, region, device, topic interests, paid and free materials.
- Bottlenecks: what has already been tried, where relevance grows but diversity falls, how the repeat “bubble” and skew toward one category manifest.
- Desired metrics: what we consider success — session depth, completion rate, returns, share of impressions of topics new to the user.
- Constraints and access: requirements for response speed, traffic volume, data storage, as well as a test environment and log exports.
Then we come back with an estimate and work plan: data composition, feed scheme after changes, control metrics, risks, and project start order — with validation on part of the traffic so the solution is confirmed by measurements, not promises.
If the inputs show that the task is solved by feed rules rather than a model, we will say so directly.
