Developing a recommendation system for a music service

Why music recommendations break as the catalog grows

While the catalog is tens of thousands of tracks and a listener’s history has accumulated over years, the user × track matrix remains dense enough, and collaborative filtering confidently finds similar users.

Once the catalog reaches tens of millions of records and the median profile is a few dozen events, density drops by orders of magnitude: proximity between listeners is now computed from random coincidences.

Spotify hit this before others and moved from pure collaborative signals to hybrid ones—audio features, text, in-session behavior.

The second failure point is popularity bias. The model learns to predict a click, and the most reliable way to get one is to show what everyone is already listening to. The long tail does not make it into the output, gets no impressions, gets no signals—and remains invisible. Diversity drops, and collections for different people become almost identical.

The third is taste profile staleness. If the interest vector is averaged over the entire history without time decay, long-ago listens pull the selection toward themselves just as strongly as yesterday’s session.

A listener’s genre shift is not tracked, and a new track without history gets a cold start and an almost zero chance of being shown.

The result is falling retention: the selection stops matching taste, sessions get shorter, the “listened to the end and came back” signal weakens, and with it the data on which the system could retrain. This creates a vicious cycle of degradation.

We break down how this is closed: dense signals for sparse profiles, controlling output diversity, and rebuilding the interest profile with time taken into account.

What rebuilding recommendations delivers: the effect in measurable terms

Recommendation metrics cannot be read as a service-wide average—the shift is visible in slices: a new listener, someone returning after a pause, a subscriber with a long history.

We compare on holdout sessions: the new model and the current output receive the same requests, and the difference is calculated for each slice separately, not from the impression made by a showcase collection.

Hit rate in the selection—how many tracks from the top of the output the listener does not skip in the first seconds. Its growth means the model stopped guessing the genre “in general” and hit the specific context: morning playlist, commute, work in headphones. At the same time, systemic biases show up—when everyone is led into one genre or the same hits.

We calculate recommendation response time on percentiles, not the average: it is important that the selection keeps up with the listener’s action—skip, like, adding to a playlist—and takes it into account in the next session, not an hour later.

The distribution tail is more expensive than the average here: it is rare long responses that knock a person out of the flow.

Listening depth grows when the selection stops being a set of familiar tracks: we calculate the share of compositions played to the end and the number of new artists that took hold in the personal library.

We check listener return the next day and the next week—if the selection changes with mood rather than on an update schedule, the person comes back more often and listens longer.

Ways to build recommendations: comparison of approaches

The choice of approach is dictated not by fashion but by data volume and listening density. While there are thousands of users and each has a dozen tracks, co-listening statistics give almost nothing: the matrix is sparse, and a new listener’s profile is empty.

That is why several mechanics usually coexist in a product—the question is not “what to choose” but “what works at which part of the lifecycle.”

What we compare

How we choose in practice

At the start, the hybrid relies on track features and editorial shelves; later the share of collaborative signal grows as interactions accumulate.

Cold start is handled separately: for a new track, the vector is calculated from content features; for a new listener, from a short survey and first listens. We select blending weights offline using completion metrics, and this is part of maintenance work, not a one-time setup.

Stages of implementing a recommendation system

We carry out the work sequentially: each stage relies on the result of the previous one. Skipping the event audit and immediately starting model training means assembling features from accidentally preserved fields and hitting the data ceiling, not the algorithm.

1. Listening event audit. We examine what the service actually records: start, completion, skip, repeat, like, adding to a playlist—and where events lose their link to the listener profile.
2. Feature schema. We agree on composition and sources: listener profile, track properties, session context, interaction history. We version the schema, otherwise backfilling produces incomparable results.
3. Data splitting. We split training and validation parts by time, not randomly: the holdout set is the last few weeks the model has not seen. This removes future leakage and gives an honest evaluation.
4. Model training. The baseline is the current output. We add complexity only for a measurable gain, comparing top-N hit rate, completion share, diversity, and novelty—one metric almost always deceives.
5. Validation on a holdout set. We run candidates on data the model has not seen and simulate online behavior. We select the option with an average gain and no failure on rare genres.
6. Online experiment. We enable it on a share of traffic and monitor completions, repeats, and churn. The rollout decision is made on a set of metrics, not just one.
7. Handover to operations. We set up quality monitoring, data drift control, a retraining procedure, and rollback to the previous version. We log model and feature versions—without this, the result is not reproducible.

We compress or stretch the order to fit your data volume, but skipping any step surfaces in operations—when the output diverges from expectations and the cause is not obvious.

What we hand over: models, data schemas, and procedures

We hand over not only the trained model but the entire package for operating it independently—otherwise any change turns into digging through code and logs.

The package is designed for a team that did not participate in development: an engineer sets up the system from documents, not from chat messages.

• Data flow schema — from listening events to the feature store: sources, arrival delay, aggregation points. A new source is connected without changes to the model code.
• Feature reference — meaning, type, calculation window, source, and owner for each feature. When music recommendations decline, it is immediately clear which feature has drifted.
• Retraining procedure — triggers (metric degradation, shift in listening), frequency, composition of the validation set, and acceptance criteria for a new version.
• Version rollback rules — the procedure for returning to the previous model, target recovery time, and metrics confirming that the rollback worked.
• System documentation — architecture, limitations, known weak points, failure points, and incident investigation procedure.
• Monitoring procedure — which metrics we calculate, how often, which values we consider degradation, and where alerts are sent.
• Reproduction instructions — data composition, steps for training from scratch, and verification that the assembled model reproduces the one handed over.

Case study: music recommendations, metrics before and after implementation

Before the project, output was supported by manual editor selections and simple rules such as “popular in genre” and “fresh releases.” Shelves were updated once a week, and by midweek listeners burned out: the same tracks were repeated in every selection, and the recommendations themselves covered only a small share of listening. People went to search or returned to their saved lists—listener retention was falling.

We assembled the player event log: completions, skips in the first seconds, repeats, additions to a playlist, transitions between tracks.

On these signals we trained a ranking model, and we did not remove manual selections but changed their role—they cover the start for new listeners and editorial shelves, while the rest is selected by a combination of audio features and the behavior of similar listeners. We compare selection quality using unified measurements, as large platforms like Spotify do.

Metrics before and after

• share of listens coming from recommendations: 18% → 46%;
• share of track skips in the first 30 seconds: 41% → 27%;
• listener retention on day 30: 22% → 29%;
• share of manual selections in total listening time: 57% → 21%.

We separately monitored stagnation: repeats of the same tracks within a selection dropped by more than half, and new artists for the listener began to appear about three times more often.

At the same time, the start for newcomers did not decline—editorial shelves hold them until listening history accumulates, and the model is connected gradually.

How user cold start differs from track cold start

This is two different tasks, and they are solved with different signals. A new listener has no listening history, but there is the track itself and the session context. A new track has not a single listen, but there is an audience whose tastes are already described.

The contours must not be confused: a rule applied to the wrong object produces either noise in the first minutes or a release that never gains impressions.

For a new listener, direct signals work: a taste survey at registration, selected genres and artists, import of their collection from an external service, region, and interface language.

Then behavioral ones kick in—listening to the end, early switching, repeat, adding to their collection.

The taste survey gives a starting point, but it must not be overestimated: people mark what “sounds solid” and listen to something else, so the first actions outweigh the questionnaire.

For a new track, there are no taste signals at all—only its own characteristics and connections to already known objects.

Therefore the task is solved through a feature vector (tempo, key, arrangement density, vocals or instrumental, recording era) and through a similarity graph: the track is shown to those who have already listened to records with a similar description by the same artist or period.

Then the completion share on a limited audience decides—it either opens full impressions for the track or closes the topic.

The difference in metrics is also fundamental. For a listener, we measure how quickly the share of early switches falls and retention in the second week grows. For a track, how many impressions are needed for the prediction to stop differing from behavior on a broad audience. One threshold for both cases does not work: for a person it is too slow, for a track too fast.

Practical conclusion: in the system, these are two independent contours with a shared feature store.

The listener contour quickly collects preferences and refines them from the first actions; the track contour tests a new release on a small audience and only then scales impressions. Combining them degrades both tasks.

Technical questions asked before the start

Before the assessment, we clarify: how much listening history has accumulated, how events get from the player, and what already works in production.

This determines whether we rely on collaborative signals or on catalog features, so we address doubts about data, operations, and quality validation right away.

How are data sources connected?

A player event stream is needed: user and track identifiers, action type, timestamp. We ingest it in batches or streaming, bring it to a unified schema, and work from a copy without affecting production databases.

How is selection quality validated?

Offline—on the holdout part of the history: completion share, time to first skip, output diversity. Online—comparing two groups of listeners on the same traffic to separate the model effect from seasonal fluctuations.

What if the system is already deployed?

We enable the new logic in shadow mode: it calculates selections but does not affect the player until metrics confirm improvement. Switching happens by listener segments, with a quick return to the previous logic.

How is background listening taken into account?

We weight events by duration: a track abandoned in the first seconds and an explicit skip are different signals. A long session without interaction is not considered interest and does not pull the selection in a direction.

Let’s discuss the task: we’ll come back with an assessment and work plan

For music recommendations to work in your service rather than remain a demo on a test set, we need the full current picture. How output is arranged now, where listening events come from, what has already been tried and which limitation you stopped at. Describe the situation in your own words—that is enough to begin the analysis.

In return, we come back with a problem statement and work plan: what we do at each stage, what data is needed from your side, where the checkpoints are, and what acceptance will look like.

• problem statement: we formulate the goal in measurable terms—what exactly we improve in the output and how we will validate it;
• review of the current system: we look at which signals are already collected, what is missing, and where quality is lost;
• scope of work: we list the stages from data preparation to rolling out output to real users;
• implementation plan: we lay out the sequence, dependencies between stages, and points where your decision is needed;
• start order: we state what we do first and what can be postponed without losing the result;
• acceptance criteria: we agree in advance on which indicators we consider the task solved.

Describe the task and constraints—we’ll come back with an assessment and work plan.