Why recommendations don't work: where Next Best Action breaks down
A typical failure looks like this: the company has a recommendation engine, but it makes decisions blindly. Selection rules live in one place, purchase history in another, support requests in a third, and website behavior lives entirely separately.
No one consolidates these signals into a single profile, so the system sees the customer in fragments and fills in the picture with assumptions.
Hence the low targeting accuracy: the segment includes those who accidentally matched on one attribute, while half of the relevant audience is left out. The profile is assembled manually — an analyst exports data, consolidates it into a table, and updates it on a schedule.
By the next export, some attributes are already outdated, and the process itself does not scale: the more customers there are, the less frequent the recalculation and the coarser the segments.
The second source of losses is delay in decision-making. While the data reaches the calculation, while segments update, while the offer queues up for sending, interest has time to cool.
The offer arrives days after the customer viewed the product and no longer works as a hint but as intrusive mailing: response drops, and trust in the channel does too.
Next Best Action is structured differently — it is a continuous loop where signal, calculation, and action are linked in time, and the next best action is determined for each customer individually rather than for an average segment.
Below we break down how this loop is assembled step by step: from consolidating disparate data sources to verifying that the offer actually influenced the result.
Implementation effect: growth in ROAS, accuracy, and retention
The benefit of a Next Best Action system shows up not in the interface but in metrics.
Below are five changes that the client records in the first weeks after launch; the unified customer profile and events are assembled regardless of whether it runs on Salesforce or on its own data mart.
Accuracy of hitting the target action
The model ranks options for a specific customer, so real-time offer personalization aligns with their next step. The share of hits on the target action and ROAS in repeat-sales scenarios grow several times over compared with a single mailing to the entire base.
Speed of response to demand
The profile is updated by events, not by an overnight recalculation, so the system adjusts the offer following behavior: abandoned cart, view, support request. A demand spike is handled in minutes — previously a campaign report took days to prepare.
Churn and repeat purchases
Ranking includes risk signals: declining purchase frequency, unsubscribes, decreasing engagement. Retention is offered before departure, so churn in risk groups falls by about a third, and the share of repeat purchases over a quarter horizon grows.
Analyst workload
Segments and rules are described once and reused, so samples are no longer assembled manually for each request. The freed-up time goes to hypothesis testing and data labeling — recommendation accuracy directly depends on it.
Ways to build recommendations: comparison of approaches
Approaches do not compete with each other — they cover different modes of operation.
While there is little data, the decision relies on manual logic and segments; when comparable history accumulates, a statistical model begins to describe it; when response to a fresh signal matters, real-time learning is added to the loop.
The choice is determined by three things: the volume and depth of history, the delay between an action and its result, and the cost of error.
| Approach | When appropriate | What limits it |
|---|---|---|
| Rules and segments | Start, cold loop, strict business constraints, little data | Rebuilt manually when demand changes, quickly hits a quality ceiling |
| Model on historical data | A comparable event log has accumulated, demand changes smoothly | Ages along with the data, useless for new items and segments |
| Real-time learning | Short deal cycle, lots of traffic, fast signal changes | Requires online features and drift control, sensitive to noise spikes |
| Hybrid scheme | Mixed assortment and heterogeneous segments | Harder to operate: needs a layer that decides whose answer carries more weight |
A hybrid scheme is assembled as a superstructure: rules set hard boundaries — forbidden combinations, limits, mandatory items — the model ranks candidates within the allowed field, and the online loop mixes in a fresh signal with limited weight.
The recommendation remains manageable: any answer can be explained and rolled back without rewriting all the logic.
For the task, we take not the most complex approach but the minimal one that sustains the target metric. Rules and segments remain a safety net at the edges where there is not yet enough statistics.
The transition point from one option to another is determined in a pilot: if the gain from the model stops covering the cost of its maintenance, the scheme is simplified.
Stages of work: from data audit to production launch
We start not with the model but with the data: if the action log is incomplete and the target event is labeled differently in different systems, the algorithm learns on noise and produces recommendations that no one applies. Therefore the work proceeds in fixed stages, and each ends with a verifiable artifact.
- Data source audit. We look at which systems provide events, with what delay and in what form, where customer identifiers are lost, and where reference data diverges.
- Event and target action schema. We establish a unified dictionary: what we consider an impression, an action, and a target result, and how each of them relates to the customer profile.
- Loop assembly. Feature recalculation, history storage, and a service that selects an action at the moment the channel is contacted and returns a response in a time that does not affect response.
- Policy setup. Limits on contact frequency, segment exclusions, channel priorities, and stop rules when quality deviates from expected.
- Validation on historical data. We train the loop on an early period and see whether it reproduces actual behavior on the next one.
- Gradual launch. Traffic share of 5–10% and a control group; we monitor the share of accepted recommendations, coverage, and response stability.
- Handover to operation. Retraining regulations, input data monitoring, a metrics dashboard, and training for the client's team.
Implementation artifacts: models, rules, documentation, regulations
Implementation is considered complete not at the moment the model launches, but when the loop goes into operation without us.
Therefore the documentation set is assembled so that the client's team can manage it themselves: any change in features or rules does not require correspondence with the contractor.
- Trained ranking loop: training configuration, feature dictionaries, and model weights — reproduced from artifacts with a single command, the result is repeatable on the same sample.
- Feature description: for each one, the source, aggregation window, data appearance delay, and acceptable missing share are specified — the analyst sees why an action received a high score.
- Access policies and restrictions: excluded segments, communication frequency ceiling, role rights to data, approval procedure for new sources.
- Integration scheme: which systems provide events, which one makes the decision, where queues are placed, and how the loop behaves when a source is unavailable.
- Instructions for analysts: how to add a feature, retrain the model, and compare quality on a holdout sample, how to read the report on issued recommendations.
- Model update regulation: retraining frequency, triggers for target metric degradation, who approves the rollout of a new version.
- Rollback plan: return to the previous version or to the base rule with a single switch, activation conditions, and the person responsible for the decision.
- Decision log and monitoring: logging of recommendations and actual response, alerts when the forecast diverges from the result.
Case study: growth in ROAS and repeat purchases after switching to personalized recommendations
The project was launched at a retailer with a broad assortment: marketing worked by a promotional calendar, the same mailing went to the entire base, and an analyst assembled segments manually in the reporting mart.
At the start, baseline metrics were recorded: conversion to the target action — 1.9%, share of recommendations hitting the top 3 — 14%, churn — 4.2% of the active base per month, campaign preparation cycle — two weeks.
We assembled an event mart and trained a ranking model on CatBoost: features — purchase history and recency, visit frequency, response to past touches, product availability, and price relative to the customer's usual price segment.
On top of the model is a rules layer: we do not offer purchased or out-of-stock items, there is a touch frequency limit and a restriction on discount depth.
How the effect was calculated
We compared over a comparable period — the same twelve weeks with a similar seasonal wave and the same traffic volume. A tenth of the base was left on the previous scenario as a control group, so the growth cannot be attributed to seasonality or a demand spike.
Conversion to the target action grew to 4.4%, the share of top-3 hits — to 58%, churn decreased by about a third. Campaign preparation takes 40 minutes instead of two weeks, recommendations are recalculated once a day based on fresh events. ROAS over the comparable period grew 4–8 times depending on the category.
The system was handed over to the client's analysts: features, rules, and quality monitoring are described, retraining follows the regulation, and rollback to the previous scenario is possible without stopping mailings.
How does Next Best Action differ from rules and manual segments?
The difference is in who formulates the selection logic.
In rules, a human sets it: “the customer has not visited for 30 days and did not open the email — offer a win-back offer.” Such a condition lives until the next revision, and a manual segment describes a single slice of the audience.
Dynamic action selection works differently: the algorithm evaluates all permissible actions based on fresh data and chooses the best one at the moment of contact.
| What we compare | Manual rules and segments | Dynamic action selection |
|---|---|---|
| Who sets the logic | Analyst, manually | Evaluation of options based on data |
| Speed of changes | Approvals and report reassembly | Recalculation of weights based on new data |
| How many scenarios it holds | Dozens, then condition conflicts | Hundreds of options without manual work |
| Reaction to new behavior | We wait until someone notices a decline | Accounted for in the next recalculation |
Scaling scenarios is the main dividing line. Twenty segments for five actions produce a hundred rules, and they begin to conflict: one promises a discount, another a manager call, and priority has to be set manually. In the model, the set of actions is unified; a new option is simply added to the evaluation.
When rules are enough: if there are fewer than ten scenarios, customer behavior is stable, and there is little data for training, manual logic is simpler and more transparent — you can see why the customer received a specific offer.
Beyond that, maintaining conditions starts to consume the team's time. Some platforms cover this built-in: in Salesforce, the next action recommendation is displayed to the manager in the interface, but it relies only on the data that reaches it.
Technical concerns: cold start, labeling, fault tolerance
A cold start does not mean waiting: for the first weeks the loop runs on rules and expert priorities, and the model is connected when a sufficient volume of confirmed outcomes has accumulated. As data grows, the weight of rules decreases automatically, without manual switches. Below are answers to the objections most often heard before launch.
What if there is almost no customer history?
For the first weeks, priorities are calculated by rules and segments — the result is there immediately. As confirmed outcomes accumulate, the calculation shifts to data, while rules remain the framework and safety net.
What about rare events?
We train not on the rarest event itself but on its precursors: series of actions, changes in frequency, reactions to past communications. Aggregation at the segment level provides a sufficient sample, and the signal does not drown in noise.
What to do with incomplete data?
We mark missing values explicitly rather than replacing them with zeros: the system knows that a feature is absent and takes this into account in the decision. A channel without history does not receive an inflated priority by default.
Will the model be diluted by business-rule restrictions?
Rules set hard boundaries: what, to whom, when, and how often it can be offered. The model optimizes the choice within these boundaries, not instead of them, so product prohibitions are observed literally.
What happens if a data source fails?
The loop returns the last valid state rather than an empty response: recommendations continue to go out in degraded mode. The failure goes to monitoring, recalculation after source recovery happens automatically, and the log stores which data each offer was built on.
Let's discuss the task: we will assemble a recommendation loop for your data stream
For the recommendation loop to be assembled for your data stream rather than for an average template, the input requires not access “to everything” but a description of how events are structured and what counts as success.
The more precise these inputs are, the shorter the path to the first hypothesis that can be tested on a live stream rather than on guesses.
What will be needed from your side when discussing the task:
- Event description: which actions are recorded, in what breakdown and with what delay they reach the storage — the set of features on which the model trains depends on this.
- List of target actions: what we consider a result — a request, a repeat purchase, a renewal, an upgrade to a higher tier. Without an explicit list, the model will start optimizing something that does not bring revenue.
- Content restrictions: which offers can be shown to the customer and which are closed by regulation, contract, or brand requirements. These are the boundaries of the decision space.
- Hypothesis testing horizon: over what interval we consider the effect reliable and by what metric we compare recommendations with the current delivery rule.
- Channels and frequency rules: where we show the recommendation, how many times, and with what “silence” after a refusal — so as not to burn out the audience.
- Procedure for starting work: who on the client's side approves features and accepts the result, in what environment we raise the first loop.
Next, we fix the hypothesis testing plan, calculate the baseline from current metrics, and launch the loop on part of the stream, comparing it with the current rule. Describe the task — we will come back with an estimate and a work plan.
