Why conversion prediction breaks on real data
A conversion model that showed accuracy above 0.9 on a holdout set often drops to random guessing in production within a few weeks.
The reason is not the algorithm: conversion prediction breaks on the properties of the data itself — a rare target event, biased history, and non-stationary user behavior. Let's break down how this is addressed at the engineering level.
Sample bias manifests most quietly. Logs are trained on events that have already passed through the current funnel: impressions, clicks, past purchases.
Users whom the system previously did not allow into the impression do not make it into the history — the model learns on a truncated space and confidently errs precisely on those segments where the decision matters most.
The fix is not "adding data" but restoring the selection mechanism: we log the reasons for non-impression, record the policy of past decisions, and recompute the weights so that the feature distribution during training matches what the model will see in production.
Rare events add a second layer of the problem. Conversion is a fraction of a percent of traffic, and a standard loss function on such an imbalance simply learns "always no." The accuracy metric looks decent, while the model is useless.
That is why the target variable is redefined: training is done on intermediate signals (micro-conversions), class weighting and probability calibration are applied, and quality is evaluated by precision-recall and lift in the top deciles rather than by overall accuracy.
Data drift finishes off what is left. Audience behavior changes: a new channel arrives, the price changes, the interface is updated — and the feature distribution drifts away from the training one.
Without divergence monitoring and regular retraining, the model quietly degrades between releases. Next — on how these three causes are addressed at the level of architecture and process.
What a Conversion Prediction Model Delivers and How It Is Built
A conversion prediction model is not a "smart button" but a loop of three parts: features, training, regular re-evaluation.
The input consists of behavioral events, touchpoint history, segment properties, and visit context; the output is the probability of the target action for each user.
Then the cutoff threshold is tuned to the task: retention in the funnel, priority in the processing queue, or a communication trigger.
What We Actually Measure
We assess model quality not by a single number. We look at ROC-AUC and PR-AUC on a holdout sample, probability calibration (how well the predicted 20% matches the actual share), and lift in the top deciles — that is what determines how many outreach attempts are not wasted.
Sample completeness matters no less: if events are collected with losses, the model learns on biased data and confidently errs on "quiet" segments.
What We Build With
Gradient boosting takes tabular features and categories, is robust to missing values, and provides ready-made feature importance scores — it is the workhorse for most tasks.
Logistic regression wins where transparency is needed: coefficients read as the influence of a factor, the model is easy to explain to the business and to check for stability.
In practice, boosting is often set as the primary model, and regression as the baseline and drift control.
| Option | When It Fits | Limitations |
|---|---|---|
| Logistic regression | Little data, interpretation needed, quick start | Poorly captures nonlinear relationships and feature interactions |
| Gradient boosting | Tabular data, heterogeneous features, accuracy needed | Requires tuning and overfitting control, weaker explainability |
| Neural network on sequences | Long touchpoint history, texts, complex scenarios | Requires data volumes and infrastructure for regular retraining |
scikit-learn covers the baseline and metrics, boosting covers the primary model.
Implementation pays off not through "magic" but because outreach prioritization becomes measurable: the share of hits in the target segment in the top deciles grows severalfold relative to random selection, and decisions rely on a reproducible calculation rather than intuition.
How we implement the model: from data audit to handover
Implementing a conversion prediction model starts not with training, but with a data audit: we look at what is actually available in the CRM, web analytics, and billing, how target events are labeled, and whether there is any leakage of the future into the features.
At this stage, it most often turns out that some conversions are lost in the logs, and the share of labeled examples is too small for honest training — this changes the work plan before the first line of code is written.
- Data audit. We consolidate sources, check the completeness and consistency of target event labeling, calculate the baseline conversion rate and the observation period. We record what data is missing and how to collect it.
- Feature engineering. We assemble behavioral, transactional, and contextual features, monitor temporal shifts so the model does not "peek" into the future. Separately, we prepare categorical features and missing value handling.
- Training and model selection. We compare several approaches on a holdout set, tune the class balance and the trigger threshold for the business task, not for abstract accuracy.
- Model validation. We validate on a temporal split, not a random one: we look at the share of correct triggers in the top segment, robustness to data shift, and behavior on rare audience segments.
- Model integration. We connect scoring to the CRM, ad platforms, or website via API, set up regular feature recalculation and prediction quality monitoring.
- Handover and support. We provide documentation, feature schemas, a retraining policy, and a dashboard with metrics so the client's team can work with the model independently.
Each stage ends with an artifact that can be verified: a data report, a feature set, metrics on a holdout set, a working integration loop.
This order allows you to stop at any step and see exactly where conversion prediction quality is being lost, rather than finding out after launch into production.
How is conversion prediction different from revenue forecasting?
Conversion prediction and revenue forecasting are two different types of tasks, and confusing them is costly: the model will be trained on the wrong target, and the output will be beautiful numbers that don't answer the business question.
The first answers "will the event happen" — the user will place an order, the lead will reach the deal, the client will activate. The second answers "how much" — what amount this user, segment, or period will bring. Hence the different mathematics.
Conversion prediction is a classification task. The model outputs the probability of an event in the range from zero to one, and the decision is made by a threshold: above the threshold we consider it "will convert," below — it won't.
The model metrics here are about the quality of ranking and class separation — ROC-AUC, precision, recall, the share of correct hits at the top.
The threshold is chosen not by the beauty of the curve, but by the economics: how many false positives the process is willing to accept, and what the cost of missing a target user is.
Revenue forecasting is a regression task. The model predicts a continuous quantity, and the metrics are different: mean absolute error, the share of explained variance, symmetric percentage error.
What matters here is not "guessing the class," but not missing on scale — underestimating the forecast for large clients distorts planning more than variance on small ones.
The practical difference shows up in the data and the target variable.
For conversion, the target is binary, and the class balance and how the event horizon is labeled matter: conversion "within a week" and "within a month" are two different models with different predictive power.
For revenue, the target is continuous, with a long tail and outliers, so people often work with the logarithm of the sum or build a two-stage scheme: first a classifier separates those who will convert at all, then regression estimates the sum for those who converted.
In practice, these tasks often go together. The classifier filters out those who are clearly not targets, the regression calculates the expected value for the rest, and the business gets both a probability and a sum.
If you mix them into one model on one target, interpretability is lost: it's unclear whether the error is in the probability or in the scale of the sum.
That's why we first pin down exactly which question the result must answer, and only then choose the type of task and the model metrics.
Case study: how a conversion prediction model changed the funnel
The client — a sales department in the B2B segment — came to us with a classic problem: managers processed all incoming leads one after another, and conversion into a deal stayed at 4-6%.
Half the time went to obviously "cold" inquiries, while hot leads waited hours for a response and went cold. Manual prioritization didn't work: every manager had their own intuition, and there was no single criterion.
What we changed
We collected historical data on leads over several years: source, on-site behavior, industry, company size, touch history. We trained a binary classification model — it estimates the probability of a deal for each lead at the moment it arrives.
The score was embedded directly into the CRM: the manager sees the priority and works first with leads above the threshold.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Prioritization accuracy | manager's intuition | 78% on a holdout sample |
| Conversion into a deal | 4-6% | 11% |
| Time to first touch of a hot lead | 4-6 hours | 20-30 minutes |
| Share of leads processed by priority | — | 100% |
We also set up drift monitoring separately: if the distribution of input data shifts, the model raises a signal and goes into retraining. Without this, accuracy quietly degrades within a couple of months — a classic trap during implementation.
What this gave the funnel
Conversion nearly doubled without increasing headcount or the advertising budget — simply because efforts became focused on promising leads. Acquisition costs per deal decreased: the same flow of leads began producing more closed deals. Priority decisions are now made in seconds and are the same for all managers.
The model is built on CatBoost — it works robustly with categorical features and tabular data, which is exactly what makes up a lead's profile.
Doubts before implementation: compatibility, migration, rollback
Before implementing a conversion prediction model, the client usually asks three questions: how it will fit into the current stack, what will happen to the rules already in operation, and how to roll back if quality drops.
Let's answer directly: the model delivers its prediction via an API or batch calculation, so it does not require changing the CRM, the storage, or the frontend — an integration point where the numerical score is sent is enough.
The old business rules remain enabled and run in parallel until we confirm that the model delivers a gain on live traffic.
What if quality in production turns out to be worse than in testing?
We do not switch all traffic at once: first, the model processes a share of requests, and the metrics are compared with the current rule on the same audience. The switch is done through configuration — the previous logic can be restored without a code deployment, within minutes. The model is versioned, and the previous version remains available for instant rollback.
What if we have our own infrastructure and a closed environment? We deploy the model inside your environment, with no data leaving it. Training and inference run on your capacity; we hand over the configuration and retraining scripts.
What if the data is spread across different systems? We assemble features from the available sources through a data preparation layer — it is isolated, so adding a new source does not break the model.
After delivery, you receive:
- training and inference code with a description of dependencies;
- deployment configuration and integration points with the current systems;
- a feature schema: where each one comes from and how often it is updated;
- a model versioning policy and a rollback procedure;
- a retraining and quality control regulation on new data;
- a report with metrics on a holdout sample and an error analysis;
- instructions for the team: how to read the prediction and what to do when there are deviations.
Describe your task and current stack — we will come back with an estimate, an implementation plan, and the boundaries of responsibility.
Timeline for implementing a conversion prediction model by stage
The timeline for implementing a conversion prediction model depends not on the volume of historical data, but on how quickly an honest training sample can be assembled and agreement reached on the success metric.
That is why the work plan is built around stages rather than calendar dates: each next step starts when the previous one has produced a verifiable result, not when a date on the schedule arrives.
The longest part is not training but data preparation: if conversion events are written to different systems with different user identifiers, stitching them together takes more time than the model itself.
The opposite situation also happens — when the history is short or there are few target events, data has to be accumulated first, and this is the only stage that cannot be accelerated through engineering.
The estimated timelines above are for a project where data is already being collected and access to it exists. If event collection or labeling first needs to be set up, the work plan shifts, and we say so upfront rather than midway through. Describe your task — we will come back with an estimate and a plan tailored to your context.
