Setting up A/B testing of ML models in production

Why A/B Testing of ML Models Breaks in Production

The most expensive failure in A/B testing of ML models is not a service outage, but a confident response from the wrong model.

The registry contains the needed version, but the input features are assembled by different code, reference data is updated separately, and the model version mismatch between environments surfaces weeks after launch.

The second failure is metrics. Testing models in production compares control and experiment on different slices: in one environment the aggregation window is wider, in the other outliers are filtered differently.

Unstable metrics produce variance greater than the model effect itself, and the experiment is closed based on noise rather than the result.

The third is non-reproducibility. A month later, no one can reconstruct which features, reference data versions, and thresholds were in place at the time of measurement.

Even when artifact versions are logged in MLflow, the registry stores the model, not the full context of its run — there is nothing to compare experiments post hoc with.

And the fourth is model rollback. If the previous artifact is not saved together with the configuration, reverting turns into manual assembly under pressure: quality is dropping right now, and recovery takes hours. The cost of delay is measured by the share of corrupted user decisions during that time.

We break down how this is closed: immutable versions of the model and features, a unified metrics computation layer for both environments, and rollback with one operation.

Measurable Effects: Latency, Rollbacks, Share of Manual Checks

A managed environment changes the cycle itself: the hypothesis is tested on control and treatment groups before the version goes out to the entire audience.

Switching the traffic share becomes an operation in the experiment settings, not a separate release, so the time from hypothesis to model rollout shrinks from days to hours — approvals are replaced by a process.

The share of manual reconciliations drops for the same reason: metrics are calculated on one labeled slice for both groups, effect significance is checked automatically, and the analyst looks only at disputed segments.

What previously required daily exports and manual table consolidation becomes a report that updates itself.

A separate class of failures also goes away, the most expensive one: "silent" degradation on a narrow segment and rollout of a version trained on an outdated slice.

A threshold quality metric stops the rollout and returns the previous version automatically — recovery time is measured in minutes, and downtime due to an error stops being an all-day event.

MLflow in this scheme is responsible for reproducibility: each model version is tied to parameters, data, and metric, so the experiment result can be rechecked, and rollback is switching to a fixed previous version, not rebuilding. The effect of the experiment becomes measurable before rollout, not after an incident.

Rollout Options: Shadow Launch, Canary, Phased

The release mode is chosen not for convenience, but by what share of traffic may be given to the new model and how costly an error on a live request is.

A shadow launch answers the question "does the new model agree with the current one," canary rollout — "is the product degrading on real users," phased release — "which option wins on the metric."

The limitations of the modes come down to request routing and the method of metrics collection. Until a metric is tied to the model version and cohort, any comparison of approaches turns into a set of guesses.

On practice, modes are combined: shadow launch as admission to live traffic, canary rollout as insurance, phased release as a way to collect statistics by segments.

Implementation Stages: From Audit to Handover to Operations

We structure the order of work so that an error in the scheme is visible before launch on real traffic, not a week later when analyzing metrics. Each stage ends with a verifiable artifact — then enabling does not turn into "we turned it on and are watching."

1. Audit of the experiment design. We look at which metric is considered the target one, where and on what volume of traffic it is collected, and whether there are enough observations to distinguish versions.

   We check whether the groups intersect with other running experiments and whether there is bias in how events enter the storage.

2. Traffic allocation. We determine the randomization unit — user, session, or request — and the assignment method: the same feature gives a stable group on repeat visits.

   We fix the traffic share for the experiment and exclusion rules so that service requests do not enter the comparison.

3. Environment setup. We deploy the comparison service: it accepts a request, routes it to the required model version, and writes the prediction and version feature to the log. Shadow mode allows running the new version on live traffic without affecting the user response.

4. Control checks. We run an A/A test for group homogeneity, compare the feature distribution between branches, and check behavior when one of the versions fails and when the comparison service is unavailable.

   We calculate the metric on both branches using the same rule — otherwise the difference will be in the calculation, not in the model.

5. Handover to operations. We give the team the launch and stop procedure, rollback description, metrics dashboard, and alert settings. We document the allocation and environment configuration so the next experiment can be launched without us.

Handover Artifacts: Diagrams, Access Policies, Procedures

After the environment is handed over, the customer's team has a set of artifacts by which new A/B tests can be launched independently — without reconstructing logic from messages and engineers' memory. Below is the list of what we hand over and why each document is needed in work.

• Environment configurations. Deployment files and environment parameters: test and production differ only in values, so a new model variant is brought up by copying the configuration, not rebuilding it.
• Data flow diagrams. The path from request to metric recording: where the variant is fixed, how the target metric is calculated, where samples diverge. With the diagram, a new engineer understands the environment in an hour.
• Access policies. Role separation: who changes the traffic share for a variant, who stops the test, who has access to raw logs. The experiment cannot be broken by an accidental edit.
• Experiment procedure. Required steps: hypothesis, target metric, minimum sample size, stopping criterion, and procedure for rollback to the control model.
• Report template. A single form for recording the result: versions of compared models, period, metrics by groups, conclusion and decision — continue, refine, or close.
• Instructions and training. A walkthrough of the environment with the customer's team and short guides on typical operations: starting a test, stopping, rollback, reading a report, adding a new metric.

Such a set removes dependence on the contractor: changes to the environment are made according to documented rules, not oral agreements. Let's discuss the task — we will come back with a work plan and a set of artifacts for your environment.

Case: Eliminated Version Mismatch Between Environment and Production

The customer rolled out several models per month, while the training environment and production lived separate lives. Artifacts were transferred by scripts, features were rebuilt manually, and version mismatch between environments surfaced already on live traffic.

Blind rollout without audience splitting did not allow degradation to be noticed before product metrics sagged.

Before restructuring, two or three manual reconciliations per week took half a day from an engineer, recovery after an incident stretched to more than an hour, and rollback had to be triggered manually in almost a quarter of rollouts.

What Changed

Both environments were brought to a single model registry on MLflow: version, feature set, and quality on the holdout sample are tied to one artifact, and transfer to production goes only through automatic manifest validation.

Rollout was switched to 5/95 traffic splitting with automatic rollback by the key metric instead of blindly replacing the entire model. Decisions are logged together with the version, so incident analysis starts with a ready picture, not manual log reconciliation.

A quarter later, manual reconciliations were reduced to one planned one, version mismatches between environment and production were never reproduced, and recovery took minutes: rollback triggered automatically by threshold. The share of paths covered by automatic pre-rollout checks grew from a third to nine tenths.

What to Do If the Model Is Already in Production but There Is No Labeling?

The absence of labeling for an experiment blocks not the comparison of versions, but the absence of a controlled switch between them.

If the model is already in production, first separate it from the service code: the model version and its parameters must be supplied from configuration, not hardcoded in the call.

Then a thin routing layer appears at the service input, which decides by one request feature which version will process it.

While there is no labeling, we rely on indirect signals: decision logs, user response, share of manual corrections after the model response, distribution of predictions by class, and drift of this distribution over time.

This is enough to catch degradation and version mismatch, while a full set of validation examples is created later — already for the collected stream of real requests.

We make the transition gradual: single-digit percentages of traffic to the new version, holding at the chosen share, then a step up.

At each step we compare the same signals that were collected before launch, and keep rollback ready — returning the version by configuration, without rebuilding and stopping the service.

Migration without downtime comes exactly from this: routing lives separately from the model, so replacing the version takes seconds and does not interrupt request processing.

We set the stopping threshold in advance and in measurable values: target metric drop, growth in error share, distribution divergence above acceptable. If the threshold is crossed, traffic returns to the previous version automatically, not by a call from the on-call engineer.

Common Technical Concerns of Customers and Engineers

Concerns at this stage are usually technical: whether there is enough data, whether seasonality distorts the picture, what to do with parallel changes to rules and the model. Below are short answers on how we close these points in experiment allocation.

Will seasonality distort the picture?

Splitting is based on a stable user identifier, so both groups fall into the same days, promotions, and peak hours. Comparison runs in parallel in one window, and we recheck the result on adjacent periods — an effect that holds only in one week does not go into conclusions.

What to do if rules and the model change at the same time?

One experiment — one change. If both are changed at once, the effect cannot be separated: it is unclear what produced the gain and what ate it. We separate edits into sequential stages or plan a scheme with testing of joint influence to estimate the contribution of each factor.

Who has access to results and what about data security?

We view reports by role: the product team gets slices by segments and metrics, engineers get events in storage for analysis. Personal data does not enter reporting, identifiers are pseudonymized, and export is limited by access policy and logged.

Let's Discuss the Task: How We Will Start Work on the Environment

So that the A/B testing environment for models does not turn into a series of reworks, we start by analyzing what already works.

From you — access to the infrastructure repository, the current scheme for bringing models into production, and an export for the target metric for past periods: without this, we cannot calculate group size and observation window duration.

Order of first steps:

• describe the current model path: where versions are stored, who switches traffic between them, what is already logged;
• fix the target metric and guardrail metrics that must not sag under any test outcome;
• choose the randomization unit — client, session, or request — and the method of assigning it to a group so the user does not jump between variants;
• agree on the calculation scheme: traffic shares, minimum detectable effect, early stopping criterion;
• design rollback: with one command, all traffic returns to the control model in a matter of minutes;
• add monitoring of input data drift and distribution divergence between groups;
• scope the work: where changes in the inference code are needed, and where configuration is enough.

Based on the analysis, we will come back with an estimate and plan: composition of stages, timelines for each, responsibility area of our team and yours. The next step — describe the task, and we will schedule the analysis.