Canary deployment ML: why rolling out a new model breaks production
A new model version goes to production as a single tag for all traffic — this is what the most common rollout failure looks like. Offline checks are passed, the holdout set shows the required quality, and the release goes to 100% of requests.
Half an hour later, monitoring shows prediction degradation: the scoring failure rate is rising, rare segments have dropped more than the others, and the logs already contain new model responses for all clients.
There is no way to roll back such a rollout. If the release is published over the previous tag in the image registry, the previous artifact is overwritten, and “returning to how it was” turns into rebuilding from source and revalidating.
While this is happening, the production environment responds with a model no one trusts, and every hour of operation costs user trust.
The point is not that the model is bad. The production environment differs from the test one: a different request stream, feature drift, uneven segments, latency from external data sources.
A metric on a static dataset says nothing about behavior on live traffic — therefore, production risk is limited before rollout, not analyzed afterward.
Canary deployment is exactly about this: the new version receives a small share of traffic, usually 1–5%, predictions are compared with the reference model by segments and distributions, and a deviation above the threshold automatically returns the previous version.
Traffic splitting and rollback rules are described declaratively — in orchestrators such as Kubernetes, this is the level of manifests, not manual actions by an on-call engineer.
Below, we break down how such a loop is assembled and what needs to be fixed before the first rollout.
Managed model rollout: measurable product impact
In a canary rollout, it is not only the release procedure that changes — what the team sees and has time to react to changes as well. Below are the effects that appear already in the first cycles, and the metrics used to verify them.
Error response rate and model observability
While the new version receives 1–5% of traffic, its metrics are calculated separately from the stable one: failure rate, prediction distribution, latency, input data shift.
Comparison is not “by eye” but against a reference slice fixed before the release — the deviation is visible on the first hundreds of requests, not a week later from user feedback.
Observability covers what is missing from ordinary service monitoring: response quality, not just the fact of a response. Collection and breakdown of metrics by model version are built on Prometheus with alerts on the deviation threshold.
Rollback time and the eliminated failure class
Returning the weight to the stable version is a routing configuration change, not a rebuild and redeploy: rollback takes minutes and does not require the whole team.
An entire class of failures disappears — mass quality degradation after release, when all users get bad responses at once.
At the same time, the traffic under observation is not zero: the canary share provides enough statistics for the decision to roll back or expand to be made from data, not intuition.
The cost of failure decreases not due to “more careful releases” but due to a limited blast radius: the error remains on a few percent of requests and is extinguished before it reaches the entire audience.
Traffic routing schemes for ML: comparison of modes
The choice of scheme is determined by what exactly is being tested: service resilience or the quality of model responses. Below are four modes that teams actually choose between, and their limitations.
| Option | When it fits | What limits it |
|---|---|---|
| Canary rollout by traffic share | Need validation on live users at controlled risk: the new version receives a small share of requests, then the share is increased based on metrics | Requires automatic rollback by metrics, otherwise the share grows “by eye”; on rare response classes, statistics accumulate slowly |
| Shadow mode | A copy of production requests goes to the new model, responses are not returned to the client — quality is compared on the real distribution | Double load on the model service; cannot assess how a new response affects user behavior |
| Blue-green switching | Need instant transfer of all traffic after checks, e.g. with an incompatible response format change | No intermediate shares: rollback is a full return, and the new environment needs warming up |
| Manual switching | Rare releases and incident analysis, when an on-call engineer deliberately makes the decision | Depends on a human: no share gradation, reaction time equals time until someone notices the problem |
What routing rules solve
Traffic share is not the only lever: distribution is set by rules based on client ID or request header so that a single session does not jump between versions.
For models, this is fundamental: the user immediately notices divergence in responses from two versions, and quality metrics blur if different cohorts are compared. Therefore, the share is usually fixed on a cohort, and comparison is performed on comparable traffic slices.
How we implement managed rollout: stages of engineering work
Rolling out an ML model differs from releasing an ordinary service: the new version may respond quickly and without errors but produce lower-quality predictions.
Therefore, the traffic share is tied not only to availability but also to quality signals, and switching is done in steps.
- Rollout audit. We analyze how model versions get to production: where artifacts are stored, how the active version is selected, whether part of the traffic can be diverted to the new one. We note places where there is no way back.
- Routing scheme. We determine where the stream is split — load balancer, gateway, or inference service — and how the traffic share is set: changed on the fly or requiring a restart.
- Model quality signals. We agree on metrics for comparing versions: inference failure rate, response time, prediction distribution, business metric on a labeled slice. For each — threshold and minimum observation volume.
- Switching rules. We fix in configuration the starting share, step and pause between stages, conditions for automatic rollout stop.
- Traffic validation. We run the canary version on a small share, compare signals with the baseline, and increase in steps. To quickly accumulate statistics, we add a shadow run on historical requests.
- Rollback procedure. We specify which signal and whose trigger returns the previous version, how long rollback takes, and how integrity is checked after it.
- Handover. We hand over the routing scheme, rule configuration, metrics dashboard, and rollback procedure, and conduct a review with the team.
Deliverables: configurations, schemes, and rollback procedures
Setting up canary deployment for ML models is not a one-time config edit but a set of artifacts that remain with your team. After a month of support, engineers on the customer side change traffic shares, read metrics, and roll back versions without our help.
- Rollout configurations — files describing versions, traffic share increase steps, and thresholds at which the rollout stops automatically.
- Data flow diagram — from the incoming request to prediction logging: where traffic splits into two versions, how responses are compared, where features are stored.
- Access policies — role separation: who changes weights, who sees metrics, who has the right to roll back. Separate visibility scopes for production and test environments.
- Observability — dashboards for model quality and service metrics, alerts on divergence between the stable and canary versions.
- Rollback procedure — trigger criteria, action sequence, decision owner, and target time to return to the stable version.
- Documentation — deployment instructions, pre-release checklist, description of environments and typical failures.
- Knowledge transfer — architecture walkthrough with your team and a recorded session so that support does not depend on us.
Case study: migrating a recommender model to canary rollout
Before implementation
The recommender model was updated once every two to three weeks, and each rollout went to all traffic at once — there was no intermediate validation on a live audience.
Quality in production was not measured continuously: they relied on offline measurements before release, and learned about degradation from a drop in orders, most often at night.
The on-call engineer rolled back manually — an hour and a half to two hours for diagnostics and version rollback, three to four such cases per month.
What was changed and what the result was
Traffic was split: 5% to the new version, then 25%, then the entire stream — progression only when conditions were met.
Online quality metrics (click-through rate on recommendations and the share of relevant products in the first results) are compared with the control group every few minutes.
The quality threshold is set in advance: a metric falling two percentage points below the control or a rise in the failure rate triggers an automatic rollback. Kubernetes switches the weights — return to the previous version happens without human involvement.
Metrics before and after: night incidents — there were three to four per month, now zero for a quarter. The release cycle shrank from two to three weeks to one, because validation on 5% of the stream takes hours and does not require manual approvals.
Rollback — from two hours of on-call work to several minutes by signal. The share of rollouts that reached full traffic without rollbacks grew to about nine out of ten: the model reaches users faster, and degradation is cut off on a small share of the audience.
How does canary rollout differ from shadow mode?
Shadow mode runs the new model on a copy of production traffic: it computes predictions in parallel with the current one, but its responses are not sent to the user.
There is no impact on users — requests are served by the previous version, and deviations are calculated on the same inputs. The comparison is fair in terms of data but one-sided.
The main limitation of shadow mode is that there is no model feedback. We see errors, latency, and input data drift, but we do not see how people's behavior changes with new responses.
The share of accepted suggestions, repeat requests, and session depth remain out of the picture — shadow mode catches technical degradation but does not answer whether the product has improved.
Canary rollout gives the new model a real share of traffic — usually a few percent — and increases it as metrics are confirmed.
There is an impact on users here, but genuine feedback appears: product metrics are calculated on the same requests as before, and they can be compared directly with the previous version.
The choice of scheme is determined by the cost of an error and the reversibility of the result. If the model response is checked by a human or is not visible to the client, they start with a small canary rollout share.
If the error is noticeable and hard to roll back, they first run shadow mode on real data, and only then open traffic. Rollback conditions are set in advance and trigger automatically — then comparing modes does not turn into manual monitoring of charts.
Technical concerns: rollback, data quality, load
Before enabling canary deployment on a production ML service, the same concerns come up: who rolls back and how, how to separate model degradation from data seasonality, and whether the service will have to be reworked.
Is manual rollback slow and risky?
Manual rollback is safe if it is switching the traffic share to the previous model artifact, not rebuilding the service. Both versions live simultaneously, weights are changed by a single config, and the return takes tens of seconds.
The right to press the button is assigned to a role, and an auto-trigger by quality metrics provides a safety net.
How do you distinguish model degradation from data seasonality?
We calculate the metric on the same slice: both versions receive requests in the same proportion by channels, regions, and hours. In parallel, we monitor input data drift — a shift in the feature distribution.
If the inputs have shifted and quality fell afterward, this is a mismatch between the training set and the production stream, not seasonality.
Does the service need to be reworked for traffic splitting?
Usually not. If the service does not store state between calls and the model is delivered as a separate artifact, splitting is done by the gateway — by share or service header.
Rework is needed where the version is hardcoded or inference runs inside a long-running process with accumulated state.
How does the canary version affect load?
It takes a share of requests, so its pool is calculated separately, with headroom for peak load.
If resources are tight, we enable shadow mode: the version receives a copy of requests without affecting the user response, and quality is calculated offline — the load on the service remains predictable.
Let's discuss your rollout task: we'll come back with an estimate and a plan
An implementation estimate is accurate when there is source data. Describe the task: what the service is, how the new model version is currently rolled out, and what counts as a failed release in your view. We will compare it with your environment and come back with a work plan, not general wording.
- Service scheme: where traffic comes from, where the load balancer is, where model responses and logs are written.
- Current rollout procedure: manual or automatic, who switches the version and by what command.
- Quality requirements: which metrics count as degradation and at what request volume it is already visible.
- Rollback rules: within what time the version must be returned and who makes the rollback decision.
- Environment constraints: where data lives, whether it is permissible to send part of the traffic to the new model.
- Team composition and change windows so that the work plan does not conflict with current releases.
- Access and roles for observability setup — who looks at metrics and who responds to them.
From this, we assemble a work plan with stages, roles, and control points, and we fix next steps after a short review of your environment. Write to us and discuss the task — we will come back with an estimate and a plan.