Canary Deployment in Kubernetes: Safe Rollout with Argo Rollouts

Canary deployment: where production rollout breaks

Classic production rollout sends a new version to all traffic at once: replicas with the new image come up, old ones are shut down, the load balancer switches everyone at once.

If there is an error in the code — an incorrect query to the data schema, a connection leak, a failure on a rare scenario — all users get it simultaneously. The first support requests arrive before graphs have time to show growth in the error rate.

Then comes the most expensive part — rolling back the release. Returning to the previous image is fast in itself, but if a schema migration went out with the code, rollback is no longer complete: data is written in the new format, some handlers do not know about it.

Add warm-up, readiness checks, and redeploy — and an unsuccessful update stretches into hours of manual decisions.

An error in new code is rarely total. More often one scenario breaks: payment for some regions, a rare input file format, a branch on an empty field.

The difference between “broke for everyone” and “broke for a few percent” is the difference between an outage and unnoticed degradation learned about from metrics, not complaints.

We break down how this is covered. Canary deployment on Argo Rollouts moves a small share of traffic to the new version first, compares the error rate and response time against the baseline, and rolls back automatically — without waiting for on-call and without a rebuild.

What gradual rollout to part of traffic gives

Gradual rollout is transferring risk from the release to a small share of traffic. The new version gets weight not all at once: 5%, then 10, 25, 50 — with a pause at each step.

While the Argo Rollouts controller sees that the canary and stable environment metrics are comparable, weight grows; on deviation, traffic returns to the previous version.

Step checks are set declaratively: share of error responses, 95th percentile latency, number of container restarts. While thresholds are met, rollout continues on its own, without manual confirmation.

Rollback time is measured in tens of seconds, because rollback is a weight change, not a rebuild and redeploy.

The main thing is the eliminated class of failures. A defect in code or configuration affects a small share of users instead of all, and investigation remains a local incident, not a general outage with downtime.

The share of affected users becomes a manageable value: it is set in advance and changed during rollout.

Release observability grows along with this: requests, logs, and traces are labeled with the version, so you can see what share of load the canary environment carries and how it affects the overall picture.

The release dashboard shows weight changes and triggered checks on one time scale — it shows which step went smoothly and which caused rollback.

Hence the choice of channel width: first release to an internal segment or one region, then expand to full weight. This scheme does not replace tests, but removes the main cost of error — mass impact.

Canary deployment vs blue-green and full rollout

All three schemes solve one task — deliver the new version to users — but manage traffic and the cost of error differently. Full rollout changes all instances at once: simple and predictable, but regression instantly reaches the entire audience.

Blue-green keeps two environments and switches traffic entirely: rollback takes seconds, but the new version immediately gets full production load.

Gradual release wins where there is observability: weight grows from fractions of a percent to one hundred, and regression is cut off at the first percentages of traffic, while only a few see it.

If traffic is small and metrics are noisy, small weight does not provide statistics — then blue-green rollout or full release with fast rollback is more honest. The selection criterion is not fashion, but metrics, rollback automation, and compatibility of schema migrations.

Stages of implementing gradual rollout in releases

Gradual rollout is not bolted on top of the pipeline — it relies on how build, configuration, and observability are arranged. Therefore implementation stages go from an audit of the release process to a regulation by which on-call rolls back a version without the release author.

1. Audit of release process. We look at how images are built, how environments differ, whether there are metrics and tracing per service. Without a baseline for errors and response time, there is nothing to compare canary deployment against.
2. Preparation of manifests and labels. The application gets stable and canary pod sets, traffic is divided by weights through a managed object like Argo Rollouts, the version is visible in labels and logs — otherwise an incident cannot be distinguished from the previous release.
3. Testing on pre-production environment. We run the same route on a copy of production: weight steps, pauses, analysis. Here we also test rollback — it should return traffic and pods, not leave a half-state.
4. Configuring metrics and thresholds. We describe what we consider degradation: share of error responses, 95th percentile latency, resource shortage. We derive thresholds from observed behavior, otherwise automatic rollback triggers on noise.
5. Trial on a real release. We conduct the first rollout with manual step confirmation, an engineer watches metrics. This shows real warm-up speed and which alerts are noisy in vain.
6. Handover of regulation. We give the team a document: steps and pauses, automatic rollback conditions, on-call actions, manual return procedure. The canary stage becomes a mandatory item in the release checklist.

Configurations, traffic schemes, and handover regulations

Implementation ends not with a working rollout, but with a set of artifacts by which your team itself changes parameters and investigates failures.

All of this stays in your repository and cluster: configurations under version control, access within your perimeter, instructions tied to process steps, not specific people.

• Rollout configurations — description of the Rollout object: share of traffic on the canary version, pauses between steps, automatic promotion conditions. Edits go through merge request, not manually in the cluster.
• Traffic scheme — how the load balancer and Service divide requests between stable and new version, including session stickiness and header-based routing for internal checks. We hand it over as a diagram and working manifests.
• Access policies — roles for promotion and rollback of rollout, separation of rights between development and operations, access to metric dashboards, prohibition of manual changes in the production namespace.
• Rollback regulation — what we consider degradation, by which metrics, who makes the decision, which commands are executed and in what order. Automatic return by error threshold is described separately from manual.
• Metric analysis rules — thresholds for request success, latency, and error rate, observation time at each step, behavior when data from the metrics service is insufficient.
• Incident documentation — investigation template: at which rollout step errors grew, which metrics showed this, what was changed in configuration. Accumulated investigations set the next step settings.
• On-call instructions — short scenario: how to pause rollout, how to return the previous version, whom to notify and at what interval.

Case: share of users affected by failure dropped 20-fold

What was before implementation

The customer released microservice updates entirely: the new version came up on all production pods at once. Any error in the build affected 100% of active users, and the signal came from outside — from support requests, not metrics. Rollback was started manually, it took up to 40 minutes, and the failure continued all that time.

What was changed

We moved rollout to a gradual scenario in Argo Rollouts: the new version first received 5% of traffic, then 20% and 50%, and only then 100%. At each step, the system compared error rate and response latency with the baseline and held the release if a metric went beyond threshold.

Steps, thresholds, and pause conditions were described in the manifest next to the service — rollout became part of the repository, not a manual operation.

What became

The share of users affected by failure dropped 20-fold: no more than 5% of the audience managed to see the problematic version. Recovery time fell from 40 minutes to 2–3 — rollback triggered automatically by metric, without an on-call engineer.

Over a quarter, not a single defective release reached full traffic, and the team stopped fearing releases at the end of the week.

What to do if releases in the cluster are already manual

Transition in an already deployed cluster is done not instead of manual rollout, but alongside it: a new perimeter is brought up in parallel, some services continue to release via the old scheme, the rest are moved to canary deployment with a gradual traffic share.

Until the new version proves itself in checks, the previous path remains alive and serves as rollback — releases do not have to stop for a single day.

Migration of the release process begins with inventory: which services are deployed, which manifests correspond to them, who presses the button and what is checked afterward.

Without this picture, step-by-step transition turns into rewriting the cluster entirely — exactly the scenario where production breaks.

Then services are moved one at a time, starting with the least critical.

A rollout object is added to the manifest, describing steps and promotion conditions, readiness checks and metrics are connected to it, the share of traffic to the new version grows in small steps, and each step is confirmed manually.

The result is visible in the same week: the version gets real load, and the promotion decision remains with the engineer.

Reversibility here matters more than speed. If metrics drop, traffic returns to the stable version with one command — minutes instead of hours of manual rebuild. When several releases in a row pass without incidents, manual confirmations are removed selectively, not all at once.

Doubts about metrics, rollback, and load on prod

Rollback thresholds are not guesses, but parameters we agree on before rollout. Release metrics: error share of the new version relative to baseline, latency at the 95th percentile of requests, and share of successful business operations.

While the canary version holds 5–10% of traffic, exceeding threshold automatically stops rollout and returns traffic to the stable version.

Below are doubts that most often surface during approval.

Who decides to stop rollout?

Automation by thresholds: exceeding error share or latency returns traffic to the stable version without on-call involvement. Manual stop remains — it is needed when something visible from outside is not in metrics, for example complaints about a specific scenario.

What are the limitations of the approach?

Canary deployment is useless if traffic is too small for statistics, if there is no observability for the required metrics, or if the release changes data format incompatibly.

Background jobs and queues are divided not by the load balancer but by the handlers themselves — for them we build a separate check based on share of processed tasks.

Let's discuss the task: we'll return with estimate and plan

To make the conversation substantive, it is enough to bring the current state of change delivery: how a release gets to prod today, whether there is a separate perimeter for checks, which metrics are already collected from the application.

Some data may be missing — we will formulate infrastructure requirements and rollout rules together, based on what exists.

In response you get a working frame, not general considerations: timeline estimate by stages, implementation plan with control points, and a scheme for dividing traffic between versions. Below is what is usually needed from the customer and what we prepare on our side.

• Access: read access to the cluster, manifest repository, and image build system — this is enough to analyze the current process without changing anything.
• Current rollout method: where exactly the decision to move to a new version is made — manually, by script, or by pipeline rules.
• Observability: which metrics are collected, where they are written, and whether two versions can be compared by the same rules.
• Infrastructure requirements: spare resources for two versions at the same time, a separate perimeter for checks, traffic management at cluster ingress.
• Rollback rules: by which indicator a rollout is considered unsuccessful and how long it takes to return to the previous version.
• Roles and boundaries: who expands traffic share, who is on call, and how control is handed over after launch.

Describe the task in free form — we will return with an estimate and implementation plan where the order of steps, scope of work, and areas of responsibility are visible.