Why manual rollouts break production: bottlenecks without an operator
Manual canary rollouts depend on a human: an engineer opens a panel, switches the traffic share, watches the graphs, and decides whether to continue or roll back. While releases are infrequent, this works.
As soon as several changes a day go to production, a window of minutes appears between switching traffic and reacting, and all that time the faulty build lives on real users.
Rollback is also manual: while someone remembers the command and restores the previous image, some requests have already received errors.
The second problem is that metrics are not tied to the release. The metrics collector sees the overall error graph of the service, but does not know that a new version was rolled out a minute ago.
A spike in five-hundreds on the graph is indistinguishable from background noise or someone else's failure, and the decision is made by intuition rather than data.
Release observability falls apart: it is unclear whether the degradation came with the new build or existed before it.
The third bottleneck is blind traffic switching. The weight between the old and new version is changed by hand, without being tied to actual behavior under load.
If the new version keeps latency within normal at 5% of traffic but breaks at 50%, this is discovered only after a full switch.
The Flagger Kubernetes operator closes all three gaps: it runs the canary rollout itself according to defined steps, collects metrics from both versions itself and compares them, and when thresholds are exceeded, triggers an automatic rollback of the release without an on-call engineer.
Let's break down how this works and what needs to be configured so that the operator makes decisions correctly.
What managed progressive delivery provides
Progressive delivery stops being a manual ritual when it is driven by an operator: Flagger moves traffic between versions on its own according to defined metrics and rolls back the release if the indicators go beyond the threshold.
The engineer describes the target state in a manifest, and from there the system maintains it without the involvement of an on-call engineer.
What changes in numbers
The share of rollbacks falls not because releases have become more cautious, but because the problem is caught on 5% of traffic rather than across the entire fleet.
The class of failures "shipped it and found out from support requests" goes away: the canary step stops based on the error or latency metric, and the version is rolled back automatically.
Service recovery time is reduced from hours of manual investigation to minutes — the rollback is initiated by the operator based on observed data, not by a person responding to a call.
The cost of downtime falls because a faulty release does not reach the main audience. Managed release provides predictability: the rollout steps, pauses, and promotion conditions are fixed in the configuration, not in the head of the person responsible. The stability of the rollout stops depending on who is on shift today.
Why this works
The operator constantly compares the actual service metrics against the thresholds and promotes traffic only when they are met. If the indicators degrade, the return to the previous version happens without manual commands.
This removes the main source of long downtimes: the delay between the appearance of a problem and a human response.
Progressive delivery modes: comparison of approaches
The Flagger operator manages rollouts through the Canary object: it defines the strategy, promotion steps, and metrics used to decide whether to shift traffic to the new version.
The strategy determines exactly how traffic is distributed between versions and what happens when metrics deviate — from automatic rollback to a pause with manual confirmation. Below is a comparison of the modes by use cases and limitations.
| Option | When it fits | Limitations |
|---|---|---|
| Canary release | Gradual validation on real traffic is needed: 5–10% of requests go to the new version, metrics are checked at each step | Requires metric collection (success rate, latency) and two deployed versions running simultaneously; on an anomaly, traffic is returned, but some users have already been affected |
| Blue-green deployment | A critical service where mixed operation of versions is unacceptable; traffic is switched entirely after validation | Requires double the resources during the rollout; rollback is instant, but validation runs on a test environment rather than on production traffic |
| Traffic mirroring | You want to test the new version on a copy of real requests without affecting users | Responses from the mirroring version are discarded; side effects (database writes, external calls) must be isolated |
| Manual confirmation | Regulatory or high-risk changes where a human makes the decision | Automation pauses and waits for confirmation — the rollout will not complete without operator involvement |
Modes cannot be combined within a single object: Flagger applies one strategy per release. In practice, mirroring is used as the first step before a canary release, and manual confirmation is added as a final pause.
The choice of mode determines how many versions to keep running simultaneously and how to configure metric collection — the speed of rollback depends on this.
How we implement the Flagger Kubernetes operator
Implementation of the operator begins with an audit of the current rollout: we look at how a release reaches production, who checks what manually, and where rollbacks take the longest.
At this stage we identify failure points — manual checks, missing metrics, "blind" releases — and determine which services we migrate to progressive delivery first.
- Rollout audit. We break down the current process: update strategy, rollback time, who makes the promotion decision. The output is a map of services and priorities: where progressive delivery delivers the maximum effect, and where a canary launch on a single service is enough.
- Metrics and thresholds scheme. We determine which signals the operator uses to decide whether to promote or roll back a release: error rate, response latency, resource saturation. We set thresholds based on the real behavior of the service, not "by eye" — otherwise the operator will roll back stable releases.
- Operator setup. We install the Flagger Kubernetes operator into the cluster, describe the canary resources, and bind the metrics source and analysis rules. Separately, we configure the promotion steps: how much traffic goes to the new version, how long the pause lasts, and under what conditions the automatic rollback triggers.
- Testing on the staging environment. We run scenarios on the test contour: a successful release, degradation by metrics, a complete failure. We make sure that the rollback returns traffic to the previous version without manual intervention, and that alerts arrive where they will be seen.
- Handover of the procedures. We give the team a description of the process: how to ship a release, how to read the status of a canary launch, and what to do when a rollback triggers. Plus configuration documentation — so that changes to thresholds don't require us.
This order closes the main risk — automation for the sake of automation. First we measure, then we configure, then we test on the staging environment, and only then do we hand it over to the team. Describe your task — we'll come back with an estimate and a plan by stages.
Artifacts: configurations, schemas, and regulations
We hand over not "access to the cluster" but a complete set that your team can use to support releases without us. Each artifact is tied to a repository and goes through review just like application code.
- Flagger operator manifests: canary and blue-green for each service, binding to ingress or service mesh, step parameters — traffic share, pause between steps, number of analysis iterations.
- Traffic schema: primary/canary routing diagram, metric measurement points, binding to the dashboard — you can see where the share of requests goes at each step.
- Access policies: operator roles and service accounts, permissions to read metrics and write to target namespaces, namespace restrictions, integration with your authentication provider.
- Analysis configuration: set of success metrics (error rate, latency, saturation), thresholds and observation window, behavior when data is insufficient.
- Release documentation: how the rollout is launched, how to read the status, which alerts arrive when analysis fails, pre-start checklist.
- Rollback regulation: conditions for automatic rollback, manual rollback on command, procedure for notifying responsible parties, post-incident review.
- Maintenance guide: adding a new service, changing thresholds, updating the operator, common errors and their causes.
Everything is placed into the infrastructure repository with a description in the README, so that a new engineer on the team deploys the rollout by the document, not by colleagues' memory.
Case study: how rollbacks were cut by 70% with an operator
Before: manual rollouts and rollbacks "by eye"
A service of 40+ microservices, releases twice a day, every rollout manual: an engineer would trigger the update, watch the graphs in the on-call dashboard and decide whether to keep the new version.
The rollback rate held at 18% of all releases, and the average time from detecting a problem to reverting to the previous version was about 25 minutes. During that time, 4-6 errors in the logs and user complaints would pile up.
What was changed
Flagger was deployed as an operator in the cluster and rollouts were switched to a canary scenario with automatic metric analysis.
The operator itself spins up the canary version, routes part of the traffic to it and every 30 seconds compares key indicators — error rate, response latency, request success rate — against threshold values.
If a metric crosses the boundary, the rollback happens without human involvement: traffic returns to the stable version, the canary pods are deleted, and an event is sent to the monitoring system.
The thresholds and the analysis window were tuned to the service profile: strict error boundaries for critical paths, soft ones with a longer observation window for background tasks.
This eliminated the false positives that used to make the on-call engineer disable the automation and go back to manual control.
Result
The rollback rate dropped from 18% to 5% — roughly 70%. The average time to revert to the stable version fell from 25 minutes to 40 seconds: the operator reacts to a metric deviation before the on-call engineer even manages to open the dashboard.
The load on the team went down — releases run without constant supervision, and engineers get involved only in root-cause analysis, not in the rollback itself.
A separate effect is the predictability of rollouts. When the rollback is automatic and fast, the team stops fearing frequent releases: the number of rollouts grew, while the number of incidents per release fell.
Release metrics are now recorded in the operator's history, and they show which version rolled back and at which threshold — this speeds up the analysis.
How does Flagger differ from standard rollout tools?
Standard Kubernetes tools — Deployment with RollingUpdate, manual CI scripts, kubectl set image — only manage the delivery of a new version. They do not answer the main question: is this the version worth keeping in production.
Someone has to manually watch the graphs, compare metrics, and decide whether to continue the rollout or roll back. With dozens of releases per week, this decision is made by eye and lags by minutes.
Flagger is an operator that closes the loop: it doesn't just roll out, it analyzes the behavior of the new version itself and rolls it back if metrics exceed thresholds.
The operator spins up a canary or blue-green scenario, gradually shifts a share of traffic to the new version, and at each step queries the monitoring system — error rate, latency, saturation.
As long as the values are within normal range, the weight increases; when a boundary is crossed, traffic returns to the stable version automatically, without an on-call engineer.
The difference shows up at the moment of failure. A manual script will stop where a human stops it — and that human might be asleep or dealing with another incident at the time.
The operator reacts within seconds and according to a predefined rule: how many errors count as a failure threshold, how many times in a row the metric must exceed it before rolling back. This turns rollback from an "by eye" decision into a reproducible procedure.
There is also a second line of differences — manageability. Basic tools cannot handle multiple parallel scenarios, align them with policies, and leave a trail for investigation.
The operator is described declaratively, stores the history of rollouts and statuses, so incident investigation proceeds by facts, not by recollection. It is precisely the automatic analysis of metrics and rollback based on them that standard tools lack.
Common technical concerns during implementation
Flagger is a Kubernetes operator that manages rollout through custom resources: it subscribes to changes in the cluster and switches traffic between versions according to defined rules.
Therefore, the first thing we check at the start is compatibility with the current cluster: which API version is in use, whether there are permissions to create CRDs and work with the ingress controller or service mesh.
If the cluster is managed and some permissions are closed, we build the rollout scheme through the available entry points rather than breaking the existing architecture.
The second concern is the impact on load. The operator does not proxy traffic and does not add its own layer to the request path: it only changes routing weights and monitors metrics.
The overhead comes down to periodic metric polling and weight recalculation, not to processing every request.
The third is the requirements for metrics. Automatic rollback requires measurable signals: error rate, latency, saturation. If there are no metrics, the rollout proceeds with manual confirmation — this is a workable mode, but without auto-rollback.
What happens if the cluster has no service mesh or advanced ingress?
Flagger can also work with basic controllers through the available routing mechanisms. During the assessment stage, we look at which traffic switching points exist in your cluster and select a rollout mode to match them — without rebuilding the entire infrastructure.
How difficult is it to maintain such a scheme going forward?
The main work is to configure the rollout rules and metric thresholds once; after that, the operator acts on its own. Maintenance comes down to updating thresholds when the load profile changes and ensuring that the metrics remain reliable.
Can we roll back if the new version behaves badly?
Yes, rollback is a standard scenario: when metrics cross the threshold, the operator returns traffic to the previous version automatically. Additionally, we set up manual confirmation for critical services where automation is undesirable.
Let's discuss your task and come back with a plan
For the estimate to be realistic rather than "based on gut feeling," we need to understand your current delivery contour: how the cluster is structured, who publishes releases and how, whether there is already a service mesh or an ingress controller that can manage traffic.
These details determine whether we put Flagger on top of an existing mesh or stand up a canary layer from scratch, and which metrics will be rollback triggers.
Next, we clarify the process: how many services are in the change zone, how often you deploy, who makes the promotion decision — a human or automation based on metrics.
This determines the depth of setup: from basic canary rollout to blue-green switching with progressive traffic shifting. If there are strict requirements for release windows or business approval, we build them into the rollout policy right away, not after the first incident.
What we need from you to come back with a plan:
- access to the description of clusters and the current delivery scheme — configuration repositories, manifests, environment diagrams;
- a list of candidate services and their criticality: what can be rolled out canary-style and what requires instant rollback;
- the metrics you already use to make decisions — latency, error rate, saturation — and where they are collected;
- the current release process: who publishes, who approves, what manual steps exist now;
- rollback requirements: how quickly the system must return to the previous version and who triggers it;
- access and security constraints: what roles the team has, what can be changed in the cluster.
Based on the outcome of the conversation, we come back with a work estimate, a draft phased implementation plan, and a list of risks we see in your infrastructure.
Describe the task in free form — what hurts now and what result is needed — and we will ask clarifying questions and propose next steps.