Blue-Green Deployment: why releases break production
Classic picture: one environment, one circuit, release rides on top of the running application. Traffic switching is done manually — an engineer edits configuration or restarts processes while some users already hold open connections. Every release turns into a downtime window whose duration no one dares to name in advance.
The flip side is release rollback. There is no ready backup of the previous version: images are overwritten, the data schema is migrated forward and does not roll back.
Returning to the past state is possible only with a new build and re-running migrations — instead of a minute, you get hours, all on a live environment.
Combined environments add noise: test runs, an acceptance environment, and production live on one cluster, resources are shared, and a spike from checks hits user traffic. Downtime stops being an emergency scenario and becomes a normal release mode.
The cost of an error is not only downtime. The team starts fearing releases: deployments are moved to night hours, release frequency drops, fixes wait in line, feedback lags by days.
We break down how this is closed: what a parallel environment gives, constant readiness of the second environment, and predictable zero-downtime traffic switching.
What switching between two environments gives
The mechanics are simple: two environments live in parallel — active and prepared, and traffic is directed to one of them by a Service switch or an Ingress rule.
Rollback is switching the selector back, not rebuilding: rollback time is measured in seconds and does not depend on image size or the number of pods. The previous version remains running and warmed, so the return is not accompanied by an error spike from a cold start.
Environment readiness and pre-release verification
Before switching, the new environment must prove readiness: readiness probes succeed, the application responds on the service route, smoke checks run on key paths, and if needed, a load test on one pod.
Argo Rollouts adds metric analysis directly into the release process: until the environment passes the specified threshold for errors and latency, traffic is not routed to it. Manual confirmation is left where the release affects payments or data.
What this gives in measurable terms
Downtime during release goes to zero: traffic switches between already running pod sets, and there is no downtime window. The share of successful releases grows because verification happens before all traffic is routed — a failed version simply does not become active.
Rollback time is reduced to the switching operation, and it can be managed without an on-call engineer and without re-running checks.
Separately, an entire class of failures related to "rolled back the code — lost the data" goes away: the database schema is changed so it is compatible with both versions, and the return becomes safe.
Comparison of release schemes: blue-green deployment and canary
A release scheme is a compromise: the lower the risk, the more resources, time, or infrastructure it requires. In Kubernetes, teams choose among four options, and each limitation shows up not in a demo but in the third release with data migration.
| Option | When it fits | Limitations |
|---|---|---|
| Blue-green scheme | Instant rollback by traffic switch is needed; stateless service or data compatible both ways | Double resource reserve during release; switching breaks long connections and sessions without external storage |
| Canary release | Traffic is large and heterogeneous, metrics exist for automatic rollout decisions | Rollout and rollback take time; reliable routing by weights or headers is required |
| Shadow traffic mirroring | You want to test a version on real traffic without risk to users | The copy does not affect responses — comparison is built separately; side writes and external calls are isolated |
| Rolling update | Simple stateless services where replacing pods one by one is enough | No environment to test the release as a whole; versions run simultaneously, rollback is also gradual |
The schemes are not mutually exclusive: Argo Rollouts guides the release in steps with pauses and automatic rollback by metrics, so a blue-green scheme is supplemented by canary rollout, and shadow mirroring is placed before traffic switching — this way, the limitations of each scheme are covered by the neighboring one.
Implementation order: from scheme analysis to handover
We build the order of work so that changing a version becomes a routine operation with predictable rollback, not a risky event. At each step of blue-green deployment implementation, the customer keeps an artifact: a scheme, configuration, or runbook.
- Review of the current release. We look at how a version enters the cluster today: manifests, image owners, probes, actual rollback time. The result is a risk map and a list of changes in the environment.
- Scheme design. Two environments with independent pod sets, shared data access, and a strict rule: exactly one version serves traffic.
- Routing setup. We switch service selectors and ingress controller rules, and check that connections close correctly and requests are not lost.
- Argo Rollouts setup. We describe the release object with references to the active and preview services, set probes, and add a manual confirmation step before traffic routing.
- Readiness verification. We run scenarios on the preview environment: key path response, error metrics, logs, and behavior under load.
- Test switching and rollback. We perform a training traffic switch and return to the previous version, measuring the time of each operation. If rollback is longer than expected, we fix the scheme.
- Runbook handover. We provide documentation: how to release, how to roll back, what to watch on duty, and which signals to consider a reason for returning the version. After that, the customer's team runs releases themselves.
Configurations, environment schemes, and rollback runbook
At the output, the customer receives not a set of disparate files but a connected package: configurations, schemes, runbooks, and instructions by which the on-call shift acts without the author of the changes. Below is exactly what is included in the handover and why each part is needed in operation.
- Deployment configuration files: descriptions of two working environments, traffic switching rules, resource limits, and readiness probes — in a repository with edit history, not on an engineer's machine.
- Environment schemes: which environments exist, how they differ in composition and data, how traffic flow goes during switching, and where the entry point for return remains.
- Access policies: read and modify roles, who may switch traffic, who only observes, and the order of access and secret rotation.
- Rollback runbook: trigger conditions, the person responsible for the decision, target return time to the previous version, and the behavior of active connections at the moment of switching.
- On-call instructions: a one-page scenario — from monitoring signal to result confirmation, indicating where to view state and where to write in case of doubts.
- Observability map: a set of metrics and thresholds that show degradation of the new version, and a list of notification recipients.
- Data schema workflow: compatibility rules for changes so that a code rollback does not break already applied migrations.
Such a package removes dependence on a specific specialist: switching and rollback are reproducible by document, not by memory.
Implementation case study: how the downtime window was reduced
What it was before
A service of a dozen microservices in Kubernetes was released once every two weeks, and only in a night window.
One release took 3-4 hours: a third of the time went to manual traffic switching, environment warm-up, and checks, and if there was an error in the new build, the service went down for 20-40 minutes — until the on-call engineer rolled back the image and brought up the previous environment. Such episodes accumulated up to seven per quarter.
What was changed and what became
A second environment with identical configuration was raised, and traffic switching was moved to the routing level using a blue-green scheme, while rollout management was given to Argo Rollouts.
The new version first took a share of traffic and only then the main flow; promotion and rollback criteria were set by metrics: error rate, response time, and readiness check state.
If indicators exceeded the threshold, the rollout stopped itself, and traffic returned to the previous environment without on-call involvement.
Release time was reduced from 3-4 hours to 20-25 minutes, and the night window stopped being mandatory.
The number of incidents on releases fell from seven per quarter to one, and the share of zero-downtime rollbacks reached 100%: returning to the previous version takes less than a minute and does not break user connections. The freed time was moved by the team to checks in the test environment.
What to do if the release pipeline is already deployed?
There is no need to rebuild the existing pipeline entirely. Blue-green is embedded as a separate release stage: build, tests, and checks remain in place; only the method of delivering the version to the cluster changes. The previous process continues to work until the new one passes validation on one service.
The phased transition starts with the cluster and environments.
Next to the working environment, a second set of pods with identical configuration is raised, differing only in labels; traffic switching is taken on by Service, and weighted distribution by Ingress or a service mesh.
Quotas and limits are calculated immediately for two environments, otherwise the green environment will displace the blue one under load.
The key limitation is data schema compatibility. We prepare migrations by the expansion principle: first add fields and tables, teach the code to read both formats, and only in the next release remove the old one. Then both versions work with one database, and switching does not require downtime.
Release migration goes in waves: first a stateless service, then queues and background tasks, and finally the core. Rollback is returning traffic to the previous environment in seconds, without rebuilding the image and re-running checks.
Development does not stop: both environments live in one cluster, and the pipeline builds to the same branch.
Before the first wave, we check readiness probes, long connection behavior, and migration application order — otherwise switching will break open sessions. After that, the scheme is replicated to other services without changes to the team's usual processes.
Common technical questions before starting
Blue-green in Kubernetes looks simple: two environments, traffic switching at the service level, and rollback by returning the pointer.
In practice, the solution rests on four questions — what exactly to duplicate, what to do with a shared database, how to separate secrets, and how to prove the green version is healthy before switching. Below are short answers to what comes up most often at the start.
Do I need to duplicate all infrastructure?
No — only the application layer: the workload, its settings, and routing rules. The database, queues, cache, and storage remain shared; the second environment accesses them through separate accounts and prefixes.
What happens to the database during switching?
The schema changes through expanding migrations compatible with both code versions: new fields are added in advance, and old ones are removed only after the previous version is decommissioned. Then rollback affects the application and does not require rolling back data.
How are secrets and keys separated?
Each environment has its own set, connected from external storage, not hardcoded in manifests. Rotation happens independently, so switching does not drag along key reissue and synchronous edits in both environments.
How do I make sure the green version is healthy, and will the load double?
Before switching, it passes readiness checks, synthetic scenarios, and shadow flow on part of the traffic; metrics and tracing are compared with the current ones.
Resources for verification are allocated above the usual level — the peak is planned in advance, and extra nodes are removed immediately after switching.
Let's discuss the task and come back with an estimate
So that the timeline estimate and work plan are based on facts, not guesses, we need a short snapshot of your environment: how the release is structured, what limits it, and what is critical not to break during an update.
We can discuss the task in free form — details will be clarified on a call with the engineer who further leads the project, and we will return with an estimate and stages.
The conversation goes faster when it is already clear:
- how updates are released now: manual steps, who confirms, and what happens between the build and the production environment;
- what must not be lost during switching: active sessions, background tasks, and cache and queue state;
- what the environment consists of: clusters, image registry, external dependencies, and accesses;
- rollback requirements: within what time you need to return to the previous version and who gives the command;
- what automation already exists: application description, manifests, build and check rules;
- constraints: release windows, maintenance mode, approvals with security and adjacent teams.
If we are talking about blue-green deployment, we will separately discuss how two versions coexist and by what criterion the decision on switching and rollback is made.
Based on these inputs, we assemble the scope of work, timeline estimate, and order of stages — and return with a work plan that can be started from, without ambiguities at the project start.