Zero-downtime deployment: what breaks during rollout in Kubernetes
Zero-downtime deployment breaks not at the image build stage, but at the moment a pod is removed from load balancing.
A typical scenario: RollingUpdate with maxUnavailable greater than zero sends pods for deletion immediately after the signal, the application doesn't have time to finish active requests, and the load balancer still keeps them in its endpoints — connections break mid-response, and some clients receive a 502 instead of data.
The second point of failure is container startup. The application responds to the readiness probe before it has warmed the cache, brought up the database connection pool, and finished reading the configuration.
Traffic is already going to an unready pod, requests fail by timeout, pods go into restart — and the service becomes unavailable exactly in the minute when the release has already increased the load on the remaining replicas.
The third failure scenario is rollback under load. The old and new versions work with the same schema for some time: the migration was made ahead, the application of the previous release doesn't know about the new columns and throws errors when updating the version. A rollback that should take seconds turns into incident investigation and data recovery.
Below we break down how this is closed: graceful termination with connection draining, honest probes with warm-up, migration order, and rollout rules under which absence of downtime is a property of the configuration, not the on-call engineer's luck.
Continuous release without downtime: what changes in operations
Downtime during a release is created not by the image change, but by the moment when the new version starts receiving traffic and the old one has already stopped.
The readiness probe keeps an instance out of load balancing until warm-up, the pre-stop hook lets open connections close, and an update strategy with zero unavailable copies preserves a margin of replicas.
The readiness probe, pre-stop hook, and a margin of one copy are standard Kubernetes mechanisms that remove the window: the service responds throughout the rollout.
A setting such as maxUnavailable: 0 with maxSurge: 1 stops being a manual agreement and becomes part of the deployment description.
Readiness for load is covered by the disruption budget: it prevents an update or node eviction from taking down too many copies at once, while a warming container fills the cache before the instance enters rotation.
Migrations are a separate source of failures, and here the "expand — switch — contract" pattern works. First, a field is added to the schema that both versions of the code understand, then traffic is switched to the new logic, and only then is the old one removed.
The release stops being an atomic jump — an incompatible schema no longer belongs to the failure classes.
Release automation brings this to the pipeline: a version is described declaratively, rollback is returning the previous description rather than a manual rebuild.
In practice, the unavailability window goes to zero, a version release takes one pass instead of a series of approvals, and the share of rollbacks drops severalfold — along with the classes "traffic loss at startup" and "migration applied before the code is ready for it."
Zero-downtime deployment: comparison of rollout strategies
The choice of strategy is determined by what is more expensive in your particular system: rollback speed, resource consumption for a second environment, or traffic routing complexity.
All modes rely on the same set of primitives — replicas, readiness checks, service load balancer — and differ only in the order in which versions replace each other.
Constraints here are more important than capabilities. A rolling update requires that old and new instances run simultaneously and understand shared data, while blue-green and canary environments require managed routing. Below are the criteria by which we select a mode for a specific service.
| Rollout strategy | When it fits | Limitations |
|---|---|---|
| Rolling update | Stateless services, regular releases | Old and new versions live in parallel: the data schema, message and response formats must be readable by both |
| Blue-green environment | Critical interfaces where instant rollback by switching traffic is needed | Double resource consumption during the rollout, support for two sets of configuration and access permissions |
| Canary release | Changes with unclear impact, measurement on a share of the audience is needed | Metrics by version and routing by traffic share are required; observation lengthens the release |
| Full recreation | Internal and batch services, background jobs | There is an unavailability window; not suitable for external interfaces |
Schema migration deserves separate mention: it is carried out in three steps — expanding the structure, migrating data, removing the old field — and only after both versions of the application have stopped accessing the previous format.
Implementation stages: from cluster audit to handover of schemes
The stages go sequentially, and each is closed with a verifiable result — without it we do not start the next step.
We do not rearrange the order: setting up the rollout before auditing environments gives false confidence, and a regulation without testing on a training environment will not survive the first release.
- Cluster audit. We analyze how services are deployed: number of replicas, readiness probes, correct pod termination, and the fate of open connections during shutdown. The bottleneck is usually one — a shared external load balancer without traffic-share management.
- Implementation plan. Environment scheme, separation of access permissions and secrets, queue of changes by service — so that releases do not interfere with each other and rollback is possible at any step.
- Rollout setup. Step-by-step update with a limit on the number of simultaneously unavailable replicas, readiness probes with working thresholds, a pause for closing connections before container shutdown.
- Data schema migrations. We break them into compatible steps: expansion, migration, removal of the old field. Old and new versions of the service run simultaneously, and rollback does not lose data.
- Testing on a training environment. Running a release under load and with simulated failures: node shutdown, connection interruption, an error in the new version. We measure traffic loss and recovery time.
- Handover of regulations. Environment schemes, configurations, release and rollback procedure, list of metrics, and conditions under which the rollout stops automatically.
After handover, your on-call engineers release versions along a documented route, and we remain in touch during the period of the first independent rollouts.
What remains with the customer: configurations, schemes, regulations
We do not leave the customer with a set of verbal agreements: everything that determines the system's behavior during zero-downtime rollout lives in the repository along with the change history.
An engineer who joins the team later will read not "that's how it historically happened," but the reason for each parameter set.
- Rollout configurations — declarative descriptions with readiness parameters, pause between waves, and conditions for automatic rollback.
- Environment schemes — a map of environments, their differences in data and load, and the order in which a build progresses between them.
- Access policies — permissions are granted to roles, not people: the boundaries of "who starts the rollout, who reads the logs" survive changes in team composition.
- Documentation — the path of a request through the updated version, mandatory checks, and the procedure when expected and actual states diverge.
- On-call regulations — which signals switch the on-call engineer into rollback mode, who receives escalation, and what is reviewed after an incident.
- Migration scenarios — the order of data schema changes compatible with the old and new application version during traffic switching.
- Pre-release checks — a set of automated runs without which the build does not proceed beyond the test environment.
Each item is a working file, not a verbal description: it can be compared with the current state of the system and the divergence can be seen immediately.
Case study: releasing a payment service without stopping traffic
The payment service was released in a night window: payment acceptance was stopped for 30–40 minutes, updated, and brought back up. When something went wrong, it was rolled back manually — another forty minutes to an hour, and some operations had to be recalculated separately. Each such release occupied the night shift of on-call engineers.
What was changed
In Kubernetes, we switched to the RollingUpdate strategy with parameters maxUnavailable: 0 and maxSurge: 1: a new replica enters load balancing only after a successful readiness probe, while the old one receives a shutdown signal and finishes already accepted requests.
Schema migrations were moved out of container startup into a separate step with locking — parallel replicas no longer apply the same schema and do not fail the release.
Fields in the database were added using the expand-contract pattern: new code reads both old and new data, while incompatible removals go in the next release.
Metrics before and after
Version release time decreased from 40–60 minutes of a night window to 6–8 minutes during working hours. The share of rollbacks dropped from roughly every fifth rollout to one in twenty, and almost all of them are automatic — based on error metrics, without manual investigation.
User visits to the error page during a release — zero: connections wait for a response from a live replica instead of being interrupted.
A few months later, manual control during rollout was removed entirely: the on-call engineer gets involved only if the automation itself returned the previous version and flagged the release.
How does rolling out a new version differ from restarting a service?
Restarting an application and rolling out a new version are different operations with different consequences. A restart brings processes up again on the same image; the code version does not change.
A version change is replacing the image in the workload description, and here the orchestrator must route traffic past the old instances without dropping a single connection.
The difference is visible in how readiness for traffic is determined. During a restart, the service returns to load balancing as soon as the process has bound the port.
During a container update, a probe is needed that checks not "is the process alive" but "is it ready to serve a request": database connection, warmed cache, availability of dependencies.
Without it, the new version receives requests during initialization — hence errors on the very first request.
The second half of the task is graceful termination. The old instance is not deleted immediately: first it is removed from load balancing, then a shutdown signal is sent, and only after a specified window are connections closed.
If you violate the order, some clients will be cut off exactly at the moment of the version change, even though the update formally succeeded.
In Kubernetes, these steps are described declaratively: how many copies to bring up in parallel, what share may be taken out of service, how long to wait for termination. These are manifest parameters, not manual actions by an engineer at release time.
This difference is exactly what separates a restart from zero-downtime deployment: the rollout takes longer, but the user sees no downtime. For services with a database, migration order is added — schema expansion before the rollout, removal of outdated fields after.
Technical concerns: schema migrations and backward compatibility
Changing the data structure is what truly breaks zero-downtime deployment. While the rollout is in progress, pods of the old and new generation live in the cluster simultaneously, and both work with the same database.
The schema must remain compatible with both versions of the code: otherwise the new one fails on old data, and the old one fails on new data.
That is why we divide migrations into stages. First expansion: we add a column or table, the code writes to both the old and new location simultaneously. Then background data migration in batches, switching reads — and only in a separate release, removal of the old field. Application rollback with this scheme does not require database rollback.
What happens if the release has to be rolled back?
While the migration is at the expansion stage, the previous version works unchanged — rollback is returning to the previous image. We move removal of old columns into a separate release after a period of observation, so that rollback remains possible.
How can we make sure the old version doesn't break?
We add a compatibility check to the pipeline: the previous release is run against the new schema on a test copy. Heavy changes — index rebuilds, type changes — are performed with non-blocking operations with a time limit; otherwise a lock on a large table will stop writes.
How is the schema versioned?
By a table of applied migrations and a check in the pipeline: a build is not rolled out if the schema version does not match the expected one. The migrations themselves are idempotent and are applied before pods start, so a repeated run does not break anything.
Let's discuss the task: we will come back with an estimate and implementation plan
So that the project estimate is based on your environment rather than general assumptions, we start the task discussion with a short technical review.
It is important for us to understand how the release is built, how traffic reaches the cluster, which processes live longer than an individual pod, and what happens to the database during rollout.
Usually one conversation and access to manifests, the release regulations, and the service diagram are enough.
What is worth preparing for this conversation:
- How the release is rolled out now: sequential replica update, parallel environment, or a share of traffic to the new version.
- Whether readiness checks are configured and how the service finishes requests when a pod is stopped.
- How schema migrations are applied: separately from the code, with backward compatibility, or in one step.
- Whether there are background jobs and queues that cannot run in two versions simultaneously.
- What checks are run before rollout and by what metrics you decide that the release succeeded.
- The acceptable maintenance window and acceptable share of lost requests during switching.
In response, we come back with a project estimate: the scope of work by stage, rollout order, rollback points, and a list of configuration changes. We break the implementation plan into steps so that the first one already removes downtime, and the rest consolidate the result. Describe the task — we will come back with an estimate and a plan.