Services for Deployment Strategy Implementation and Rollback

Why are deployment strategies for zero-downtime impossible without Observability?

Standard deployment strategies like Rolling Update, Blue-Green, and Canary often fail without proper observability. The Rolling Update in Kubernetes works for stateless services, but incompatible API changes or database migrations break this scheme. Blue-Green solves rollback in seconds, but requires double resources — for a service with 50 replicas, that's an additional 40% infrastructure cost. Canary reduces risk to 5% of traffic, but without metrics (error rate, latency) it's blind guessing. Our experience shows: the right strategy reduces MTTR from 4 hours to 15 minutes and cuts infrastructure costs by 20–30%.

We have implemented deployment strategies for 50+ projects over 5 years on the market. In this article, we'll analyze when each approach works and how to avoid typical mistakes.

What metrics are critical for safe canary?

Rolling Update: a default that isn't always suitable

Rolling Update is the default strategy in Kubernetes. Pods are updated in batches: a new pod with the new version starts, the old one is killed, repeated until all pods are updated.

This works well for stateless services with backward-compatible changes. The problem arises when two versions are incompatible — for example, the new API version doesn't understand requests from the old client, or a database schema change without backward compatibility. In 15% of projects, we encountered downtime specifically due to schema incompatibility.

In Kubernetes, Rolling Update parameters are set in spec.strategy:

strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 # how many new pods above desired to run simultaneously maxUnavailable: 0 # how many pods may be unavailable during the process 

maxUnavailable: 0 means zero-downtime: before killing the old pod, the new one must become Ready. Slower but safer. Compare: Blue-Green requires twice the resources, while Rolling Update with maxSurge: 1 uses only one more replica — 2–3 times more economical.

Blue-Green: instant rollback at the cost of double resources

We keep two identical environments — blue (current production) and green (new version). Deploy to green, run smoke tests, switch traffic. If something goes wrong — switch back to blue. Rollback takes 5 seconds.

According to Wikipedia, Blue-Green deployment is a technique that reduces downtime and risk by running two identical production environments. In Kubernetes, this is implemented via two Deployments and one Service that changes its selector when switching:

kubectl patch service my-app -p '{"spec":{"selector":{"version":"green"}}}' 

Or via Ingress with annotations for NGINX:

nginx.ingress.kubernetes.io/canary: "true" nginx.ingress.kubernetes.io/canary-weight: "100" 

The main downside of Blue-Green is double resource consumption during deployment. For small services, this is negligible. For a cluster with 50+ replicas per service, it's significant: avoiding Blue-Green can yield substantial savings on infrastructure.

Database migrations and Blue-Green

If the schema migration is incompatible with the old application version, switching back to blue is impossible. This is avoided via the expand-contract pattern: first deploy an expanding migration (new fields optional, old version continues working), then switch to green, then remove old fields with a contracting migration. This adds 1–2 days to the implementation timeline but eliminates the risk of data consistency loss.

Setting up Canary rollout with automatic metric analysis

Canary is the most cautious strategy for changes with unpredictable impact on users. 5% of traffic goes to the new version, we monitor error rate and latency, if everything is good — expand to 20%, 50%, 100%.

Manually in Kubernetes, this can be emulated via replica ratio: if you need 10% traffic to canary — you need 1 canary replica to 9 stable replicas. Not very convenient to manage. Argo Rollouts handles canary 10 times faster than manual management — it automates steps and checks.

A proper tool for canary is Argo Rollouts. It adds the Rollout CRD as a replacement for Deployment and can manage promotion steps automatically:

apiVersion: argoproj.io/v1alpha1 kind: Rollout spec: strategy: canary: steps: - setWeight: 5 - pause: {duration: 5m} - setWeight: 20 - pause: {duration: 10m} - analysis: templates: - templateName: success-rate - setWeight: 100 canaryService: my-app-canary stableService: my-app-stable 

After each setWeight — a pause for observation. analysis — automatic metric check via AnalysisTemplate. If the success rate drops below the threshold — Rollout automatically rolls back to the stable version without manual intervention.

AnalysisTemplate for Prometheus:

apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: metrics: - name: success-rate interval: 1m successCondition: result[0] >= 0.95 failureLimit: 3 provider: prometheus: address: http://prometheus.monitoring.svc:9090 query: | sum(rate(http_requests_total{app="my-app",status!~"5.."}[2m])) / sum(rate(http_requests_total{app="my-app"}[2m])) 

Three consecutive failed checks — automatic rollback. This only works if you have proper service-level metrics. If Prometheus is not scraping your application — canary without analysis works blind. We guarantee zero-downtime only with configured observability.

Feature Flags: deploy separately from release

Feature Flags decouple code deployment from enabling features for users. Code goes to production, but works only for specific users, traffic percentage, or environment.

This allows more frequent deployments (less risk per deployment) and A/B testing without separate deployments. Tools: LaunchDarkly, Unleash (self-hosted), Flagsmith, or a simple solution based on configuration in Redis/etcd.

The main issue with Feature Flags is technical debt. Flags must be removed after full rollout. A codebase with 50+ old flags becomes unreadable, and conditional logic breaks unexpectedly. Our experience: in one project, we removed 120 flags in two weeks — this reduced latency by 15% and lowered support costs significantly. Flags should live in a task tracker with a removal date, like regular technical debt.

How to choose a deployment strategy for your service?

On practice, a single system often uses multiple strategies simultaneously: infrastructure services go through Rolling Update, product changes through Canary with Argo Rollouts, major features are covered by Feature Flags.

Comparison of tools for canary deployment

For 90% of projects, Argo Rollouts is sufficient — it integrates with Prometheus, doesn't require a service mesh, and is easy to maintain.

What is included in the work (deliverables)?

1. Audit of the current deployment process and service architecture — identify bottlenecks and risks.
2. Strategy selection (or combination) for your stack and budget — consider SLO, SLA, infrastructure cost.
3. Infrastructure setup: Helm charts, manifests, Argo Rollouts, Feature Flags server — all configs under version control.
4. CI/CD integration (GitLab CI, GitHub Actions) — ready-made pipelines with canary and rollback stages.
5. Observability setup: Prometheus/Grafana dashboards, SLO alerts — so every deployment is visible.
6. Runbook documentation for deployment and rollback — so any on-call can execute in 10 minutes.
7. Team training (2–3 sessions) and access handover — your developers can run canary independently.
8. Post-release support for 2 weeks — we guarantee the strategy works and provides zero-downtime.

Deliverables you receive: detailed runbook, configured CI/CD pipelines, observability dashboards, trained team, and 14 days of post-release support.

Time estimates

• Setup of Blue-Green in Kubernetes: 1–2 days
• Implementation of Argo Rollouts with Canary and AnalysisTemplate: 3–5 days
• Setup of Feature Flags with Unleash: 2–4 days
• Design and implementation of a comprehensive strategy for multiple services: 1–2 weeks

Cost is calculated individually based on your infrastructure and service count. We have 5 years of market experience and 50+ completed projects — you get a proven approach without trial and error.

Order deployment strategy implementation — get zero-downtime in just a week. Contact us for a consultation on choosing a strategy for your service. We will evaluate your project and provide a fixed-price quote with clear timelines.