Immutable Deployment: Guaranteed Release Reproducibility

"Works on staging, doesn't work on prod" — a classic that almost always boils down to one thing: production servers have accumulated state. Someone manually installed a package without committing. Someone edited a config directly. Artifacts from old deployments pile up. Immutable Deployment eliminates this category of problems entirely: the server is not modified — it is replaced. The concept is described in the article on Immutable infrastructure. We implement this strategy turnkey, from audit to GitOps setup, and guarantee predictability for every release. Our experience: over 6 years in DevOps, over 40 successful GitOps implementation projects. Annual savings from eliminating configuration drift can be significant, and mean time to resolution drops to 5–10 minutes — 6–8 times faster compared to the mutable approach.

Why Immutable Deployment is Better Than Mutable Servers?

Mutable servers are like a snowball — every manual action leaves a mark. Over time, differences between dev and prod become catastrophic: different library versions, configs, permissions. The Immutable approach eliminates this: the same build instructions produce the same image on any machine at any time. The image is built once, tagged with the commit hash or semantic version, and that same image passes through all environments — from dev to prod. No apt-get upgrade in production, no sed -i on live configs. This reduces rollback time from 20–30 minutes to 2–3 minutes, and the number of incidents related to configuration drift decreases by 95%. Even deployment frequency can increase from once a week to several times a day — without loss of stability.

In Docker, this looks strict: the base image must have an exact version, not latest:

FROM python:3.11.9-slim-bookworm
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
USER nonroot
ENTRYPOINT ["gunicorn", "app:wsgi"]

The image tag in a Kubernetes Deployment is always specific, never :latest:

spec: containers: - name: api image: registry.example.com/api:a3f8c21d imagePullPolicy: IfNotPresent 

imagePullPolicy: IfNotPresent — if the image with that tag already exists on the node, don't pull again. With immutable tags, this is safe.

How to Version Configuration?

Image immutability means that environment configuration — secrets, URLs, feature flags — cannot be part of the image. It is passed via environment variables or mounted ConfigMap/Secret. Importantly: ConfigMap and Secret should also be versioned. api-config-v3 is a new version of the config, api-config-v2 remains and is used by old pods during a rolling update. After the full transition, the old version is deleted.

The External Secrets Operator + Vault handles this more cleanly: secrets are stored in Vault, ESO syncs them to Kubernetes Secrets, recreating them when Vault changes.

Deploy via Replacement, Not Update

The standard kubectl set image does a rolling update: old pods go away, new ones come. This is immutable at the pod level, but the Deployment object remains mutable. A cleaner approach is GitOps: any change only via PR in Git, ArgoCD syncs the cluster with the repository.

---
# argocd/application.yaml
spec:
  source:
    repoURL: https://github.com/company/k8s-manifests
    targetRevision: HEAD
    path: apps/api
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

selfHeal: true — if someone manually runs kubectl edit deployment, ArgoCD reverts it back to what is in Git. That is immutability at the cluster state level.

Rollback Is a Deployment of the Previous Image

With an immutable strategy, rollback is simply deploying the old tag. No magic kubectl rollout undo that restores the previous ReplicaSet. Just change the tag in the manifest (via Git revert) and ArgoCD syncs.

git revert HEAD # revert the last commit with the new tag
git push origin main # ArgoCD sees the change and deploys the old image

This takes 2-3 minutes. Predictable and reproducible. The average cost of a configuration drift incident can be substantial, so fast rollback pays off immediately.

Implementation Process: Step-by-Step Guide

1. Infrastructure audit: check Dockerfiles for exact versions, CI/CD for mutable tags, Kubernetes clusters for state.
2. Pipeline setup: build → tag (SHA + environment) → push → deploy. All tags immutable.
3. GitOps setup: deploy ArgoCD or FluxCD with self-heal and prune policies.
4. ConfigMap and Secret versioning: each config gets a version; when changed, a new one is created, the old one is deleted after transition.
5. Rollback testing: verify that rollback works via Git revert.
6. Documentation and training: write runbook and conduct a workshop for the team.

stages:
  - build
  - push
  - deploy
variables:
  IMAGE_TAG: ${CI_COMMIT_SHORT_SHA}
build:
  stage: build
  script:
    - docker build -t registry.example.com/app:${IMAGE_TAG} .
    - docker tag registry.example.com/app:${IMAGE_TAG} registry.example.com/app:latest-production
push:
  stage: push
  script:
    - docker push registry.example.com/app:${IMAGE_TAG}
deploy:
  stage: deploy
  script:
    - sed -i "s|image:.*|image: registry.example.com/app:${IMAGE_TAG}|g" k8s/deployment.yaml
    - git commit -m "Update image tag to ${IMAGE_TAG}"
    - git push

What's Included in the Work

• Infrastructure audit: Dockerfiles, CI/CD, Kubernetes clusters
• Migration to immutable tags with commit hashes
• GitOps setup (ArgoCD/Flux) with self-heal policy
• ConfigMap and Secret versioning with rolling update support
• Runbook documentation and team training
• Post-release support for 2 weeks

Cost is calculated individually after an audit. Get a consultation — we'll assess your project in 1 day. Request an infrastructure audit today to avoid configuration drift problems. Our DevOps engineers' services for setting up seamless deployment will help you establish a CI/CD pipeline.