Immutable Infrastructure: Replacing Servers Instead of Updating

A server lives for three years. Over time, packages are updated manually, configs are changed, temporary fixes are applied "for five minutes." No single engineer knows the exact current state of that machine. ansible-playbook fails on non-standard values someone tweaked directly. Such a server is called a snowflake server — unique and fragile. Typical situation: after another puppet apply, half of production goes down because one instance had its nginx.conf manually rewritten. There's nothing to roll back to — each server is individual. You have to debug each one separately, losing hours of downtime. Our team of DevOps engineers with 10+ years of experience deals with these problems daily and helps companies transition to a reliable immutable infrastructure model.

Immutable infrastructure is a principled rejection of updating servers. Instead of apt-get upgrade or ansible — create a new image, launch new instances, delete the old ones. A server is never changed after deployment. This approach guarantees environment identity and simplifies rollbacks.

Why is immutable infrastructure more beneficial?

Compare the old approach with the new in a table:

Phoenix Server Pattern

The concept is simple: every server is temporary. Like the Phoenix bird, it burns and a new one appears. Application configuration and code are baked into the image. Data is stored outside the server — in S3, RDS, EFS.

For VM infrastructure, this is implemented via Packer + Terraform:

# packer/api-server.pkr.hcl
source "amazon-ebs" "api" {
  ami_name = "api-server-${var.version}-${formatdate("YYYYMMDD", timestamp())}"
  instance_type = "t3.medium"
  region = "eu-west-1"
  source_ami_filter {
    filters = {
      name = "ubuntu/images/hvm-ssd/ubuntu-22.04-amd64-server-*"
      root-device-type = "ebs"
      virtualization-type = "hvm"
    }
    most_recent = true
    owners = ["099720109477"]
  }
}

build {
  sources = ["source.amazon-ebs.api"]
  provisioner "ansible" {
    playbook_file = "ansible/api-server.yml"
    extra_arguments = ["--extra-vars", "app_version=${var.version}"]
  }
  post-processor "manifest" {
    output = "packer-manifest.json"
  }
}

Packer builds the AMI. Ansible provisioner bakes the application directly into the image. After the build — packer-manifest.json contains the new AMI ID.

Terraform: Replacing Instances via Launch Template

resource "aws_launch_template" "api" {
  name_prefix = "api-server-"
  image_id = var.ami_id # ID from packer-manifest.json
  instance_type = "t3.medium"

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name = "api-server"
      Version = var.app_version
    }
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_group" "api" {
  name = "api-${var.app_version}"
  desired_capacity = 3
  min_size = 2
  max_size = 6
  vpc_zone_identifier = var.private_subnet_ids

  launch_template {
    id = aws_launch_template.api.id
    version = "$Latest"
  }

  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 50
      instance_warmup = 120
    }
  }

  lifecycle {
    create_before_destroy = true
  }
}

instance_refresh — a built-in AWS mechanism for replacing instances without downtime. When the image_id in the Launch Template changes, the ASG gradually replaces old instances with new ones, maintaining a minimum of 50% healthy.

CI/CD Pipeline: From Commit to New Image

# .github/workflows/deploy.yml
jobs:
  build-ami:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build AMI with Packer
        run: |
          packer init packer/
          packer build \
            -var "version=${{ github.sha }}" \
            packer/api-server.pkr.hcl
      - name: Extract AMI ID
        id: ami
        run: |
          AMI_ID=$(jq -r '.builds[-1].artifact_id' packer-manifest.json | cut -d: -f2)
          echo "ami_id=$AMI_ID" >> $GITHUB_OUTPUT
  deploy:
    needs: build-ami
    steps:
      - name: Update Terraform
        run: |
          terraform apply \
            -var "ami_id=${{ needs.build-ami.outputs.ami_id }}" \
            -var "app_version=${{ github.sha }}" \
            -auto-approve

Every commit to main → new AMI → update ASG. No ssh into production servers. No ansible-playbook against live instances.

How to Implement Immutable Infrastructure: Step-by-Step Plan

• Audit current infrastructure: identify all components storing state on disk and plan their migration.
• Extract data: move databases to RDS, files to S3, sessions to Redis. Set up log externalization.
• Create Packer templates: for each server type — a separate template with Ansible provisioner, baking the application and dependencies.
• Configure Terraform modules: Launch Template with image_id from Packer, Auto Scaling Group with instance refresh.
• Build CI/CD pipeline: GitHub Actions or GitLab CI: commit → build AMI → update Terraform → deploy to prod.
• Test and rollback: verify instance replacement, perform a rollback to the previous AMI.
• Train the team: hand over documentation and runbook.

Critical: What Must Not Be on the Server

Immutable infrastructure only works if several rules are followed:

All data outside the server. Database — in RDS. Uploaded files — in S3. Sessions — in Redis. Logs — to CloudWatch Logs or Loki. If data is stored on instance disk, it is lost upon replacement.

Configuration via environment variables. No config files created during deployment. Only environment variables from Secrets Manager or Parameter Store:

# user-data script for instance
export DB_URL=$(aws ssm get-parameter --name /api/db-url --with-decryption --query Value --output text)
export SECRET_KEY=$(aws secretsmanager get-secret-value --secret-id api/secret-key --query SecretString --output text)
systemctl start api

No SSH into production. If an engineer can log in and change something manually — immutability is broken. Use SSM Session Manager instead of direct SSH, and only for emergency diagnostics with full audit log.

Rollback Is Deploying a Previous AMI

# Roll back to the previous AMI
terraform apply \
  -var "ami_id=ami-0abcdef1234567890" \
  -var "app_version=previous-sha" \
  -auto-approve

5-7 minutes to replace instances. Predictable. No surprises like "rollback went through but the config is different." Immutable infrastructure is 10x faster in recovery speed compared to the traditional approach — empirical data from our projects.

What to Do with Legacy Components That Store State on Disk?

The hardest part during migration is legacy components that store state on disk: applications with file-based sessions, cron jobs writing to /tmp, apps with hardcoded config paths. These need to be refactored before switching to immutable. We help conduct an audit, identify such places, and plan migration. This takes more time than setting up Packer + Terraform itself, but without it immutable won't work.

What's Included in a Turnkey Solution

We handle the entire cycle: from audit to documentation handover. As a result, you get:

• Packer templates for building images of all components
• Terraform modules with Auto Scaling and instance refresh
• CI/CD pipeline (GitHub Actions / GitLab CI) with automatic deployment and rollback
• Runbook for operations and recovery
• Training for your team on the new infrastructure
• Guarantee of stable operation for 30 days after implementation