Terraform Drift Detection: Automation, CI/CD, Alerts

Terraform Drift Detection: Automation, CI/CD, Alerts

Imagine: your production runs stable, but after a terraform apply a service suddenly crashes. It turns out a colleague manually changed a security group in the AWS console "for 5 minutes", and the next apply overwrote that edit. That's drift — the divergence between code and reality. It accumulates silently and manifests at the worst moment. According to industry data, more than half of cloud infrastructure incidents are linked to drift. Source On one project with 500 resource records, drift accumulated for 3 weeks until a critical Kubernetes node failed. After our automatic plan configuration, discrepancies were detected within 10 minutes before apply. We are a team of DevOps engineers with cloud infrastructure experience. We will set up automatic drift detection turnkey so you always know about differences before they cause an incident.

What Is Drift and Why It's Dangerous

Drift is the difference between what is described in your Terraform configuration (and stored in state) and what actually exists in the cloud. Causes: manual edits in the console, automatic provider field updates, applies from outdated branches. Drift can lead to unexpected downtime, data loss, or security breaches. For example, an accidental change to firewall rules can expose a database.

terraform plan is the main tool for drift detection. It compares the state with real resources and shows changes. If a plan shows changes where nobody touched the code, that's a drift signal.

Why Automating Drift Detection Is Critical

Manually running plan before each deploy is not enough. Drift can occur between releases. Automation provides:

• Early detection of discrepancies (before applying changes)
• MTTR reduction from hours to minutes — 10x faster than manual search. Our client reduced MTTR from 2 hours to 10 minutes, saving an estimated $2,500 per incident.
• Audit of cloud changes that bypassed Git

We guarantee infrastructure stability: after setup, you receive regular drift notifications in your preferred channel. Our automation detects drift within 5 minutes of occurrence. We have processed over 200 drift incidents in the past year.

How We Configure Drift Detection

We set it up on any CI/CD: GitLab CI, GitHub Actions, Atlantis, Terraform Cloud, Spacelift. Example for GitLab CI:

# .gitlab-ci.yml drift-detection:
  stage: check
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
  script:
    - terraform init -reconfigure
    - terraform plan -detailed-exitcode -out=drift.plan 2>&1 | tee plan.log
  after_script:
    - |
      if [ $? -eq 2 ]; then
        curl -X POST $SLACK_WEBHOOK -d '{"text":"Terraform drift detected in production!"}'
      fi

The key is -detailed-exitcode: exit code 0 — no changes, 1 — error, 2 — drift. We react only to code 2.

For Atlantis, we use a dedicated drift PR that is recreated on each run. Spacelift and Terraform Cloud have built-in drift detection — we enable it within one day.

Comparison of Drift Detection Methods

Recommended Check Frequency

What to Do with Detected Drift

Three options:

• Revert via apply — if the cloud change is unwanted (someone manually edited), terraform apply restores the configuration to match the code. This is the main scenario.
• Update config — if the change is intentional, update the .tf file, create a PR, merge. After merge drift disappears.
• Add lifecycle { ignore_changes } — if a field is managed by an external service (e.g., desired_capacity in Auto Scaling Group).

resource "aws_autoscaling_group" "main" { lifecycle { ignore_changes = [desired_capacity] } } 

Our Process

1. Audit current Terraform infrastructure: analyze state files, pipeline, providers. Determine scope and bottlenecks. Gather all state files and corresponding cloud provider accounts.
2. Select the optimal drift detection method for your stack and budget. Consider change frequency, resource count, security requirements.
3. Configure a scheduled job (CI/CD or external service) with the needed frequency. Tailor to your infrastructure.
4. Integrate notifications (Slack, PagerDuty, Telegram) with change details. Add alert aggregation to reduce noise.
5. Document the process and incident runbook. Train your team on drift handling.
6. Support after launch: 1 month of consultation and adjustments.

What's Included

• Audit results: report on current infrastructure state, identified risks, and recommendations.
• Configured drift detection: chosen method with desired frequency and notifications.
• Documentation: process description, runbook, team instructions.
• Training: workshop for your engineers on drift and response.
• Support: one month of post-deployment accompaniment, including setting adjustments.

Metrics and Visualization

For large projects, we supplement alerting with metrics: export the number of resources with drift to Prometheus via a custom exporter. But for 90% of teams, Slack notifications with terraform plan output are sufficient. Optionally, configure Grafana dashboards to track drift across environments.

Timeline and How to Start

Standard setup takes 1 to 3 days depending on the number of state files and platform. The setup typically takes 2 days, reducing to 1 day for simple environments. Contact us — we'll assess your project and propose a solution. Get a consultation on drift automation today.