GitOps Terraform: why state drifts from code
State describes what is already created in the cloud, while code describes what should be.
As long as only discipline stands between them, drift appears quickly: access was needed urgently, an engineer edits a load balancer rule or disk size directly in the console, and the infrastructure description does not reflect it.
A week later, the next apply returns everything to its previous form — and breaks what was fixed manually.
The second source of drift is the state file itself. It is the only mutable artifact, and without proper locking, two simultaneous applies overwrite each other's records: some resources remain in state, some are lost.
Storage on a local machine amplifies the problem — state exists in a single copy with one person, and recovery after failure turns into archaeology.
The cost manifests not at the moment of the edit, but during incident investigation.
The change history is empty because the edit did not go through version control; the plan shows dozens of "new" resources that have actually been running for a long time; rollback to the previous release drags in someone else's changes.
Environments also diverge independently: what was manually fixed in staging will never reach the production environment, and vice versa.
We explain how this is closed: state moves to shared protected storage with locking, changes go only through a verifiable description, and apply is launched from a single point where the author and reason are visible.
This is exactly what GitOps is built on — the repository becomes the single source of truth about infrastructure.
What changes after implementation: measurable effects
Moving to state management through a repository changes not the fact of applying changes itself, but their cost in time and risk. Five consequences are visible already in the first month of work and only strengthen further.
Delivery speed. The plan and its apply follow one route: commit, review, automated check, apply.
Manual calls from an engineer's machine disappear, so a typical infrastructure change travels from edit to working environment in minutes instead of hours of waiting for a window and an available owner.
Recovery time after failure. Configuration and state sit side by side and are versioned, so rollback is a commit reversal, not rebuilding the environment from memory.
Recovery is reduced from hours to minutes and can be reproduced by any engineer on the team, not only by the one who made the edit.
Eliminated failure classes. State locking, copy divergence between sites, direct console edits, and changes "bypassing the process" cease to be possible — each class is closed by a rule, not an agreement. Errors remain in the configuration itself, and they are caught at review before apply.
Transparency of changes. Each action leaves a commit with author, reason, and history: you can see what changed, when, and why. Incident investigation stops depending on someone's memory or chat messages.
The share of manual operations falls to review and architecture decisions — routine applies go to automation. Infrastructure becomes predictable not only in description, but also in maintenance.
GitOps Terraform modes: comparison of approaches
The state management mode determines who applies changes and what is considered the source of truth for infrastructure. Below are four options encountered in practice — from local state to apply on merge request.
They are compatible step by step: the transition can be done in parts without rewriting environment descriptions in Terraform.
| Option | When suitable | Limitations |
|---|---|---|
| Local state | Trial, one engineer, one-off environment | No locking, copies diverge, recovery only from manual copies |
| Remote storage, manual apply | Team of up to eight people, infrequent releases | Changes are applied outside the repository, change verification is not provided, access is granted entirely to the environment |
| Remote storage, apply from pipeline on merge request | Multiple environments, need change trail and verification before apply | Branching discipline required; emergency edits go outside the normal order |
| Managed storage with environment-scoped access separation | Multiple teams, prod and test environments in one description | Roles are configured in advance, some capabilities are limited by the provider |
What we check during transition
State locking prevents two runs from changing one environment simultaneously: without it, parallel applies leave divergences between description and reality.
We build access separation so that only the pipeline can write to remote state storage, and engineers work through a merge request — then change is verified before apply, and rollback reduces to restoring the previous version.
Implementation stages: from audit to handover
We start with an infrastructure audit, not with installing tools: without resource inventory, any state separation scheme will be wrong, and it will have to be redone along with already migrated configurations.
Then we build the order in which each Terraform configuration change goes through Git, is verified before apply, and remains in history with author and reason for edit.
- Infrastructure audit. We collect a list of resources, environments, and providers, mark manually created items and divergences between description and actual state.
- State separation scheme. We determine granularity: separate sets by environments, domains, and criticality, so that failure of one does not block others. We configure remote storage and locks against parallel edits.
- Repositories and access. We lay out modules and configurations, define branching, mandatory review, and rights to protected branches.
- Change verification before apply. We connect automatic plan calculation in the merge request, static analysis, and policies: what is forbidden does not pass further. Apply to the working environment — only after explicit confirmation.
- Test run. We run creation, modification, and import of existing resources on a non-production environment, and separately check state recovery from backup.
- Handover to operation. We document the working order, walk through typical scenarios with the team, configure drift notifications and on-call.
- Rollback regulations. We record how to roll back a failed apply and what to do in case of state conflict while edits are in progress.
At the output, the team works with infrastructure in the same way as with application code, and access to manual console edits is closed or narrowed to emergency.
Implementation artifacts: configurations, schemes, regulations
Implementation ends not with a green plan in the last commit, but at the moment when the customer's team can change infrastructure without the scheme's author.
Therefore we hand over not repository access, but a set of artifacts by which an on-call engineer restores state from scratch: what is described by code, where state is stored, who has the right to change it, and how to roll back if apply failed.
- Terraform module catalog — purpose of each module, input and output variables, dependencies between them, as well as places that break when the provider major version changes.
- Environment separation scheme — which environments exist, which state storage is assigned to each, how state locking is configured, and who can apply changes.
- Access policies — role matrix: rights to repositories, state storage, cloud accounts, mandatory approvals, and protection of the branch from which apply is launched.
- Apply agent configurations — list of repositories under management, plan templates by directories, concurrency rules, and binding projects to environments.
- Change regulations — how many approvals are needed for the production environment, who owns the catalog, in what order an urgent change outside the normal process goes.
- Recovery instructions — rollback to the previous state revision, recovery from versioned storage, actions when real infrastructure diverges from the described one, manual import of resources created bypassing code.
- Verification map — which automated checks detect drift, where notifications arrive, who and within what time investigates the signal.
- Typical incident regulations — step-by-step scenarios for state loss, a partially applied plan, and cloud provider unavailability.
All documents lie next to the code in the same repository, so the description and real configuration do not diverge.
Case: removed manual edits and state drift
Before implementation, engineers applied changes from local copies, and state was edited by commands importing and deleting objects directly.
According to the lock journal, a third of changes passed bypassing the shared pipeline: divergence between infrastructure description and reality surfaced only at incident, and rollback was assembled manually and took about 45 minutes.
We deployed Atlantis in the cluster and bound it to repositories: the plan is published as a comment to the merge request, apply is launched after approval, and write to state storage is allowed only to the service account of the service.
Engineers retained read access, emergency apply — only according to regulations with the reason recorded.
| Metric | Before | After |
|---|---|---|
| Share of manual changes | 34% | 2% |
| Rollback time | ~45 minutes | ~6 minutes |
| State lock conflicts | 5 per month | 0 |
| Configuration divergences from reality | 4 per quarter | 0 over two quarters |
Metrics were taken from three independent sources: the state lock journal, object version history in storage, and incident duration from the tracking system.
We compared periods comparable in load — configuration divergence ceased to be a cause of outages, and rollback reduced to one action.
What to do if infrastructure is already deployed manually?
If infrastructure is deployed manually, this is not an obstacle — there is simply no description: resources live at the provider, but not in configuration.
We start with inventory: collect the actual parameters of networks, machines, load balancers, and access rights and compare them with the description in the repository. While comparison is incomplete, the plan must not be applied — it will try to recreate what already works.
Next comes import of resources into the Terraform state. The readiness criterion is not "import passed", but an empty plan: if the check shows zero divergences, the description reflects reality.
We investigate each divergence separately: some are closed by a configuration edit, some by parameter clarification, some require targeted rebuild of the resource in an agreed window.
We split state by failure boundaries: networks, data, applications — separate sets. Then an edit in one environment does not drag a rebuild of others, and failure is limited to one area.
State locking and prohibition of manual console edits remove conflict: changes go only through the repository and verification.
We build the transition plan so that services do not stop. First the environment works in verification mode — the plan is built but not applied. Then we switch one module to apply and observe several releases. We revoke manual rights only after the new environment has worked without rollbacks.
Insurance along the entire path — state versioning and a snapshot before each apply. Recovery from a snapshot returns the environment to a known state, so migration proceeds in portions, without a downtime window and without stopping services.
Technical doubts and frequent questions about implementation
Engineering doubts during transition to GitOps concern not the principle itself "state changes only through an approved request", but boundaries: environment isolation, access to secrets, what remains after project handover. We answer them as we discuss at the design stage, not after the first incident.
The basic rule from which everything else follows: one directory — one state — one environment. We work on Atlantis — the service forms a plan before apply and locks state during the operation, so parallel runs do not corrupt the state file.
How are environments isolated — is this not three copies of one?
Each environment has its own directory, its own state, and its own provider credentials: the production path is inaccessible from intermediate environment checks.
Resource name coincidence ceases to be a risk — the plan for the intermediate environment physically does not see production state.
Will secrets end up in the repository and logs?
The repository contains only references, values are supplied to the runner from secrets storage by environment path. Sensitive fields are marked and are not printed in clear text, and access to production values is granted by a separate role with a limited circle of holders.
How are roles separated and change audit conducted?
Rights are tied to groups in the version control system: an engineer proposes a change, apply to the intermediate environment follows one rule, to production — only through approval. Each apply leaves a trail: who initiated, at which commit, what changed in state.
What is included in support after launch?
Updating providers and the apply service, revising access rules and rotating credentials, analyzing failures on the intermediate environment before prod, incident on-call. Plus regulations for connecting a new environment or team without editing the core.
Let's discuss the task: scope of work and start order
To make the conversation substantive, it is enough to send what already exists. We need not a presentation, but a working picture: how repositories are organized, where Terraform states are stored, who and how changes them today, and what restrictions are set by security.
To start, such a set is enough:
- repository scheme: one common or several related, where modules and environment descriptions are located;
- list of state storage locations and what has not yet been moved to shared storage;
- list of cloud accounts and projects accessed by infrastructure, together with access rules;
- engineer roles and who is currently allowed to apply changes directly;
- adopted approval order: who reviews, which edits are considered critical, whether mandatory windows exist;
- known limitations: environment isolation, audit requirements, prohibition of automatic apply;
- two or three examples of recent major changes — they show real bottlenecks.
Then the order is simple. We study the materials and return with clarifying questions on access and responsibility boundaries, then assemble an implementation plan by stages with control points, and only after that give an estimate of timelines and scope of work.
We deploy the GitOps scheme for states step by step so that existing infrastructure remains working at every step.
Describe the task and send what you already have at hand — we will return with an estimate and work plan.