Why manual Terragrunt configurations diverge as environments grow
While there are only two environments, Terragrunt configurations are maintained by copying a directory.
dev, stage, and prod differ by a couple of variable values, and each new environment is a copy of the previous one along with settings from other environments: cluster address, pool size, backup policy.
After a few releases, no one remembers why stage has a different set of resources enabled, and why prod still has a parameter from a previous run.
Then infrastructure configuration drift begins. One service pulls a module from the current branch, another is pinned to an older one, a third defines resources directly inside the environment and lives separately from the shared module.
A change in shared code reaches only some services, and the discrepancy is discovered in the cloud rather than during review.
Environment discrepancy surfaces during deployment: the plan shows dozens of changes where one line was expected, and a deployment failure occurs at the apply stage, when some resources have already been recreated.
Such an error reproduces only in one environment, so neither tests nor code checks catch it. Rollback also does not happen in a single run — the state has to be sorted out manually.
The root cause is not in the files themselves, but in manual configuration management: there is no single source of truth for parameters, no shared module version across all environments, no check that catches divergence before apply. We'll break down how this is solved.
What a unified infrastructure description layer provides: fewer manual edits
Terragrunt here is not an end in itself, but a way to maintain a unified infrastructure description layer: test, pre-production, and production environments are assembled from the same modules, and differences are moved into parameters. Below are the effects that are visible in metrics, not in feelings.
Speed of environment provisioning and recovery
- A new stand for testing or load testing is assembled from the same modules: environment deployment time is measured in tens of minutes instead of days of back-and-forth and manual steps.
- Recovery after a failure is a re-run of the same description: the stand returns to a known state, not "however it turned out".
- An environment is torn down with one command, so temporary stands do not live for months and do not accumulate divergence from the repository.
Predictability of changes and rollback
- The change plan shows in advance what will be created, changed, and deleted, so the reduction in manual edits is visible by the number of operations outside the repository.
- Rollback of infrastructure changes is a return to the previous description: the state is brought to it by the same plan, without sorting out manual actions 'from memory'.
- Environment reproducibility rests on shared modules: discrepancies are caught at the plan stage, not in the production environment.
"Previously, a stand for load testing was raised over two days and half of it was fixed manually — now it's one command, and it replicates production one-to-one," says our engineer.
Terragrunt, workspaces, or modules: choosing for environments
When there is more than one Terraform configuration, a fork appears: keep environments in workspaces of one directory, separate them into directories with modules, or put Terragrunt on top.
The effort to add a new environment and how quickly development and production begin to diverge depend on the configuration organization scheme.
| Option | When it fits | Limitations |
|---|---|---|
| Workspaces | The resource set is the same, environments differ in sizes and addresses | Shared state file and provider: an error in development affects production |
| Directories and modules without a wrapper | The resource composition per environment is different, there are few services | Backends and variables are copied manually, each service is a new set of files |
| Terragrunt | There are many environments and services, composition and parameters differ | An extra abstraction layer: inheritance and paths must be understood, otherwise errors in dependencies |
Limitations of each option
Workspaces save directory structure but do not provide isolation: a single plan or apply failure affects all environments at once.
Directories and modules isolate reliably, but as the number of services grows, maintaining dozens of nearly identical descriptions becomes more expensive than the service itself.
Terragrunt removes duplication but requires discipline: an incorrectly assembled path or dependency order manifests only during apply.
How to choose as the number of services grows
Up to five to seven services and two environments, modules with separate directories are sufficient — predictable and without extra layers. Beyond that, the criterion is one: how many manual edits are needed to bring up a new service in all environments.
If edits are made in several places, we move to Terragrunt with a shared set of modules and an environment defined by parameters — adding a service comes down to one description.
Implementation stages: audit, design, setup, validation, handover
The work proceeds sequentially — from inventory to a regulation that remains with your team. Without such order, Terragrunt quickly turns into another layer of copy-paste on top of Terraform, so structure and checks are designed together, not as an afterthought.
Audit and inventory
- Audit of infrastructure configurations: we collect repositories and state files, find duplicate resource descriptions, manual edits in the provider console, and discrepancies between environments.
- Design of the environment structure: a common root file with state storage and provider settings, separate directories for dev/stage/prod, reusable modules instead of copies.
- Environment migration plan: migration order, state transfer without recreating live resources, a rollback point at each step.
Validation before production
- Validation of infrastructure changes: plan in the pipeline before apply, review by a second engineer, format and policy checks, prohibition of destructive operations without confirmation.
- Dry run on a copy of the production environment: the environment is raised from a snapshot, the plan is applied, a repeat plan should be empty, rollback is checked separately.
- Handover of infrastructure to the team: documentation, change regulation, new environment template, engineer training, and access separation.
After that, a new environment is raised by copying the environment directory and editing variables — without rewriting configurations and without manual actions in the console.
What remains in the repository: modules, directories, policies, regulations
We hand over not a set of one-off scripts, but a repository from which the environment is raised repeatedly: the same run on a clean account produces the same infrastructure. Below is what exactly is in the deliverable and how it is used after launch.
Repository contents
- Directory hierarchy: environment → region → service. A new location is added by copying a level, modules do not need to be edited.
- Reusable modules with pinned versions: network, clusters, databases, load balancers. Updates are done by changing the version, not by editing the live environment.
- Terragrunt configurations: environment values are moved to separate files, shared logic is inherited — divergence between stands is visible when comparing versions.
- Remote state storage with locking: parallel runs by two engineers do not corrupt the state, change history is available for each resource.
- Access separation: read, preview, and apply are separated; permissions on the production environment are narrower than on the test one.
- Pre-apply validation policies: prohibition of open ports, protection against deletion of critical resources, mandatory owner labels.
Documentation and change regulation
- Change regulation: any change goes through preview and review by a second engineer, apply is from the shared branch, not from a workstation.
- On-call shift instructions: how to roll back a release, where to view state, whom to contact during an incident.
- Access matrix and repository map on one page — a new engineer starts working without verbal retellings.
Case study: how environment deployment stopped being manual work
The customer is a product team that ran three environments in one cloud. The infrastructure partly lived in configurations and partly was raised manually from the console: it was impossible to reproduce the stand from scratch from the repository.
What it was before
Deploying a new environment took two days and about 40 manual steps by two engineers. Every third run ended in rollback — discrepancies in resource names and parameters surfaced only at the apply stage. Analyzing one such failure took up to four hours of manual comparison.
What it became after
We decomposed the configurations into reusable modules and separate environment layers, leaving only the differences in them: sizes, zones, variables.
The wrapper over the configurations — Terragrunt — was configured so that parameters are substituted from one place, and the state of each layer is stored separately.
The environment is raised with one command in 20–25 minutes, and three manual steps remain. The share of rollbacks decreased from one third of runs to one case in twenty, and returning to the previous state takes minutes.
"We used to assemble a new stand for a client together and push back deadlines. Now one engineer raises it before lunch, and we no longer had rollbacks due to configuration discrepancies." — customer's technical director
How Terragrunt differs from workspaces and when it is unnecessary
Terraform workspaces are isolated states within a single configuration. The same code is deployed to dev, stage, and prod; only the workspace name differs, and variables are supplied through value files.
The scheme works as long as environments differ only in parameters: node size, number of replicas, tags.
As soon as the structure and state address diverge, workarounds begin. Workspaces have one backend for the entire configuration, so the state storage and providers cannot be set differently for different environments.
The apply order of modules is arranged manually, and copy-pasting the module block with new paths becomes the main source of divergence between stands.
Terragrunt is an add-on over Terraform that removes this manual work: it generates backend and provider blocks before run, keeps state separately for each environment, describes dependencies between modules and applies them in the correct order.
The price is an additional layer: the team must understand the template language, and version updates go through two chains at once.
The boundary of applicability is formulated briefly. "The add-on is justified when environments differ in structure and state address, and there are more than ten modules.
If the stack is the same and everything fits in a couple of modules, it is an unnecessary layer that will have to be maintained," says one of our engineers.
The practical criterion is simple. If the structure matches and only variables and scale differ, built-in workspaces are enough.
If backend, module composition, provider versions diverge, or a strict apply order between layers is needed, built-in tools are no longer enough, and the add-on pays for itself.
Concerns about state storage and access: addressing point by point
State is the most sensitive part of the scheme: if the state file is lost or overwritten by a parallel run, the tool stops seeing already created resources and offers to create them again. Below is a breakdown of concerns that most often arise before starting.
Two engineers will run changes at the same time — won't the state break?
State is stored in remote storage, locking is automatic: the second run waits or fails with an error instead of writing over.
In Terragrunt, state is split by modules, so locking works at the level of a specific module, not the entire infrastructure — teams do not block each other for no reason.
How to separate access rights between teams?
Rights are granted by roles in the cloud and paths in storage: a team writes only to its own modules and environments, and reads others' without the ability to change them. Secrets do not enter the repository — they are substituted from the secrets store at run time.
What to do if a change needs to be rolled back?
State is versioned, so the previous revision is available. Rollback is returning the configuration and building a plan: you can see which resources will be recreated and which will be changed; before apply, the plan goes for approval.
Can it be implemented gradually without stopping current processes?
Yes. We start with one environment alongside the existing scheme, compare the plan with the actual state, and make sure there are no discrepancies. Only after that do we move the next environment — the remaining processes continue to work as before.
Let's discuss your environment scheme and come back with an estimate and plan
To make the implementation estimate substantive rather than a rough guess, it is enough to send what already exists in repositories and in the team's heads: this makes it visible where configurations are duplicated between environments and where the difference is truly justified.
Terragrunt primarily removes copy-paste and parameter divergence, so without an environment scheme any estimate will be approximate.
- Environment scheme: how many environments, how they differ, what must remain shared — regions, sizes, set of services.
- Current Terraform configurations: directory structure, where state is stored, how providers and modules are connected.
- Secrets handling: what is stored in the open, where it is stored, what should be substituted at apply time.
- Access requirements: who needs access to the production environment, who grants it, how the change log is maintained.
- Rules for versioning modules and providers: whether versions are pinned, how update and rollback proceed.
- Release procedure: build pipeline, manual steps, change windows, who confirms apply.
- Expected result: which parts should become reproducible and what will remain a manual exception.
After discussing the task, we come back with a work plan: repository review, a pilot on one environment, migration of the rest to a shared root with modules, documentation handover, and team training.
The procedure for working with a contractor is transparent — each stage ends with a verifiable result in your infrastructure, not a report about work done.
Describe the task in free form and attach what you have: we will come back with an estimate, plan, and work boundaries.