Infrastructure Testing: Terratest and kitchen-terraform for Terraform

Terratest: why manual Terraform validation doesn't scale

Manual infrastructure testing looks reliable right up until the first change that passes review and quietly breaks the environment.

Infrastructure as code describes the desired state, but the description itself is not executed by anyone — an engineer reads terraform plan with their eyes and decides that everything looks reasonable.

Terraform validation in this mode comes down to comparing the diff with expectations in their head, rather than actually applying and measuring the result.

Then a divergence appears between the description and the environment. Someone fixed a firewall rule through the console, refined an access policy manually, added a resource outside the repository — and the next apply either overwrites the manual edit or fails due to a conflict. Drift accumulates silently, because the repository is clean, but reality no longer is.

Access permission and dependency errors surface at the worst moment — in the middle of applying changes. A role was created, the policy attachment fails, some resources already exist, some do not, state is locked.

Rollback is done manually, from memory, under pressure: the more modules involved, the higher the chance of leaving the environment in an intermediate state.

Hence the scaling limit: the more environments and modules there are, the more expensive each manual validation is and the less likely it is to be performed at all.

Terratest closes this gap by provisioning real infrastructure in an isolated project, validating its availability and behavior with code, and destroying it after the run. We explain how such validation fits into the release pipeline.

What automated validation of infrastructure code provides

Automated validation of infrastructure code answers three questions: what exactly will break, how long to wait for an answer, and whether the result in production will match what was seen in the test environment. Below are the effects that can be seen in measurable terms.

Which failure classes disappear

An apply plan describes intent, but does not verify that the service responds after it. Tests on Terratest provision an environment from scratch and validate the result: network availability, permission bindings, resource presence after re-apply.

A whole class of failures — "the configuration applied, but the service did not come up" — disappears; such errors are caught before merge, not at rollout time.

How long validation of a change takes

Manual validation of a change in a test environment takes hours: provision, apply, verify. One run on an isolated copy fits into minutes, and runs for different branches proceed in parallel.

The developer receives the answer "the change is safe" before merge, and the deployment queue does not sit idle waiting for a reviewer.

Predictability of deployment to production environments

The validation scenario is the same for the test and production environments — only the variable sets and scale differ. A re-apply check is added: if the second run does not change the plan, the configuration is idempotent and does not "drift" over time.

The share of deployments requiring manual fixes after apply decreases, and rollback relies on the same proven scenario.

Terratest and kitchen-terraform: approaches to configuration validation

Configuration validation is rarely built on a single method: static analysis catches typos and rule violations before the first apply, but it does not confirm the behavior of real infrastructure.

Therefore, in a mature scheme, the levels are combined — from fast checks on every change to a run in an isolated sandbox before rollout to production.

Terratest and kitchen-terraform cover exactly the integration level: they provision an isolated sandbox, apply the configuration, validate the actual result, and then clean up the resources behind themselves.

A local infrastructure emulator is noticeably faster, but does not reproduce provider behavior. Below are the options teams actually choose between.

Validation options

In practice, the levels are combined: static rules and plan review on every change, the sandbox at acceptance, the emulator in the engineer's day-to-day work. Below we'll look at how this fits into the build and who is responsible for each level.

Stages of implementing infrastructure code validation

We implement infrastructure code validation step by step — if you dump the entire set of tests at once, the team will get dozens of red builds and stop trusting them.

Therefore, we first capture exactly what can break in the environment, and only then write checks for those risks.

1. Infrastructure audit. We analyze resource definitions, modules, and change history: where editing one resource triggered recreation of dependent ones, where configuration diverged from the actual state. The result is a list of bottlenecks with an estimate of frequency and impact of failure.
2. Validation scheme. We determine what is checked at the plan stage (resource composition, connections, attribute values) and what only in a deployed environment: service availability, access permissions, idempotency of re-apply.
3. Sandbox setup. We assemble an isolated project with test accounts, lifetime limits, and automatic resource cleanup — the environment does not accumulate garbage and does not interfere with neighboring teams.
4. Validation suite. Terratest covers "provision — validate — destroy" scenarios, kitchen-terraform covers comparing the actual state with the expected one. The suite is split by levels: fast loop, integration, end-to-end.
5. Connection to the build pipeline. Fast checks run on every commit, the full sandbox on release branches. Checks run in parallel in separate projects, and the result blocks applying changes if it fails.
6. Handling false positives. We treat flaky checks by cause — races, shared resources, too-short timeouts — not by disabling them. We keep the rule: a disabled check returns the same day.
7. Handover. We hand over the maintenance procedure, description of validation levels, and instructions for adding a new module — the team extends the suite without us.

The result is that every change is validated in minutes, not after deployment to production. Describe your task — we will come back with an estimate and plan.

Implementation artifacts: configurations, diagrams, procedures

Work is considered delivered not based on a successful run, but on a set of artifacts your team can continue working with without us.

Everything we design remains in your repository and your version control system — with change history and the ability to roll back to any previous revision.

• Infrastructure definition modules — parameterized blocks with variables and default values: environments are provisioned from one set, differences are set by values, not copies of configurations.
• Terratest validation suite — a set of scenarios that, before changes are applied, provisions an environment, validates connections between resources, and removes it afterward; plus static checks for formatting and reference correctness.
• Sandbox configuration — an isolated test environment where checks run end-to-end before production, with separate permissions and no access to production data.
• Role and access scheme — access rights separated by role: who makes changes, who approves, who has read-only access to state; changes are applied from a separate account.
• Infrastructure documentation — a map of resources and dependencies, description of variables, extension points, and which checks cover which area.
• Maintenance procedure — the procedure for making changes: mandatory checks, review requirements, conditions under which a change is not allowed into production.
• Rollback instructions — a step-by-step scenario for returning to the previous state, indicating what is checked after rollback.

We hand over the set with a walkthrough on your side: we show how to run the checks, add a new one, and extend a module for a task that does not exist yet.

Case study: automated validation before applying changes

The client is a platform team with thirty environments and a shared module branch.

Change validation consisted of manually reviewing the plan: three out of ten changes reached production and were rolled back, and an error such as "resource removed from state" could live in the configuration for weeks until the next run found it.

What we did

Tests on Terratest appeared for each module: they provision real infrastructure in an isolated project, apply the configuration, validate the actual properties of created objects, and remove everything afterward.

For machines, we added kitchen-terraform — it checks not only the apply, but also the internal state: running services, permissions, paths, port availability.

The plan became a mandatory step with expectation control: the number of created and deleted objects is compared against the reference, and an unplanned deletion stops the validation process until an engineer reviews it.

What changed

The share of changes that reached production and were rolled back decreased from 30% to zero in a quarter. Validation that took two days of manual work fits into 25 minutes and runs in parallel across environments.

The failure class "type conflict and deletion of in-use resources" is fully closed: a test for re-apply and subsequent deletion is mandatory for every module, and a release without it will not build.

Another effect is predictability of triage: the engineer gets a log with a specific mismatch, not a set of guesses. The average time from problem detection to fix decreased fourfold, because the reproduction environment is provisioned by a test, not manually.

What to do if part of the infrastructure is described manually?

Manual infrastructure appears in almost the same way: an urgent release, a resource is provisioned in the provider console, and a "temporary" solution lives for months.

You cannot destroy and recreate such resources from code — these objects already have data and traffic. The first step here is not rewriting, but importing existing resources into the state definition so that code and reality converge without downtime.

Resource inventory: we collect everything that actually exists through provider interfaces, compare it with the configuration, and mark it — managed, unmanaged, duplicated. Separately, we capture connections between objects: network, access rules, volume attachments.

These dependencies define the migration order; otherwise, the very first migration can break the route to the service.

Staged migration proceeds one environment or service at a time, and the readiness criterion for a stage is simple — an empty change plan: the code describes exactly what exists and is not going to delete anything. For resources with data, we first prohibit recreation, and only then edit the definitions.

We connect checks before the next stage: Terratest runs the plan and apply in an isolated environment and confirms that the network, access permissions, and load balancer assemble in the required configuration.

We keep state in remote storage with locking and versioning — rollback to the previous snapshot takes minutes.

Bottom line: first inventory, then one environment at a time with an empty plan, and only after that — a policy that cuts off manual edits. Describe what you have created outside the definitions — we will come back with an estimate and stage order.

Common technical questions about infrastructure validation

Engineering doubts about infrastructure validation usually boil down to four topics: how much checks slow down deployment, where to store secrets, whether the suite can be run locally, and who maintains it in working order.

Below are short answers that we discuss before the start, so we don't clarify this in the middle of implementation.

What about secrets in checks?

Secrets do not end up in code or resource definitions: values are pulled from environment variables and a secret store at run time. They are masked in logs.

Only variable names and examples of dummy values remain in the repository — the suite can be shown and handed over without risk of leakage.

Can checks be run locally?

Yes, if the engineer has access to the test account — the suite is launched from the workstation with the same command as on the validation server. Where access is undesirable, the same run happens in a container. This removes the queue and lets you debug a module immediately after editing.

Who maintains the validation suite?

The suite lives together with the modules: a resource change and a test update go in one commit, otherwise the check diverges from reality. Once a quarter we do a review — remove duplicates and outdated scenarios so the suite does not become a drag on deployment.

Let's discuss the task: how the first review works

The first infrastructure review is not an introduction or a conversation about technologies, but work with artifacts. We ask for access to repositories with infrastructure definitions, run logs, and a list of environments so we can see the actual state, not a retelling.

As an output, you receive a map, a list of risks, and an estimate and stage-by-stage work plan — the decision on each stage remains with you.

What we bring and review at the first meeting:

• A map of repositories and modules: where state is defined, what is reused, where there is duplication and divergence.
• A list of environments and differences between them — what is configured manually and what is defined by code and reproducible.
• An inventory of checks: which already exist, which change paths are uncovered, and where an error will reach production.
• A review of the Terratest test environment: how configuration apply is validated, how long the run takes, what prevents running it on every change.
• A scheme of change delivery and failure points: where the process stops, where a manual step is needed, and why.
• Rollback risks: where there are no state backups, which changes are irreversible, and how this is handled.
• An estimate and work plan: stages, readiness criteria for each, team composition, and the acceptance procedure for the result.

Describe your task — we will come back with an estimate and plan and specify the timeline for the first review. If, based on its results, you have enough capacity on your own, we will hand over the materials in your format and will not propose unnecessary stages.