Smoke testing infra: what breaks immediately after deployment
Deployment finishes with exit code zero, but ten minutes later some services don't respond — a familiar scenario.
Infrastructure as code describes the desired state, but doesn't confirm it was achieved: a module created a load balancer, but the routing rule didn't arrive because a variable turned out to be empty.
There are three recurring causes behind this. Incomplete environment — some secrets, certificates, and connection parameters remained from the previous environment or weren't overridden.
Broken connections — a service came up but can't see the database, broker, or cache because the security group or DNS record weren't updated. Ungranted permissions — a role was created, but the access policy to storage or image registry wasn't attached to it.
A separate category is configuration checking that isn't run anywhere as a standalone step.
Modules apply successfully, but no one verifies the result against expectations: are the required ports open, does the readiness endpoint respond, have migrations been applied, are dependent services reachable over the network.
The error surfaces at the user already, and time to diagnosis is measured not in minutes but in hours of cross-team messaging.
Smoke testing infra closes exactly this gap. A short set of post-deployment checks in minutes separates "shipped" from "works": node availability, inter-service connectivity, operation permissions, configuration integrity.
The check runs against the actually provisioned environment, not against the description in the repository, so it catches the discrepancy between intended and actual state.
Next we discuss how such a check is embedded into release, what to include in it, and how to configure automatic rollback when it fails.
Smoke testing of the environment: what check automation provides
Environment smoke checks are a short set of checks right after provisioning: node availability, name resolution, load balancer response, write to storage, traffic passing from an external address to the application. The set takes minutes and catches breakage before the first user enters the environment.
Check automation reduces release time not through "acceleration" as such, but by removing manual acceptance. The engineer no longer walks services by list — checks run in parallel, and the result with a verdict is written to the build log. Release stops waiting for a human and being postponed to the next day.
Most emergency rollbacks are not about bad code but about an unready environment: an unapplied schema migration, a missing secret, an unclosed network rule.
The smoke set checks exactly these invariants and moves rollback from "noticed in production" to "failed at check." The share of emergency rollbacks drops because some changes never reach release.
Service downtime consists of detection time and recovery time, and automation compresses the first to the provisioning moment.
When infrastructure is described declaratively in Terraform, the environment is reproduced by reapplying the description, not by manual assembly from memory — downtime duration stops depending on who is on duty.
A whole class of failures also goes away, related to environment drift: checks on staging and target environment show the difference in settings before release, not a week after.
Check options: manual checklist, scripts, continuous control
A post-deployment check answers one question: the environment works in the form it was handed over. The options differ not in depth but in who runs the check, when it triggers, and how much maintenance effort it requires. We compare them without imposing one approach.
| Option | When suitable | Limitations | Labor intensity |
|---|---|---|---|
| Manual checklist | New environment types and checks requiring judgment: access rights, behavior of external systems | Steps skipped in haste, result not reproducible, history not saved | Low at start, grows with number of environments |
| Check scripts | Regular deployments of same-type stands, repeatable core of checks | Break when environment changes, cover only pre-described scenarios | Medium, owner needed |
| Continuous control | Long-lived environments where it's important to notice degradation between releases | Confirms state but not correctness of new release; thresholds and notification channel needed | Grows with number of metrics |
In practice the options don't compete but complement each other: scripts cover the repeatable core, the checklist remains for what requires judgment, continuous control stands watch between releases.
The main limitation is obsolescence: checks tied to a specific topology stop being correct after the first environment change, so their revision is part of the procedure.
The choice is determined by deployment frequency and number of environments: the more there are, the earlier the manual checklist stops scaling.
Implementation stages: from environment audit to handover of procedures
The environment after provisioning is formally ready, but before the first deploy no one checks whether services respond, migrations apply, and the stand survives a node restart.
We structure the work so checks appear before the first release, and false positives are caught on a copy, not in production.
- Environment audit — inventory of services, connections between them, ports, and external integrations. The output is a map of what makes sense to check and a list of failures that have already occurred.
- Check scheme — divide scenarios into fast and deep: connectivity, service availability, read/write to storage, migrations, recovery after node restart. For each, fix the "passed" criterion.
- Check setup — move the scheme into existing automation and bind it to provisioning events. Checks don't replace monitoring: they answer whether the environment can be handed over further.
- Run on a twin stand — deploy a copy of the production topology, measure full run time and behavior when a node fails. Here we also select the maximum response wait time and retry count.
- False positive handling — separate degradation from noise: services not yet settled, start races, temporary unavailability of external dependencies. Target — zero red runs on a healthy environment.
- Procedure handover — describe actions when a check fails, the owner of each scenario, and the order of extending the set when new services are added.
Check suites, environment schemes, and release procedures
Checks are stored not in chat but in the repository next to the infrastructure description.
Suites run immediately after provisioning: manually during incident analysis and automatically in the release pipeline — as a mandatory gate before handing the stand over to development teams.
Each delivery element is described in the check documentation: what exactly is checked, what state is considered normal, what to do on discrepancy, and whom to contact for permissions.
Suites are versioned together with environment schemes — when node composition changes, it's visible which checks are outdated, and rollback of rules doesn't break already working stands.
- Basic-level check suites: node availability, name resolution, load balancer response, storage readiness.
- Application-layer checks: service response, message queues, service account permissions, access to secrets.
- Environment schemes: node composition, networks and subnets, load balancing, connections between services and external systems.
- Release procedure: steps from provisioning to handover to operation, stop points, and stand readiness criteria.
- Access rules: separation of engineer and service account roles, procedure for granting and revoking rights, action audit.
- Check documentation: suite purpose, expected results, analysis of typical discrepancies, procedure on failure.
- Check versioning: change history, binding to environment scheme version, notes about outdated suites.
Case study: reduced share of emergency releases after deployment
Baseline
Deployment to a Kubernetes cluster ended with green pipeline status but didn't mean the service actually worked: 18% of releases were rolled back in the first hours. Causes repeated — unapplied schema migration, unavailable dependency, wrong secret in configuration.
Diagnosis took 40 to 60 minutes: the engineer manually compared cluster events, logs, and environment variables, and in night releases checks went through a shortened list.
What changed and what we got
Checks became a mandatory step after provisioning and before traffic switching: endpoint availability, pod readiness, dependency service responses, migration application, service account permissions.
The run takes 90–120 seconds; on failure traffic stays on the previous release — rollback stopped being a manual operation. We cut false positives by waiting for pod readiness within a specified time rather than failing on the first request.
The share of rollbacks decreased from 18% to 3%, recovery time — from 50 minutes to 7. Emergency releases outside the procedure window dropped almost threefold: the failure cause is visible in the run report, not reconstructed from logs after the fact.
The check suite is versioned together with infrastructure code, so a new service gets them when added, not after the first incident.
How do smoke checks differ from full environment regression?
A smoke check answers one question: can traffic be allowed into this environment right now.
Coverage — 15-25 checks that pass in minutes: did the load balancer come up, does the application respond, are the database, queues, DNS, and certificates available, does login and creation of a test entity pass end to end.
Everything else is noise for it — it doesn't look for errors in business logic, it confirms viability.
Full regression checks behavior: hundreds of scenarios across all significant branches, including rare combinations of roles, permissions, and data states.
It takes hours, requires a prepared dataset and stable environment, and its result is the verdict "functionality not broken," not "environment is breathing."
The boundary lies in the object of checking: smoke looks at infrastructure and component connectivity, regression at application logic. The first gives a binary answer in minutes, the second a detailed report in hours.
If smoke starts answering "is everything working correctly," it stops being fast and turns into a trimmed-down regression.
Mixing them is risky in both directions. Bloated smoke takes tens of minutes, unstable scenarios leak into it, and the team gets used to shipping over a red result.
Replacing regression with smoke gives the opposite effect: deployment passes green, but defects surface for users because connectivity was checked, not behavior.
The practical separation is simple: smoke — a mandatory barrier in the deployment pipeline, blocking traffic transition to a new version; regression — on schedule or before a major change.
Their readiness criteria are also different, and that's exactly why they are kept as separate suites.
Addressing doubts: check coverage and flaky triggers
Check coverage is set not by their number but by the list of failure paths.
After provisioning we check not only "node came up," but also connectivity between segments, load balancer response, service account permissions, storage availability, and correctness of DNS records.
This way a configuration defect surfaces before the first application deployment, not when it's already in use.
We treat flaky triggers with cause, not ignoring: wait for readiness via state polling instead of fixed pauses, separate retries for "infrastructure still coming up" and for a real error, isolation of checks from each other.
A test that sometimes fails for no reason is worse than a missing one — the team stops believing it and misses a real failure.
Will coverage be enough if some environments are deployed manually?
For such sections we fix control points outside: address availability, service response, state of queues and schedules. We move manual steps into the procedure so that during handover it's visible where to expect discrepancies.
How to distinguish a false positive from a real incident?
Each check writes what exactly it waited for and didn't get: response code, target node, wait duration. From this trace it's visible whether it's degradation or the test touched a not-yet-ready component. Readiness checks in Kubernetes are built on object statuses, not on pauses.
What about maintaining the check suite after handover?
We hand it over together with the procedure: which checks are mandatory before rollout, how to connect a new environment, where to view run history.
For the first while we support it ourselves — adjust thresholds and add scenarios for infrastructure changes; after that the customer's team maintains the suite.
Let's discuss your task and return with a check plan
To return with an estimate and check plan rather than generalities, we need three things from you: description of the environment (cloud, own servers, or mixed scheme), current provisioning method, and list of incidents you've already caught manually.
If you have rollout logs and configurations, attach them; that's enough to see failure points without long questioning.
Then we work in two passes: first analyze the environment and collect the list of checks, then agree on scope. We return with a package:
- infrastructure map with places where failure goes unnoticed all the way to production;
- composition of post-provisioning checks: service availability, dependency state, correctness of permissions and environment variables, behavior on node restart;
- scheme for embedding into the delivery pipeline: which checks block rollout, which only signal the on-call engineer;
- requirements for access and test environments where we run checks;
- estimate of work by stages with division: what we cover, what remains on your side;
- implementation timeline — from scope agreement to first runs on the production environment and documentation handover.
Describe the task in free form: what environment, how it's deployed, what already breaks. We'll return with an estimate, check plan, and implementation timeline.