Smoke tests after deployment: where the release breaks
A release is considered successful when the pipeline reaches the last step and returns a 'green' status. In practice, this only means that the artifact was built and tests passed on the staging environment.
After traffic switching, the service may not respond: dependencies didn't start, a secret wasn't picked up, the port is occupied. A failed deployment is formally not rolled back — it is considered delivered.
A separate class of failures is configuration desynchronization and environment failure. The set of variables on staging and in production diverges, a schema migration is applied without a rollback, an external dependency responds with a different code than during acceptance. This is not visible in the build log: checks end before the product starts serving real requests.
The cost of a failed deployment is a downtime window, the length of which is determined not by regulations but by how quickly a user writes to support. Then a manual rollback under pressure, searching for the last working version, and re-deployment. The later the failure is detected, the less confidence that the previous state is fully restored.
Smoke tests after deployment are an automated check after release: a short set of calls to critical paths that runs immediately after deployment and before traffic is opened. Below we analyze how this is covered — from selecting checks to automatic release rollback.
What automated post-release checking gives
Automated release checking answers one question: does the main user scenario work right now. A set of 10–20 short checks hits critical paths — login, checkout, catalog display, queuing a task.
Such a set does not replace full regression: its task is to give an early signal and not let degradation spread further across environments.
Failure detection time drops from hours to minutes: the report is generated in the same run as the deployment, so the on-call engineer learns about the failure before the first support request.
The share of manual service walkthroughs is reduced to zero on typical releases — instead of a thirty-point checklist, the engineer investigates one failed step and its cause.
Eliminated class of failures — unavailable dependencies, failed migration, cold cache, wrong environment configuration.
Environment readiness checking relies on the standard orchestrator mechanisms in Kubernetes and does not require application modifications: the result is visible before traffic goes to the new instance.
Release predictability no longer depends on who exactly rolls out the release: one set of checks, one report, the same bar for test and production environments. The rollback decision is made based on facts, not feelings.
Modes of running smoke checks: comparison of options
The modes differ not in depth, but in where the decision point is: before deployment on all nodes or after, on part of the traffic.
Running checks in the pipeline gives an answer in minutes, canary release — during the observation time, but catches degradation that a point set does not see. Below are real alternatives to choose between at the start.
| Option | When suitable | Limitations |
|---|---|---|
| Manual checklist after release | Rare releases, legacy system without tests | Depends on a person, low repeatability, failure is learned from users |
| Smoke set for the release critical path | Main mode: login, payment, key scenario | Catches obvious breakages, but does not see partial degradation |
| Selective checks on changed services | Frequent small releases, monorepo | A dependency map is needed, otherwise related modules are skipped |
| Canary release with automatic rollback | High cost of error, metrics and load balancer available | Longer setup, thresholds and observation window needed |
In practice, options are combined: a mandatory gate of selective checks along the release critical path, then a canary release with automatic rollback based on metrics.
The limitations of checks should be fixed in advance — a smoke set does not replace load testing and does not confirm data correctness; it answers the question that the system is alive and key scenarios pass.
Procedure for implementing checks in the delivery pipeline
The implementation procedure depends on the maturity of the release cycle, so we start with an audit, not with setup: otherwise checks duplicate manual steps and do not cover the most expensive failure points. Below is the work procedure we follow to implement checks in the delivery pipeline.
- Release cycle audit. A map of the path from commit to production: where manual operations remain, how often releases happen, how long rollback to the previous version takes.
- Check scheme. 8–12 scenarios for the question 'is the system alive': login availability, authorization, key read and write operation, availability of external dependencies. A longer set makes checking a bottleneck of the release.
- Trigger point and blocking. Checks run after deployment to staging but before traffic switching: failure stops the rollout and returns the previous version.
- Pipeline setup. Test user credentials — in the secret store, test data is created and deleted within the run, a report with logs is attached to the build with a version tag.
- Training failure test. We intentionally break a dependency and make sure the pipeline stops, rollback passes, and the notification reaches the responsible person.
- Handover to operations. We provide the regulations: who adds a scenario, what to do when an external interface changes, how to handle false positives, plus training for the on-call shift.
What remains in the repository and documentation
The set of artifacts is fixed before handover so that the on-call engineer does not reconstruct the check logic from the build history.
Everything that determines the behavior of smoke tests after deployment lies next to the service code and goes through review as a regular change.
- Check sets — one scenario per critical path: login, authorization, key database query, external integration, background worker response. Each is labeled: blocks release or only warns.
- Environment configuration templates — addresses, service accounts and timeouts for staging, pre-production and production; secrets are substituted from the store.
- Access control — separate roles for manual run, reading reports, and editing scenarios: changing a check does not mean getting release rights.
- Rollback regulations — which result is considered a failure, who makes the decision, to what state the release returns, and how it is confirmed that the previous version responds again.
- Release documentation — change card: what is affected, which checks were added or removed, which values are considered normal.
- Instructions for the on-call — analysis of typical triggers: how to distinguish a false from a real failure, where to look in the log, whom to escalate to.
- Handover to the team — assisted walkthrough on the first releases and the point after which support is handled by the internal team.
Case study: how failure detection time was reduced
The client is a team that releases updates several times a day: eight services and a shared API for mobile app and web.
Regression before release followed a 40-point checklist: login, checkout, payment and notifications were done manually by an engineer — 25-35 minutes per release.
Failures in connections between services and in configurations reached users. From rollout to the first signal took from 40 minutes to several hours, and rollback was started blindly, without understanding what exactly broke.
What was changed
We assembled a smoke set of 14 scenarios answering one question — is the product alive: API availability, end-to-end order, payment in the test environment, event delivery to the queue, service behavior when a neighbor is unavailable. The run takes 2-3 minutes.
The run was tied to two points: immediately after deployment to the test environment and after a canary release to 10% of traffic.
Failure of any step stops the rollout and triggers rollback, and results go to Prometheus — you can see which scenario started failing after a specific change.
Failure detection time was reduced to 2-4 minutes instead of 40-90. Failures of the class 'service did not start' and 'incorrect configuration' stopped reaching users: over a month, out of several dozen releases, none reached production in a non-working state.
Manual pre-release checking was reduced by about 80% — only release candidates go through the checklist. The team started deploying smaller and more often: the cost of error dropped, and release stability increased.
What to do if deployment runs but there are no checks?
If the pipeline deploys code, and after deployment no checks are run, the release is considered successful by return code, not by the fact of a working service.
A green build and a failed production environment is not a tool failure but a missing stage: between 'artifact deployed' and 'traffic switched' there is no gate capable of stopping the deployment. The symptoms repeat from release to release, and the diagnosis is made quickly from them.
| Symptom | Cause | First action |
|---|---|---|
| Build is green, service returns gateway error | Readiness check is formal or absent | Separate readiness route, polling with retries |
| Errors are found by users | No run over critical paths | 5-7 smoke tests: login, read and write of key entity, public route |
| Migrations passed, application does not start | Schema compatibility with previous version not checked | Check schema version before traffic switching |
| Failure was learned about hours later | No run timeout and notification | Run timeout and notification to on-call in the same channel |
| Rollback takes longer than the failure itself | Previous artifact not saved, procedure not rehearsed | Store the latest artifacts and run rollback on staging |
The implementation priority is simple: first a readiness probe that distinguishes a started process from a service actually receiving traffic, then a short set of smoke tests for critical paths, then a run on a canary group with automatic rollback on failure.
This order gives an effect already on the next releases: the time from deployment to failure signal is reduced from hours to minutes, and smoke tests after deployment cease to be a manual operation for the on-call.
Technical doubts about smoke checks
Smoke checks break not from lack of tools, but from noise: false positives on every second release, an environment that behaves not like production, and test data that drifts between runs.
Below are four doubts most often heard at the evaluation stage, and how we address them engineering-wise.
How do you filter out false positives so alerts are not disabled within a week?
We calculate the share of noisy runs and keep a budget: if a check fails for an infrastructure reason more often than once in twenty deployments, it is either fixed or removed from the blocking set to quarantine with a note.
Unstable checks get a retry with a limited number of attempts and an explicit timeout for the dependency — this way we distinguish a real service failure from a slow response from an external provider.
The run result goes to the same channel as deployment metrics, so the cause is visible without reading logs manually.
In which environment should checks be run after deployment?
We check on the same environment where we deployed: a separate 'prod-like' staging gives false calm, because configuration and connection discrepancies surface with users already.
The smoke set reads only those addresses and roles that are actually available after release, and access is granted by a service account with minimal rights.
For pre-production, the environment is raised from the same templates as production — otherwise the check does not transfer.
What to do with test data so as not to affect real data?
Each run works with its own data: run prefix in identifiers, separate test tenant, automatic cleanup by time-to-live. Checks by default only read — creation and deletion of objects are moved to a separate short group that can be disabled on a sensitive environment.
This way the set does not depend on the database state and does not accumulate garbage between releases.
Who maintains the check set after handover?
The set lives as code — in the same repository as the service: adding a route or changing a contract goes as a single change with review.
We fix the regulations: when a new critical scenario appears, a check is added to the release checklist, and once a quarter the set is reviewed for outdated steps.
After handover, your engineers edit checks without us — all that remains is to read the documentation on the set structure.
Let's discuss a check scheme for your release cycle
So that the post-deployment check scheme fits your release cycle rather than breaks it, we start with a short discussion of the task.
We need just a few inputs: how build deployment is organized, how many environments and how often you release, what is considered a critical path for the user.
From the customer side, usually needed:
- access to the repository and build launch settings, or an export of current configurations;
- list of environments and the order of build promotion between them;
- list of key scenarios whose failure should block the release;
- accounts and test payment data for running checks;
- a window during which it is acceptable to pause deployment when a check triggers;
- contact of the on-call engineer who will confirm rollback criteria to the previous version.
In return, we come back with an estimate of the scope of work and an implementation plan: which checks we put on automatic launch after deployment, how long the set takes, under what conditions it blocks build promotion, and how it triggers rollback.
We will propose the check scheme in a form ready for work, not for retelling: a list of steps with expected result, launch and rollback configuration, separation of checks into mandatory and informational, order of notifying on-call staff.
Then — release support: we expand the set along with product changes so that it does not become outdated together with critical paths.