Why E2E smoke tests fail in production
E2E smoke tests are run after every release to get an answer in minutes: key scenarios—login, checkout, payment—are alive.
In production, the run fails not because of a defect in the code but because of the environment: the test account hits a request limit, the external payment gateway responds differently than on the test environment, and waiting for an element is based on a fixed pause that is not enough under real load. As a result, a red signal appears in the on-call channel noticeably more often than a real failure.
The cost is loss of trust in the check. When most failures are false, on-call stops investigating every run: first looks "by eye", then simply restarts the suite, and then ignores it.
At that moment, a regression after release—a broken payment step, a fallen authorization block—passes into prod unnoticed, and they learn about it from user reports, not from monitoring. Rollback is prepared manually, downtime is counted in hours.
Manual checks do not close this gap. The pre-release checklist grows with the product: first ten scenarios, then thirty, and it still covers only what was managed to be run manually.
The bottlenecks of the release cycle turn out to be not in the build but in manual verification—it takes more and more time and gives no guarantees.
The root is in checks tied to shared test data and external services, without explicit state expectations. We build a suite on Playwright with data isolation and a separate environment for integrations—we break down below how this is closed.
What regular checking of key scenarios after release gives
E2E smoke tests are a short suite of 10–20 scenarios that cover the critical path of the application: authorization, checkout, payment, API data delivery. The suite runs after every deployment, not once a day, so a failure is visible on the same release that introduced it.
Time to detect a failure — minutes instead of hours
While checks are run by a person, hours pass between release and the first signal, and sometimes until the next morning. An automated run returns a result in a few minutes, and the alert goes to on-call before users write about the problem. The incident has to be investigated using fresh logs, not by reconstructing events.
Shorter downtime — fewer losses
The cost of downtime is determined not by the failure itself but by the number of operations that did not get through during it. If an incident is closed in a five-minute window instead of overnight, the volume of lost orders and requests drops many times over.
At the same time, support load decreases: instead of a stream of customer inquiries, the team works with one automated signal.
A class of failures that unit tests do not catch goes away
The smoke suite identifies what cannot be reproduced locally: broken environment configuration, an unavailable database, a failed external service, an unapplied migration.
These are failures at the level of service interaction, and they are reliably caught only on a live environment.
Post-release checks run by themselves
The run is built into the deployment pipeline: the build is assembled, the application is deployed, the suite starts without human involvement.
If a scenario fails, the release is stopped or rolled back to the previous version—on-call gets a ready fact, not the question "what happened".
Comparison of approaches to production smoke checks
The approach is chosen by the class of failure that needs to be caught and by the speed of reaction. A manual checklist covers post-deployment acceptance but provides no history and is silent between releases.
Synthetic checks cover critical paths on a schedule and raise a notification within minutes; real session data gives a slice across the entire base, but only after some people could not pay for an order.
In practice, the options are combined: synthetics catch an obvious failure, canary release limits the blast radius to a share of traffic with auto-rollback, and session analytics clarifies exactly where degradation is. Below are the boundaries of each approach and the conditions under which it pays off.
| Approach | When it fits | Limitations |
|---|---|---|
| Manual checklist | One-off deployments, infrequent release cycle, no autotests yet | Silent between checks, depends on the person performing it, no run history |
| External availability monitoring | Need to know that the service responds and certificates have not expired | Looks at the response code, not the scenario: an empty page or broken form will remain unnoticed |
| Synthetic E2E smoke checks | There is a list of critical paths: login, payment, data delivery | Require maintenance of selectors, test accounts, and clean test data; rare branches remain outside coverage |
| Canary release | Frequent deployments, there is a success metric and traffic share control | Requires auto-rollback and an error threshold; such a run will not catch delayed degradation |
| Real user monitoring | Need a picture across the entire base and different devices | Reactive: the signal comes after users experience a failure, with a lot of noise from their environments |
Implementation stages: from audit to handover
Implementation proceeds in stages, and each is closed with an artifact: a list of scenarios, run configuration, and a regulation for on-call. The order is structured so that checks begin catching regressions even before transfer to the live environment.
- Scenario audit. We inventory critical paths: login to the personal account, checkout, exchanges with external services. We include in the suite what counts as an incident when it fails, not cosmetic issues on the page.
- Run scheme. We define what counts as a failure: response code, page content, response time. We prepare test data so that the run does not interfere with real users or spoil statistics.
- Setting up trigger-based runs. To the schedule we add event-based ones: a run after deployment and on a monitoring signal. Results go to the shared alert channel, alongside other monitoring signals.
- Threshold validation. We select sensitivity: how many false positives we tolerate, what response delay is normal. We revise the threshold at which on-call stops opening notifications.
- Stabilization. The first runs produce noise: unstable external services, fluctuating data. We fix the cause rather than disable the scenario.
- Handover of the regulation. We provide documentation, a "what to do on failure" instruction, and an escalation scheme. After that, the customer's team adds scenarios itself.
After handover, the checks live independently: it is enough to add a new scenario to the common suite.
Handover package: configurations, schemes, regulations
Monitoring of the production environment rests not on the checks themselves but on the agreements around them. We hand over the e2e smoke tests package together with a description of how to change, extend, and troubleshoot failures.
All artifacts are stored in the project repository and versioned together with the code—when the contractor or team composition changes, nothing has to be restored from memory.
- Check configurations — environments, test accounts, data sets, step timeouts, trigger threshold, and the list of alert recipients. They are changed without editing scenario code.
- Scenario schemes — a map of critical paths: login, search, checkout, payment, personal account. For each step, the expected result and the criterion by which the step is considered failed are specified.
- Access policy — service accounts with minimal privileges, a separate secret storage, key rotation procedure, and prohibition of using real user data.
- Incident response regulation — criticality levels, who first takes the alert, within what time they respond, when development is involved, and how to distinguish a false positive from real degradation.
- Maintenance instructions — how to add a scenario, where to edit configuration, how to run the entire suite before deployment.
- Reporting — format of run results, launch history, a slice by response time and share of stable checks over a period.
- Handover and training — a walkthrough with the on-call team, a joint pass through the first incidents, and answers on disputed scenarios.
Case study: reduced time to detect a failure after release
The client is a service with a personal account, payment, and document export; releases go out three to four times a week. Autotests ran before deployment, but after deploy no one confirmed that critical paths were alive.
A regression in checkout was discovered from support inquiries—several hours later, when some orders had already failed, and the cause lay in a recent change to the delivery calculation configuration.
What we changed
We built a suite of e2e smoke tests on Playwright: login, search, cart, payment, personal account, document export—only those scenarios whose failure means lost orders and inquiries.
The suite runs on the test environment before deployment and immediately after it against the production address, then on a schedule every 15 minutes.
On failure, a signal goes to the on-call channel with the step where everything broke and a trace: the diagnosis is read in a minute, without manual reproduction.
Stability was provided by test data isolation, resilient selectors, and retrying only network errors—otherwise on-call stops trusting the signals.
What happened
The median time to detect a failure dropped from 4–6 hours to 4 minutes; about 90% of failures are recorded before the first user inquiry. The number of missed defects is single digits per quarter instead of dozens: urgent fixes are being displaced by the normal plan.
The share of releases after which rollback was required decreased from roughly one in five to one in twenty.
We keep false positives below 5%, so night signals are not ignored.
Trust in releases has returned: deployments again happen daily, Friday release stopped being a separate risk, and the on-call engineer receives the scenario step and trace instead of a "site is down" message.
How do synthetic checks differ from user monitoring?
The difference begins with who initiates the check. Synthetic monitoring is a robot that, on a schedule, goes through a pre-described scenario in an external environment and knows the expected result of each step.
User monitoring works the opposite way: data is collected from real sessions—live browsers, devices, networks, and routes that people actually go through.
Because of this, synthetics give high signal accuracy: a failure unambiguously means a failure, and the alert goes to on-call without accumulating statistics or selecting thresholds.
The check comes from outside, so the failure is visible before the first customer writes about it. The price is that scenario coverage is exactly what we described: a path not included in the suite is not checked at all.
Real user monitoring answers a different question—what is actually happening. It shows the share of successful sessions, response time on specific routes, and the step where people drop off, for example during payment.
But the signal is delayed: we learn about a problem after it has affected customers, and noise from slow devices and third-party extensions has to be filtered out by data volume.
Therefore, the approaches do not replace each other. Synthetic checks serve as early notification of unavailability and regressions on critical paths; real session data is the source of truth and a hint about which scenarios to add to coverage.
This way the suite of end-to-end smoke checks (E2E smoke tests) remains alive: a new integration appeared or a route changed—the check is updated together with it, not after an incident.
Common technical questions and doubts about checks
Checks are integrated into the existing environment rather than replacing it, so doubts usually concern compatibility, access, and the cost of false positives. Below is what most often comes up during task discussion.
We have legacy without autotests—will checks even take root?
Yes. Smoke checks work from the outside, through real interfaces: access to source code and internal modules is not needed. We start with 5–7 critical paths—login, checkout, payment, data delivery—and extend the suite where preconditions are stable.
How does this fit with CI/CD, and will it block deployment over trifles?
Checks run in two environments. In the build—a short suite on the test stand; it stops deployment on failure.
In prod—the same suite on a schedule and immediately after release, but with notification: payment or login failure raises an alarm, and a markup discrepancy goes as a task.
False positives—how much time will investigation take?
We treat them by causes: condition-based waiting instead of a fixed pause, test data isolation per run, one automatic retry of a failed check with video and network log recording.
A stable suite yields fractions of a percent of false positives; anything higher is investigated as a defect in the check itself, not as noise.
What about production access and security?
Separate service accounts with minimal privileges, read-only where possible. Secrets live in a secure storage of the build environment and do not get into reports.
Payments are checked in the provider's test mode, test records are marked with a service flag and deleted on a schedule.
Let's discuss the task: how to prepare for the start
E2E smoke tests in production pay off when they are launched from day one and cover exactly those paths where releases break most often.
Therefore, we start with a task review: we look at what already exists in the project, which scenarios are critical, and how release quality is currently controlled.
Based on the results, we come back with an estimate and plan—the composition of checks, the order of work, and the connection points to your environment.
What is needed to prepare for the start:
- A list of critical paths: login and registration, payment, checkout, personal account—what is checked on every release.
- Source data for checks: test accounts with the required roles, order data, access to external services in test mode.
- Access to an environment that replicates production in settings and component versions—otherwise the check passes while the problem remains.
- Launch rules: after every build, on a schedule, or manually before release—we choose together.
- A responsible person on your side who approves scenarios and accepts the results report.
- Estimate and plan: stages, check priority, and the order of extending the suite without rewriting what is already done.
The first suite covers the main scenarios and gives an unambiguous signal: the release is safe or there is a regression with the exact failure point. The next step is to describe the task: what the product is, which scenarios are critical, and what hurts during releases. We will come back with an estimate and plan and propose a start date.
