Synthetic Monitoring: failures that only customers see
Server metrics are green, queues are empty, there is not a single exception in the application logs — yet customers write that the cart is not working.
This is what a hidden website failure looks like: response code 200, the request reached the application, but the user sees a frozen screen at the third step of checkout.
Server availability monitoring confirms that the service is alive and stays silent — from the infrastructure's point of view, everything is fine.
Degradation without errors in the logs is the most expensive class of failures.
The payment gateway responds, but after forty seconds instead of two; a third-party analytics script blocks the rendering of the "Pay" button; the cache serves an outdated cart file after a release.
Each case is formally successful and does not create a single line in the log, and only conversion suffers — its drop is noticed only by revenue.
Synthetic monitoring closes this blind spot: an external robot goes through key scenarios — login, search, adding to cart, payment — on a schedule and from several regions.
It sees exactly what the customer sees: rendering time, availability of elements, correctness of the response at each step.
Checking key scenarios changes the economics of an incident: instead of a complaint after an hour or two, the team receives a signal within minutes, and the release rollback begins before traffic drops.
The share of user paths under automated checks no longer depends on who is on duty today. We explain how this is closed in practice.
What scheduled imitation of user actions gives
The scenario describes the entire path: open the page, wait for rendering, log in, add a product to the cart, place an order, check the payment gateway response.
A separate group of checks works at the level of software calls — it catches degradation of data delivery when the site still opens visually.
The run goes on a schedule with a step in minutes, not "when we notice", and is repeated from several regions: what is unavailable to users in one network is not masked by a successful response from another.
That is why the time to detect a failure is measured in minutes, not hours until the first support email. An early alert comes to the on-duty channel tied to a specific step of the scenario — you can see what broke: login, payment, catalog output.
The share of failures found before a customer contacts becomes a manageable value, and it can be seen on a weekly chart.
A separate layer is monitoring external integrations. A check at the boundary with an external system shows whether the degradation is on our side or the provider's, and removes the "everything works on our side" back-and-forth.
We set thresholds with a margin: for example, a scenario is considered failed after two consecutive failed attempts from different regions, so that a single network spike does not wake the on-duty engineer.
We build such scenarios on Grafana and link them to availability dashboards so that the numbers on the chart and in the alert match. The load on support decreases because some failures are closed before tickets appear, and the rest come with ready context. Discuss the task — we will return with a plan of scenarios and their schedule.
Synthetic Monitoring modes: availability, entry point, browser
The three check modes differ in depth: the closer the scenario is to the behavior of a live user, the more classes of failures it catches — and the more expensive support becomes.
Choosing the "most complete" option is unprofitable: a browser scenario breaks with every redesign and requires manual editing, while an availability check works for years without intervention.
| Option | What it checks | What failures it catches | Support cost |
|---|---|---|---|
| Availability check | Response code and time at the node address | Unavailability, connection loss, expired certificate | Minimal: one configuration line |
| Entry point check | Response of the key address in full, including body and redirect chain | Incorrect redirects, empty page, cached content | Low: edits when structure changes |
| Software interface check | Structure and content of the response, not just the code | Contract mismatch, empty fields, sample degradation | Medium: depends on contract changes |
| Browser scenario | Step-by-step walkthrough: login, search, checkout | Broken forms, failed scripts, failure at the last step | High: scenario is rewritten on redesign |
Modes should be separated by failure class, not by page "importance".
Availability answers the question "is the node alive", entry point — "does the page serve the required content", software interface — "does data come in the expected structure", browser scenario — "does the user reach the result".
Run frequency and check geography
The interval is set by the cost of downtime: availability is checked once a minute, a browser scenario — once every 5–15 minutes, otherwise the load on production grows without benefit.
We distribute runs across 3–5 regions where users are present: this way a local route failure is not masked as a general outage, and a discrepancy in results between locations immediately points to the network, not the application.
We keep summary charts and failure history in Grafana — they show whether it is degradation or a one-time timeout. We raise an alarm only for repeated failures from two locations: the on-duty engineer does not wake up because of a single spike.
How we build checks: from inventory to handover
Checks begin not with a tool, but with an inventory of what should work at all. Then we build a coverage scheme, calibrate thresholds, and without fail break the system manually — otherwise the first real failure will also be the first test of alerts. Below are the six stages that every setup goes through.
-
Audit of critical scenarios. We collect user paths: login, search, checkout, payment, personal account. We separately mark those tied to external services and background jobs — these are the ones that most often fail unnoticed.
-
Check coverage scheme. For each scenario, we define control points: availability, response code, presence of a key element on the page, passage of the entire request chain. We make sure that checks do not duplicate each other and do not create unnecessary load.
-
Setting up checks. Schedule, run geography, authorization, test accounts and data. Scenarios run in an isolated environment, do not write to production tables, and do not interfere with real traffic.
-
Calibration of trigger thresholds. We take a baseline for each scenario over a period, then set a threshold for response time and the number of consecutive errors. This way, nighttime load spikes do not wake the on-duty engineer, while sustained degradation does.
-
Test on a real failure. We stop the service, block an external call, break the connection — and see that the alert was sent, in what time, and whether it contains enough data for diagnostics. After that, we adjust the thresholds.
-
Handover to operations. We hand over the rules, instructions for the on-duty engineer, escalation scheme, and list of responsible persons. We run the question "what to do" for each alert — if there is no answer, the check is not finished.
After handover, the coverage lives on: once a quarter we review scenarios for new sections and integrations so that monitoring does not diverge from the product.
What remains in the project after setting up checks
Setting up synthetic monitoring ends not with enabling alerts, but with a set of artifacts by which the on-duty engineer and the customer understand what exactly is monitored and what to do when triggered.
Without them, checks turn into a stream of noise within a couple of months, which is first tolerated and then turned off.
- Check coverage map — which user scenarios and external entry points are covered, where the control boundary lies, and what is consciously left out of checks.
- Check parameters — composition of steps, run frequency, set of launch points, response time thresholds, and allowed response codes.
- Alert rules — what wakes the on-duty engineer, what goes to the morning queue, who the channel is addressed to, and how a message avalanche is suppressed during a mass failure.
- Response regulations — incident pickup times, escalation points, area of responsibility of our team, and contact on the customer side.
- Instructions for the on-duty engineer — what to check in the first minutes, how to distinguish an application failure from an external provider failure, where to look at the run history.
- Monitoring documentation — description of scenarios, owners, procedure for adding new checks and decommissioning outdated ones.
- Check change log — when and for what reason the scenario was changed, so that the growth in the number of checks does not turn into an unmanageable set.
All of this lies next to the code in the repository and undergoes the same review on changes as the application itself — checks live together with the product, not separately from it.
Case study: hidden payment failures on a catalog website
A failure that was not in server metrics
A catalog with several thousand items, payment — through an external payment gateway. Server metrics looked smooth: application response time is normal, the share of 500s is not growing, queues are empty.
At the same time, support received weekly complaints "payment went through, no order", and each surfaced 4–11 hours after the failure began — when dozens of buyers were already affected.
The cause lay at the junction of services: the gateway correctly accepted the payment creation request, but the success notification did not reach the application — the provider was changing the event format during scheduled maintenance.
The application considered the order unpaid and did not send it to assembly, and the logs were empty. This is the hidden failure of an external service: each side individually responds normally.
What we set up and what changed
We assembled a full purchase scenario: product search, cart, checkout, delivery selection, transition to the payment page, waiting for confirmation in a test environment, and checking that an order with the correct status appeared in the personal account.
The run is every 5 minutes from four regions via Checkly, checking not only the response code but also the page content: this catches the case when the server returns "success" with an empty cart.
Metrics before and after: time to detect failures — from 4–11 hours to 2 minutes, share of lost payments during peak hours — from 3–5% to 0.2%, share of scenarios reaching the end — 99.8%.
The on-duty engineer receives an alert with the number of the failed step, a screenshot, and request tracing, so restoring the scenario takes one analysis, not a blind search for the cause.
The same scenario runs after every release — regression at the payment step is visible before buyers write about it.
How do synthetic checks differ from real visitor data?
A synthetic check is a robot that goes through the described path on a schedule: opens the page, logs into the account, adds a product to the cart, reaches payment, and records the response and time of each step.
The run does not depend on traffic: even at night it will show that checkout returns an error. That is why synthetic checks give an early signal and are suitable for on-duty alerts.
| What we compare | Synthetic checks | Real visitor data |
|---|---|---|
| Data source | Scenario that the robot executes on a schedule | Sessions of live people |
| When the problem is visible | At the moment of failure, before the first contact | After visitors are affected |
| Representativeness | Fixed set of paths, devices, and regions | Real spread of devices, networks, geography |
| Weak spot | Does not see paths that are not in the scenario | Little data at low traffic, noise in the sample |
Real visitor data answers a different question — what was happening to people. They show the spread of devices and networks, the share of abandoned carts, and the step at which visitors leave.
The weak spot is delay and noise: a failure is visible after live people are affected, and at low traffic the sample is insufficient to distinguish a problem from chance.
The approaches complement each other. Synthetic checks record the moment of failure and wake the on-duty engineer, statistics show the scale: how many people are affected and which groups suffered more.
If you alert only by statistics, you learn about the failure from support requests — that is, after orders are already lost.
The main limitation of synthetic checks is representativeness: the scenario checks exactly the paths that are laid in it. The set of scenarios is compared with actual routes, otherwise a new funnel or a regional version of the site will remain unobserved.
Statistics does not create such gaps, but it also does not predict — it collects what has already happened.
Technical doubts before launching checks
Checks come from external nodes and behave like a rare, careful visitor: a few requests per minute per scenario, and the scheduler spreads the crawls so that they do not coincide with traffic peaks.
We check closed sections with a separate technical account with minimal rights — this does not affect real data.
We suppress false positives with confirmation from two independent nodes and a threshold of two consecutive failures: a confirmed incident goes to the on-duty channel, not a single network failure.
We fix the division of responsibility in advance — an infrastructure failure goes to operations, a business scenario failure goes to the product owner.
Will checks create unnecessary load?
The crawl does not use your server resources beyond a normal page request, and the schedule can be thinned during marketing peaks without changing the scenarios themselves.
How to check closed sections that require authorization?
We create a read-only account, store its session data in a secure vault, and rotate it. The scenario goes through the entire chain: login form, role-based redirect, and availability of the target screen — that is where it most often breaks after a release.
Where do alerts go and who handles them?
The signal goes to the channel where the team is on duty, indicating the step, node, and response code — this is enough to reproduce the failure.
Before launch, we fix who responds to unavailability and who responds to a checkout failure, so that the alert does not hang between departments.
Let's discuss the task and return with a check plan
To estimate the scope and propose a work order, we need a minimum: a list of scenarios that affect revenue or the flow of requests, a list of environments with windows when runs will not interfere with real traffic, and access.
On the customer side, it is enough to name critical user paths and indicate who approves the scenarios.
In response, we send a plan that we follow further:
- List of scenarios — login, search, checkout, payment, personal account; we separately mark critical and observable ones.
- Coverage scheme — locations and regions, run frequency, distribution across load windows, authorization steps.
- Implementation order — what we enable in the first wave, what we add after stabilization, so as not to drown in alerts.
- Stages and control points — what we check against at each step and what we show upon completion.
- Responsible persons on both sides — who approves the scenarios, who accepts the trigger, and during which hours they respond.
- Alert rules — thresholds and recipients, so that signals do not turn into noise.
- Baseline snapshot — we record availability and response time before launch, so that there is something to compare with.
- Handover to the team — documentation, scenario templates, and instructions for adding checks without us.
Describe the task — we will return with an estimate and a check plan.