Synthetic monitoring: when an outage is visible only to the customer
A service responds to a liveness check, graphs are flat — yet the share of completed orders sags.
The outage is visible only to the customer: the route to one of the nodes goes into a black hole, a certificate has expired on an edge load balancer, a provider in a specific region is cutting traffic to the payment integration.
Internal monitoring looks at processes and resources from inside the perimeter — for it, such a failure does not exist.
Synthetic monitoring covers exactly this class of outages.
Scenarios on Playwright are run on a schedule from external points in different regions and networks: open the catalog, add an item to the cart, take the order through to confirmation, check the response from the payment integration.
The path is the same as a real buyer's — outside the perimeter, through the public entry point and the content delivery network.
Each service availability check records not only the fact of a response, but also step-by-step timing, codes, redirects, and the presence of a key block on the page.
A discrepancy between points immediately localizes the problem down to a region, carrier, or specific node — the on-call engineer does not have to figure out whether "everything is down for everyone." An alert triggers on a step-time threshold, not only on complete unavailability.
The practical effect is a change in the order of events: we learn about an outage in a specific region, not the customer who has already left.
Lost orders due to unavailability stop being an invisible line in a report: the outage has a start time, affected points, and confirmation of recovery. Let's break down how this is covered.
What regular service availability checking provides
Regular checking is not a port ping, but a reproduction of the user path from external networks: a probe once a minute from several independent points opens the page, logs in, adds an item, and reaches order confirmation.
Scenarios are described as code and versioned alongside infrastructure configuration — via Checkly, so the check goes through the same review as all other changes.
A successful response code does not yet mean the service is working: a form can silently lose submissions, an external payment service can respond with a failure, a certificate can expire at night.
Such outages are found by the probe before the first complaint: the signal goes to the on-call engineer within one to three minutes, whereas the customer writes to support dozens of minutes later.
We set the threshold on a series of failed attempts so that a single network blip does not wake the on-call engineer.
Localizing the cause of the failure is the second half of the effect. The probe shows at which step it breaks: name resolution, certificate validation, load balancer response, or page content.
The breakdown by points separates a complete outage from unavailability from one region — the engineer starts not with the question "what went down," but with a specific segment of the chain.
Hence also the service recovery time: troubleshooting takes minutes instead of an hour of searching. Some classes of outages disappear entirely — expired certificates, dropped external integrations, scenarios that fail on certain devices.
A reduction in the number of customer complaints is the consequence: support stops being a monitoring channel, and the share of outages that we learn about from users falls from the majority to a handful.
Synthetic monitoring modes: comparing approaches
The check mode is chosen based on what exactly needs to be proven. A single-step page check is one request to the address with response code and response time control: this is enough for entry points and static sections to confirm that the service responds.
A multi-step order scenario repeats the buyer's path — selection, cart, checkout, confirmation — and verifies responses at each step. It catches integration breaks that a single request misses: the page opens, but the order does not go through.
| Option | When it fits | Limitations |
|---|---|---|
| Single-step page check | entry points, static sections, quick availability control | does not see outages after the first page: "the page responded" does not yet mean "the order is being placed" |
| Check from multiple regions | when quality depends on traffic routing and customer geography | noise grows: local networks produce false positives, schedule and threshold synchronization is needed |
| Multi-step order scenario | critical business paths: cart, payment, confirmation | more expensive to maintain — test data and accounts, dependence on external services, steps are modified with every interface change |
| Scenario with login to the personal account | closed sections where an outage is visible only to an authorized customer | service accounts and access rotation are needed, the check is sensitive to security policies |
The common limitation of synthetic checks is that they confirm the path is passable, but do not show how a real customer goes through it.
Therefore, scenarios are supplemented with measurements from real sessions, and response-time thresholds are set with a margin so that the notification comes on a failure, not on a traffic spike.
Implementation stages: from reviewing entry points to handover to support
Synthetic monitoring does not begin with dashboards and graphs, but with an inventory of what should generally be available from the outside.
As long as there is no list of critical entry points and their owners, checks turn into a set of random requests that catch the wrong things. The order of work below is a sequence in which each next step relies on the result of the previous one.
- Audit of entry points. We collect external addresses, APIs, authorization forms, partner and payment endpoints, and separately mark those whose failure is not visible to internal monitoring. For each point, we record the owner and expected behavior.
- Prioritization. We divide the list into critical and secondary scenarios: check intensity and alert loudness must correspond to the real impact on operations and customers.
- Check design. For each point, we describe the scenario: response code, redirect chain, presence of a key fragment in the response body, response time, certificate validity period. We check from multiple regions to distinguish a service failure from a network accident on the route.
- Sensitivity thresholds. We configure not "one error — alarm," but a series of failed attempts or a share of failures over a window; separate thresholds for unavailability and for response-time degradation.
- Alert channels. Routing by criticality: on-call engineer, team chat, ticket in the tracking system. Deduplication and suppression of duplicates are mandatory, otherwise one incident generates dozens of messages and attention to them drops.
- False-positive testing. We run checks in observation mode, compare with real incidents from the previous period, and look at noise during non-working hours and scheduled maintenance. We adjust timeouts, windows, and message texts here, not after the first night alert.
- Handover to support. We provide the on-call instructions: what a trigger means, what diagnostic steps to take, whom to involve. We separately record the procedure for revising scenarios during releases and changes to entry points.
What we hand over together with configured checks
Checks are useless if after handover no one understands what exactly they control and what to do when they trigger.
Therefore, together with synthetic monitoring, we hand over everything the on-call shift needs to work independently: not only a set of scenarios, but also the logic of their changes, notification routes, and the procedure for an incident.
The documentation describes checks at a level of detail that lets your team modify them with their own hands, without contacting us.
- Service coverage map — which entry points, key user scenarios, and external dependencies are checked, which areas are deliberately left without checks and why.
- Check configurations — scenario steps, run frequency, launch points from different networks, expected response codes and thresholds after which a check is considered failed.
- Alert settings — notification routes by on-call duty, rules for grouping related triggers, suppression during scheduled maintenance, resending if no one acknowledged the signal.
- Response procedure — what is considered an incident, who acknowledges the signal and within what time, in which cases a check is temporarily disabled as false and who makes that decision.
- On-call instructions — a step-by-step breakdown of a typical trigger: from opening the check report to notifying the responsible parties, escalation, and logging.
- Check documentation — description of each scenario, its owner, binding to the service, procedure for adjusting thresholds and adding new checks.
- Knowledge transfer to the team — review on live triggers, answers to questions about disputed thresholds, issuing access and distributing roles.
After handover, we usually stay for a short support period so the team goes through several real incidents alongside us. Then the checks live their own life: new scenarios are added as services grow, outdated ones are removed, and the coverage map remains current.
Case: how we removed false positives and sped up response
When the customer's infrastructure grew to several dozen services, synthetic checks were launched from a single point and triggered on a hard response-time threshold.
Any network spike or scheduled database maintenance raised a wave of alerts: in a week, the on-call engineer received more than a hundred signals, of which two or three were real.
A real incident drowned in this stream — it was noticed on average after half an hour, because no one looked at the next trigger as a signal anymore.
What we rebuilt
Checks were separated by levels. External scenarios — login, authorization, the key user path — were moved to several independent points of presence to distinguish a service outage from problems on a specific carrier's channel.
The threshold was replaced with confirmation: an alert was sent only after several consecutive failed attempts, and short response-time deviations were moved to observation without waking the on-call engineer.
Scenarios were described as code and assembled in Grafana, where the signal is immediately visible next to service metrics.
What changed
The share of false positives fell from 86% to 7%, and noise in alerts stopped being background.
The average response time to a real failure decreased from half an hour to two or three minutes: now one signal arrives, tied to a specific user scenario, with a ready procedure for the on-call engineer.
Incident triage takes minutes instead of hours — the check history immediately shows what exactly broke and at which step.
What to do if checks are already configured but the signals are useless?
Most often the issue is not the tool, but that checks live from the first launch: they poll one address, know nothing about part of the services, and trigger later than the customer notices the failure.
The signal arrives when the problem has already spread across users, and the on-call engineer gets used to ignoring it — trust in alerts breaks exactly here.
Signals lag for two reasons: the check period is longer than the acceptable downtime, and confirming a failure takes several cycles.
We count from the target detection time — how long a service can fail to respond before the customer sees it — and use that to select the frequency and number of confirmations.
At the same time, we separate external and internal checks: from inside a closed perimeter, routing failures and certificate issues are not visible.
A real failure is separated from noise by moving from "process responds" checks to end-to-end path scenarios: login, key operation, result output.
False positives go away when a scenario is not tied to a single external service, has a response-time margin, and runs from multiple points. We address the remaining noise specifically — adjust the threshold or remove the check, rather than muting the channel.
On-call duty and escalation are configured by service: the alert reaches those who can fix it, with quiet hours for scheduled maintenance and a rule of "not acknowledged — escalated up the chain." Then a nighttime signal becomes a reason to act, not background.
The rebuild is iterative: inventory of what is being checked now, then scenario checks of critical paths, then cleanup based on the results of each trigger. The trust metric here is the share of alerts that actually led to some action.
Doubts and limitations: what is important to consider in advance
A synthetic check sees exactly the path that was built into it. If a scenario logs in under a service account and accesses one address, it will not notice a failure at the payment or export step, and a green status will create false confidence.
A check from an external network will not show degradation between services inside the perimeter, and vice versa: launched inside, it will not see problems at the provider or in DNS.
Load is a separate topic. Frequent checks with a full traversal of the scenario at each location add up to a stream of requests noticeable on a weak test environment and on external services with limits.
Therefore, the frequency and depth of the scenario are calculated in advance, and heavy checks are moved to a separate environment.
The check is green, but customers are complaining. Why?
The scenario follows one fixed route, while live traffic goes through other steps, cache, region, and customer version.
Synthetic monitoring reliably catches a complete outage, but does not see partial degradation: a slow response, errors on some requests, problems for certain user groups. Alongside it, you need real traffic monitoring and logs.
Should scenarios with authorization be checked?
Yes: login, session renewal, and logout break most often, and without them the check will not reach the personal account. Credentials are stored in secrets, not in the scenario text, under a separate read-only role with a restriction on the addresses from which login is allowed.
Internal and external checks — which to choose?
Both layers cover different things. External ones show what the customer sees: domain availability, certificate validity, response on key pages. Internal ones show the state of services behind the load balancer. With only one layer, the picture will be one-sided.
Let's discuss the task and come back with an estimate and plan
To come back with an estimate and work plan, rather than clarifying questions, we need a few inputs about the infrastructure and expectations. We collect them in the first conversation — it sets the starting order and boundaries of the checks.
- list of services and entry points whose availability matters, marked by criticality: what can be down for an hour, and what must not go down at all;
- user scenarios that need to be checked end-to-end — login, search, order placement, payment, external interface response;
- how you currently learn about a failure — complaints, on-call engineer, a separate system — and how much time passes before a signal;
- where the services are hosted and from which networks checks should come, so as to see the picture from the outside, not only from within;
- access requirements, approval mode, and scheduled maintenance windows, so that checks do not wake the on-call engineer during planned maintenance;
- expected starting order: a pilot on one critical environment or the entire list of services at once.
From this data, we assemble the scope of check work: which scenarios to automate, at what frequency and from which points to run them, how to configure notifications and cut off false positives.
Then we come back with an estimate of implementation timelines, stages, responsible parties, and acceptance criteria. Describe the task in any convenient form — we'll discuss it and come back with a plan.