SLA monitoring setup: where the web application fails
SLA monitoring setup begins not with a dashboard, but with breaking down the request chain: name resolution, secure connection establishment, load balancer, application process, database connection pool, synchronous calls to external services.
Web application downtime rarely occurs all at once—first one link fails, while the remaining layers continue to respond. The task of the first stage is to understand which segment loses the user request and by what signal this is visible from the outside.
Then we split availability monitoring into external and internal. Probes from multiple points see only the outer layer: whether the page responds, whether the response fits the allotted time, whether the key scenario passes — login, search, checkout.
The application's own metrics know what is not visible from outside: connection pool saturation, growth of the processing queue, share of error responses on a specific route.
We stitch both loops along one request path and take collection into Prometheus, otherwise the liveness check reports success while the user scenario is already down.
False positives are not a trifle but the direct cost of setup.
A single probe from one point, a threshold without a time hold, lack of grouping of similar signals: the on-call engineer gets a flood of notifications, gets used to ignoring them, and a real failure drowns in the noise.
Delay consists of the polling period, the condition evaluation window, and the delivery route—seconds at each step turn into tens of minutes before the engineer even learns about the problem.
For operations, this means an incident can be neither proven nor analyzed: there is no separation of symptom and cause, no degradation history, and availability commitments cannot be confirmed.
We analyze how this is closed: from a map of failure points and agreed thresholds to escalation rules that do not wake anyone unnecessarily.
What control provides: maintenance windows and failure classes
Until control is labeled, any increase in response time looks the same: the on-call engineer gets a notification, goes to look at logs, and only then understands whether this is a planned restart or a database failure.
Labeling removes this uncertainty—planned work goes into separate maintenance windows, and failures get a class by consequences.
What changes in operations
Maintenance windows stop being an oral agreement.
Scheduled work and migrations are published as intervals with explicit exclusion from availability calculation—the schedule does not dip because of planned actions, and the fact of work is visible to both the customer and the on-call engineer.
Failure classes are separated by trigger thresholds.
Degradation, partial failure, and full unavailability get different conditions and different notification channels: an increase in response time raises the on-call engineer according to the schedule, while unavailability of the external loop—immediately and with escalation.
Window boundaries and the classes themselves are taken from service management practice (ITIL), so they do not have to be invented anew.
How we measure the effect
The main metric is recovery time. Before labeling, it was counted "from call to call" and included diagnostics: a typical incident took 40-60 minutes.
After, the starting point shifts to the first threshold trigger, and the route to the responsible person is known in advance—8-15 minutes for most scenarios.
The second metric is the share of triggers that led to real actions. If the on-call engineer opens a notification and does nothing, the threshold is set incorrectly. We keep this share above 80% and review the rules after every analysis.
Control formats: external service, self-hosted service, hybrid
The control format is chosen by two questions: whether you need to see the service from the inside and whether it is acceptable for the monitoring system to go down together with the server.
An external service checks what the user sees, but it does not know about the state of processes, queues, and free disk space. Self-hosted monitoring gives a complete picture, but it lives on the same environment as the application.
| Option | When it fits | Limitations |
|---|---|---|
| External monitoring service | Need independent availability checks from different regions | Does not see processes, queues, and server resources |
| Self-hosted monitoring | Own environment, full control over data and rules | Server failure also takes down monitoring—no one left to check |
| Hybrid monitoring | Critical service where both pictures are needed at once | Harder to set up, requires an agreed alerting model |
In practice, pure options are rare. External checks catch unavailability and an expired certificate, but they do not show that the service responds slowly because of an overflowing queue.
An agent on the server sees the internal state, but if the hardware fails it is silent—and without an independent check the failure remains unnoticed.
When the hybrid format is justified
Hybrid monitoring is needed where downtime costs more than setup: an agent on the server collects process and resource metrics, external checks independently record availability, response codes, and response time from different points.
There is one condition—agree in advance which system considers an incident open, otherwise the on-call engineer gets two notifications about one failure and over time stops reacting to both.
Implementation stages: from inventory to runbook handover
Inventory of services is not a formality: while there is no list of components and their owners, any metric hangs in the air. That is why we start with an audit—we compare what actually works with the documentation and record discrepancies.
Then we build a dependency map: without it, it is unclear which node failure should be considered an incident and which is background noise.
- Audit of infrastructure and applications: list of services, owners, external integrations, operating modes.
- Dependency map: "client — service — database — queue" relationships, critical paths, failure points.
- Implementation plan: which indicators we collect first, trigger thresholds, notification windows, rollout order across environments.
- Setup of collection and notifications: availability, response time, and response correctness checks, routing notifications to on-call engineers.
- Testing on real failures: simulate a node failure, response delay, database unavailability—fix false positives and thresholds.
- Runbook handover: description of metrics, instructions for on-call engineers, escalation and incident analysis procedure.
Roles and areas of responsibility at each step
We conduct the audit and dependency map together with your team—without knowledge of business logic, some critical paths are not visible. We agree thresholds and escalation rules with those who will be on call: a runbook written "from the outside" is not followed.
Pre-handover check
We run a control scenario: stop the service, introduce a delay, disconnect a dependency—and verify that the notification reached the right on-call engineer within the expected time.
We consider runbooks accepted when the on-call engineer independently reproduces incident analysis according to the instructions.
What remains with you: configurations, diagrams, and runbooks
All settings after implementation remain with you: monitoring configuration, alerting scheme, response runbook, and working documentation. The package is assembled so that support can be handled by your team or another contractor—without reconstructing the logic over calls.
- Monitoring configuration: list of checks—availability, response time, key user scenarios; trigger thresholds and false-positive suppression windows.
- Alerting scheme: routes by criticality levels, delivery channels, on-call groups, escalation rules, and waiting time at each step.
- Response runbook: incident levels, target response and recovery time, on-call engineer actions, responsibility handover points.
- Instructions for on-call engineers: initial checks, typical failure causes, message templates for internal and external recipients.
- Infrastructure diagram: services, dependencies between them, external points of presence—marked where checks are located.
- Access and change procedure: observer roles, change log, versioning of configurations in your repository.
What the documentation package looks like
The package has three layers: technical (configurations and diagrams), operational (runbook, instructions, status panels), and handover (access map, responsible contacts, change procedure).
Everything is stored in your repository and your cloud, in text formats readable without our tool. During handover, we go through the points with the on-call engineer: how to add a check, how to disable a noisy notification, how to roll back a change.
Case: eliminated false positives and halved downtime
We analyze an implementation for a high-load service: a catalog with order placement and payment, peak occurs in the evening hours, on-call is 24/7.
Before the work, monitoring rested on one threshold and did not distinguish "the page responds" from "the order is actually placed."
What it was before
A check once a minute hit the homepage: any spike in response time generated a stream of notifications—about 120 per week, of which only a few were real. The on-call engineer got used to the noise and reviewed signals selectively.
A real payment failure was discovered after 40 minutes from customer complaints. Service availability stayed at 99.4%, downtime—about 4 hours per month, recovery time—up to 50 minutes: diagnostics started from scratch each time.
What it became after
Monitoring was split into levels: synthetic scenarios of the full order path, application metrics, and business event counters—the number of successful payments per minute.
A signal goes to the on-call engineer only when at least two levels confirm, and during scheduled work and release rollout a suppression window opens. Notifications and escalations are consolidated in PagerDuty, so the on-call engineer sees one incident instead of ten.
False positives dropped to 15 per week, detection time—from 40 to 4 minutes. Recovery time was halved, to 20 minutes: by the time of analysis there are already charts and request history for the problematic path. Service availability grew to 99.95%, downtime—from 4 hours to 40 minutes per month.
How does availability monitoring differ from service level control?
Availability monitoring answers one question: does the service respond to a request. This is a binary signal—response code and response time at a given address.
The loop catches a complete outage but does not notice that the service is running slowly or returns errors only on part of requests.
Service level control is broader: it describes how the service serves the user. Share of successful requests, 95th percentile response time, failure rate on critical scenarios.
Formally the address is "available," but if order checkout takes 8 seconds, the service level is breached, and the availability check stays silent.
An external check runs from an external node and shows exactly what the user sees. Synthetic checks run the scenario end-to-end—login, search, payment—and give an early signal of degradation, but they do not replace data on real traffic.
| Option | What it shows | Limitations |
|---|---|---|
| External check | Address availability, response time | Does not see processes inside the service |
| Synthetic checks | Completion of the key scenario | Fixed path, does not catch rare branches |
| Application metrics | Queues, errors, processing time | Requires code instrumentation |
| Service level control | Share of successful requests, target values | Requires agreed targets and path labeling |
Availability is the foundation: without it, the remaining metrics do not count. Service level is the add-on that translates technical signals into target values: how many nines we maintain, what share of requests fits the response time, at what deviation an alert is raised.
Without the second loop, the team learns about the problem from users, not from the system. That is why we set up both levels and link them into one picture with a response runbook.
Technical doubts: what is clarified before the start
Before the start, engineering teams ask similar questions—not about the importance of control, but about how it will affect their work: how many false alarms there will be, what to do during peak season, where our responsibility ends. Below are answers to those most often raised when analyzing the task.
How do you filter out false positives?
An alert is raised only after several consecutive failed checks from different observation points—a single latency spike or a brief communication outage does not wake the on-call engineer.
We configure thresholds separately for synthetic checks of user scenarios and for application metrics: their nature is different, and a common threshold produces noise.
What happens under peak loads?
A static threshold during peak season or during a campaign gives either an avalanche of alerts or misses real degradation.
That is why some checks use a dynamic threshold—comparison with the same hour and day of the week—and for sale periods we create a separate profile with an extended set of metrics.
Where is the boundary of responsibility?
We are responsible for metric collection, trigger rules, alert delivery, and incident analysis for the application.
On the customer side remain access to infrastructure, code changes, and approval of the escalation runbook: to whom and where the signal goes, who decides on rollback. For external services, we agree in advance who communicates with their support.
What does the implementation phasing look like?
We start with a pilot on two or three critical scenarios—login to the customer account, end-to-end order checkout. We observe the reaction on real traffic, adjust thresholds, and only then roll out to the remaining services. This way extra alarms are filtered out before monitoring starts affecting on-call schedules.
Let's discuss the task: where we start and what we need from you
To come back with a timeline estimate and work plan, not generalities, we need source data and access. The first analysis is one conversation: we look at how the application is structured now, what failures have already occurred, and what counts as unavailability for you.
Then a short pilot service on the agreed environment: we set up checks, collect statistics, and compare it with real incidents. This shows which thresholds produce false positives and which miss degradation.
Based on the pilot results, we deploy monitoring across the entire environment and perform monitoring handover with documentation and on-call rules.
What we need from you at the start:
- critical scenarios—login, payment, search, order checkout: what we consider unavailability;
- application diagram and external dependencies: database, queues, payment and partner integrations;
- read access to logs and current monitoring systems, web server and load balancer configurations;
- incident history: what broke, how it was detected, how long recovery took;
- maintenance windows and acceptable response and recovery time for each scenario;
- on-call contact and escalation rules—whom and how to notify about a trigger;
- requirements for metric storage and access policy to them.
Some data may be missing—we will collect it together during the analysis. Describe the task and send what you have: we will come back with a timeline estimate, work plan, and pilot service proposal.
