SLO-based alerting: why the on-call engineer mutes notifications
Threshold alerts fire when an individual metric crosses a specified boundary — node load, queue length, error rate per minute.
Such a signal by itself does not say whether users are affected: a burst of retries in the queue is absorbed by retry attempts, and the service rides out a short rise in error rate unnoticed.
Over a day, the on-call engineer receives hundreds of notifications, of which only a few require action. They stop reading the channel — the important gets lost in the stream, while real service degradation develops slowly and does not sharply cross any threshold.
It is detected from user complaints, not monitoring: a missed incident stretches reaction time from minutes to hours.
SLO-based alerting changes the reference point: the trigger becomes not a metric threshold but error budget burn. The service wakes the on-call engineer only when current degradation threatens to exhaust the allowable share of failed requests over the observation window.
This works through burn rate across multiple windows at once. The short window catches a sharp spike and wakes the on-call engineer immediately; the long one catches sustained slow drift that threshold alerts miss.
Both signals point to one incident instead of dozens of notifications on related metrics, and the alert comes with context: which SLO is at risk, over what window, and how fast the budget is burning.
On-call noise drops severalfold, but the main thing is that the class of failures the team catches before users changes. Below we discuss how such a loop is configured and what it needs from the service.
What an error budget gives: less noise, faster reaction
An error budget turns service reliability from a matter of taste into a measurable quantity.
For the observation window, an allowable volume of failures is set, and the rule fires not on any deviation of a metric but on the burn rate of this reserve: the on-call engineer gets paged when, at the current rate, the budget will be exhausted before the end of the window.
Multi-window rules with a fast and slow window separate two different scenarios. The short window catches a sharp drop in minutes; the long one catches slow drift that is not visible in single measurements.
The threshold is tied to the question “will we exhaust the budget,” so static boundaries that for years produced false positives go away.
What changes measurably:
- the share of notifications that an engineer starts work on, instead of “looked and closed as known noise”;
- time to detection of degradation — from the first anomalous responses to a page to the on-call engineer;
- classes of failures that stop reaching the user: partial degradation becomes visible earlier than total failure.
Rules are built on symptoms — successful request share, latency, availability of key scenarios — not on causes such as load on an individual node. This reversal is described in Google SRE: the page comes for what the user noticed, not for what merely looks suspicious on a graph.
Comparison of approaches to degradation alerting
The choice of alerting mechanism determines not the number of notifications but whether the team notices degradation before users.
Static thresholds are cheap and predictable but know nothing about seasonality and gradual load growth; anomaly models catch the unfamiliar, but are noisy on every release and demand surge; reliability objectives with an error budget tie the signal to user experience but require a calculated budget and full telemetry coverage.
| Approach | When appropriate | Where it breaks |
|---|---|---|
| Static thresholds | Resources with a flat profile: free space, queue length, absolute number of errors | The threshold has to be revised with every traffic increase, and slow degradation does not cross the boundary at all |
| Anomaly alerts based on historical data | Seasonal load and failures that cannot be described in advance by a rule | A training period with degradation fixes the failure as the norm, and releases produce a stream of false positives |
| Multi-window policies on error budget | User-facing services with an explicit reliability objective | Need a calculated budget, labeling of fast and slow window, and metric coverage — otherwise notifications lose meaning |
| Combination of layers | Mature system where critical paths are separated from auxiliary ones | Requires revision of existing rules and an owner role responsible for reviewing objectives |
In practice, approaches do not compete but diverge by layers: multi-window policies cover critical user paths, static thresholds cover resources and background queues, anomalies remain an auxiliary layer for what is not yet described by an objective. Combining them gives fewer notifications with greater sensitivity.
Implementation: from service inventory to on-call schedule
The order of stages is not accidental: if you start with thresholds without understanding which services are critical and how they relate to each other, alerts will fire on everything — and the on-call engineer will quickly stop looking at them. We go from inventory to handover to the on-call team; at each step there is a verifiable result.
- Service inventory. List of services with owners, criticality, and what we consider an error from the user's point of view, not by an internal metric.
- Dependency map. We mark shared nodes: degradation of one of them produces an avalanche of alerts, and five teams wake up because of one cause.
- Objectives agreement. With the service owner, we fix the target availability level, calculation rule, and error budget on a sliding window.
- Policy configuration. A short burst of errors and sustained degradation are different signals, so fast and slow thresholds are calculated separately, with different notification speed.
- Escalation routes. We determine who receives the notification first, after how long the second line joins, and where incident history is kept.
- Training incident. We intentionally degrade the service and check the chain: the alert arrived, the on-call engineer found the cause per the instructions, escalation worked. We measure time to response.
- Handover to the on-call team. On-call regulations, troubleshooting instructions, false-positive review, and threshold tuning in the first weeks.
What we hand over: configurations, dashboards, procedures
We hand over not a set of rules in someone else's panel but configuration under version control: service descriptions, policies, and routes lie next to the code, changes go through review and are rolled back with one edit.
Such a set remains working without us — a new service is connected per instructions, thresholds are edited deliberately, not “someone once configured it.”
- Service descriptions — list of critical services with owners, dependencies, and target availability and latency levels from which error budgets are calculated.
- Alerting policies — which rules fire, at what budget burn rate, waiting and silencing windows, so that a single failure does not wake the on-call engineer.
- Escalation routes — who is first in the queue, after how many minutes the second line joins, and at what moment automation yields to a manual decision.
- On-call dashboards — remaining budget, current burn, firing alerts, and latest changes; open via one link from any device.
- Incident review procedure — order of recording, criteria for freezing releases when the budget is burned, and format for recording causes so that conclusions are not lost.
- Documentation and knowledge transfer — where configurations are stored, how to connect a new service, how to agree on a threshold change, and who is responsible for revision.
Separately, there is a procedure for periodic revision: once a quarter we check which rules never fired and which are noisy, and remove the excess. Without this, any set of alerts overgrows and loses the trust of on-call engineers.
Case: how SLO policies reduced on-call noise
On-call rested on static thresholds: a separate rule for each metric, values set at service launch and not revised as load grew. Per shift, 60–80 notifications arrived, and most described short spikes that users did not notice.
The on-call engineer learned to filter the stream by eye and, along with the noise, discarded signals of real degradation.
What changed
Thresholds for individual metrics were replaced by an error budget: a notification is raised not on a one-off deviation but when its burn rate is sustainably above normal — and confirmed in two windows at once, short and long.
The short one catches a sharp failure, the long one catches slow accumulation of errors that a single threshold missed. Reliability metrics were collected in Prometheus, rules were described as code and passed checks together with service changes.
Switching policies was phased. For several weeks the system ran in shadow mode: new notifications went to a separate channel and were compared with manual incident reviews.
Then services were migrated one by one, leaving previous thresholds as a safety net until the accuracy of the new rules was confirmed on live failures.
| Indicator | Before | After |
|---|---|---|
| Notifications per on-call shift | 60–80 | 8–12 |
| Share of false positives | about 70% | less than 10% |
| Degradation detection | 15–40 minutes | 2–5 minutes |
| Review of one notification | 20–30 minutes | 5–7 minutes |
Notification flow decreased without an increase in response time: severity is now set by the target reliability level, not by a subjectively chosen threshold.
How is an error budget different from a threshold alert?
A threshold alert monitors the instantaneous value of a metric: the share of failed requests exceeded a given percentage within a minute — it fired.
Error budget calculates differently: it accumulates the share of failed requests over the observation window and compares it with the allowable remainder.
The difference is fundamental — the threshold catches any spike, even if users did not notice it, while the budget reflects real reliability loss over the period.
The mechanics are simple. With an availability target of 99.9%, the error budget is 0.1% of all requests over the observation window, usually a sliding window of several weeks.
As long as the share of failed requests is within this quota, the service is normal and there is no reason to wake the on-call engineer. Once the quota is exhausted, the alert means further operation is at the expense of the reliability we promised.
Then budget burn comes into play — the rate at which the remainder is spent. It is calculated in two windows simultaneously: the short one shows a fast spike and wakes the on-call engineer, the long one shows slow degradation that is not visible on an instantaneous graph.
Thus a failure that consumed a noticeable part of the quota in an hour does not go unnoticed, and gradual deterioration over two weeks is recorded separately.
Practical effect: there are several times fewer alerts, and each remaining one truly requires intervention.
The logic is described in Site Reliability Engineering and has been proven on high-load services — we transfer it to the customer's infrastructure, selecting observation windows and burn thresholds for their traffic and on-call mode.
Technical concerns: windows, aggregation, false noise
Aggregation window, rare services, and blurred ownership break SLO-based alerting more often than the calculation math itself. Below are short answers to questions engineers ask before starting implementation.
How to choose an aggregation window?
One window is not enough: the short one catches fast degradation but is noisy on spikes, the long one is resistant to random failures but lags.
Therefore we take a pair of windows of different lengths and different error budget burn thresholds — the fast signal goes to the on-call engineer, the slow one creates a ticket for review. We select thresholds based on the service's historical data, not “by eye”.
What to do with rare services?
At low traffic, the error share jumps from a single failure, and percentage-based alerting gives false positives. For such services, we calculate the absolute number of failed operations or record the complete absence of successful events — that is a stable signal. We formulate the reliability objective for them separately, with a wider observation window.
Who owns the reliability objective?
The owner of the objective is the service team, not the infrastructure group: it changes code and is responsible for system behavior. We store the objective description and alerting rules next to the code in the repository, and changes go through review.
This way the threshold does not “drift” silently, and the edit history explains why it is exactly this way.
How to separate a real failure from noise?
We raise a notification on error budget exhaustion, not on every individual failure: single failures and restarts do not wake the on-call engineer. We group firings by service and customer path, and collapse duplicates.
If the signal arrives more often than the team can react, we revise the threshold; otherwise the on-call engineer gets used to ignoring notifications.
Let's discuss the task: where to start and what is needed
It is worth starting the conversation by discussing the task in concrete terms: which services hold revenue, what is considered normal for them, and where the boundary between “working” and “on fire” currently lies.
We need source data on the services and the current state of monitoring to come back with an estimate and work plan, not general promises.
What we expect from you in the first message:
- List of services with each one's role in the product and an indication of the owner on the customer side — from it we will understand which request processing paths even make sense to cover with target levels.
- Current availability and response time objectives, if they are already formulated, or quality complaints: “the page opens slowly”, “the channel disappears at night” — this is enough for the first loop.
- Where metrics and logs are currently collected, and whether there is a historical series for several months. Without history, one cannot calculate how often the objective was violated before implementation, and later there will be nothing to compare with.
- Existing dashboards, alert rules, and notification channels: messenger, email, on-call phone, incident tracking system.
- Read access to monitoring systems and to the infrastructure description — as a rule, they are granted for the duration of the review and can be revoked at any time.
- On-call and escalation procedure: who receives the alert at night and by what sign understands that it is time to wake the second engineer.
- Review of the latest incidents: what fired falsely, what was missed entirely, and how much time was spent confirming the problem manually.
After that we will return with an implementation timeline by stages, scope of work, and who needs to be involved on your side at each step. Describe the task — and we will give an estimate and plan.