Setting Up Error Budget Monitoring in Prometheus and Grafana

Error budget monitoring: where feedback breaks

Errors in a service are always present — the question is how many are acceptable. If the limit is not measured, service reliability is discussed based on feelings: one engineer considers one failure per thousand requests normal, another — isolated outages. A debate about release quality turns into a debate about tastes.

The service error limit closes exactly this gap.

It sets the share of requests that may be lost or corrupted over a period and turns it into a remaining balance: while the budget is not spent, releases proceed at high speed; when the remaining balance is zero, changes are stopped and the causes of failures are investigated. The decision stops depending on the negotiating weight of the participants.

This breaks down with fragmented metrics. Error rate lives in one dashboard, latency — in a second, unavailability — in a third, and no one keeps the overall count.

Error budget monitoring implies consolidating these sources into a single value — otherwise the budget looks healthy, while users leave slow pages where formally there were no failures.

The second failure mode is threshold alerts set by old intuition. A threshold that triggers on every fluctuation devalues on-call duty: the engineer gets used to closing the notification without looking, and a real incident gets lost in the noise. An alert should rely on the budget burn rate, not on a static number.

We'll discuss how to build such feedback on Prometheus and Grafana: from agreeing on the target level to rules that stop a release on their own.

What a managed error limit gives a service

The error limit is derived from the target availability level: the service is allowed to spend a certain share of failed requests over the observation window, and this remaining balance becomes a shared counter for development and operations.

While the reserve is positive, releases proceed at their usual rhythm; when it dwindles, changes begin to compete for the right to roll out.

Alerts are tied not to individual spikes but to the rate at which the remaining balance is depleted, so a short failure does not wake the on-call engineer, while sustained degradation signals before the service drops below the target level.

The share of false alerts falls, and incident response time stops being a feeling and becomes a quantity: you can see how much time passes from the first deviation to intervention and at which step the most is lost.

The release queue also follows the rule: when the limit is exhausted, reliability is restored first and only then is functionality rolled out.

This removes a class of failures that previously arose from the accumulation of small unnoticed degradations — each individually below the threshold, but together they push the service beyond the target level.

The result is measured in shares: a decrease in the share of failed requests, fewer manual investigations, and fewer alerts that are closed without action.

The calculation of the remaining balance and its burn rate lives in Prometheus and is displayed on a shared board — engineers see the reserve in hours and discuss decisions, not whether "things are bad right now."

Calculation modes: window, rate, and strictness

The calculation mode determines how quickly the system notices degradation and how the remaining budget is interpreted.

In Prometheus, the budget is calculated by a recording rule over counters of successful and failed responses, and the Grafana panel merely displays the already calculated value — therefore the strictness of the signal is set here, not in the dashboard.

In practice, modes are combined: the sliding window gives the on-call engineer an operational signal, the calendar period provides the basis for reviewing results, and the aggregate limit holds overall spending.

A discrepancy between request-based and time-based budgets is a good indicator: the error rate is normal, but users complain about unavailability, which means the problem is in timeouts and retries, not in response codes.

Implementation: from metric audit to monitoring policy

The work proceeds in six stages and begins not with graphs but with an inventory of what is already being collected.

The order is fundamental: the error budget is calculated from the same data that triggers notifications; otherwise the on-call engineer sees a discrepancy between the panel and the alert and stops trusting both numbers.

1. Metric audit. We analyze the sources: which metrics already exist, where the gaps are, which duplicate each other, where cardinality grows faster than usefulness. The output is a list of working SLIs for each service and a list of what will have to be added.
2. Calculation scheme. We fix the window (sliding 30 days), the target availability level, the definition of a "bad" event, and the aggregation rules by service. Aggregates are calculated in advance, not on every panel request — otherwise a dashboard with a dozen services starts to lag and lies about timing.
3. Rules and thresholds. We set up multi-level notifications: a warning when a quarter of the budget is burned, an alert when half is burned, a release stop when it is exhausted. A threshold on burn rate catches a rapid failure before the daily graph shows it.
4. Validation on historical data. We run the scheme against past incidents: would the signals have caught them, how many were false, did the thresholds match actual degradation. We calibrate thresholds based on facts, not intuition.
5. On-call policy. We describe the actions for each level: whom to wake, what to look at first, when to freeze rollout. We coordinate the policy with those who take on-call shifts — a document written without them does not work.
6. Handover and training. We walk through live scenarios with the team and agree on reviewing every trigger. After that the scheme lives on its own: rules and thresholds are revised based on quarterly results.

What remains in the repository after handover

After handover, you are left with a working reliability control loop, not a set of fragmented settings. All artifacts are in your repository and your environment: the team can open any of them, see where each number comes from, and change the system's behavior without us.

• Rule configurations — source files for metric recording and computed expressions for the error budget, organized by service and environment, with versioning and meaningful names.
• Calculation scheme — a document that records which events count as failures, how they are normalized to the total request volume, and over which window they are averaged. When investigating an incident, the logic does not have to be guessed.
• Notification policies — thresholds, grouping, and routes: what goes to the on-call engineer immediately, what waits for confirmation, what only goes into the report.
• On-call instructions — a step-by-step walkthrough: how to read the budget burn chart, what decisions to make at different spend rates, where to escalate, and what to record afterward.
• Policy for revising levels — frequency and criteria: when the reliability target is tightened, when it is relaxed, who makes the decision, and how it is recorded in history.
• Dashboard templates — panels for an individual service and a summary across all, with the same calculation logic, so a new team does not draw its own version from scratch.
• Rule checks — a set of tests that catch broken expressions and empty queries before rollout, not at the moment of an incident.

Case study: payment service, alerts, and response time

A payment service with a continuous stream of transactions: several on-call engineers per shift, all alerts fall into one shared channel, and no one calculated the error budget.

Alert rules were configured with thresholds set "by eye" and were not revised — there were more signals, but less trust in them.

What it was like

More than half of the triggers were closed as false: a short latency spike, isolated errors on one instance, noise from background jobs. The on-call shift went into manual investigations — 5-6 per shift, of which only a few were useful.

Detection time was measured in tens of minutes: a real outage drowned in a stream of false ones, and some outages surfaced only after customers reached out.

What was changed

We tied alerts to the error budget: for the payment API we fixed availability and response time targets, and the signal is now calculated by the budget spend rate within the window, not by a single spike.

We split notifications into two classes — those that wake the on-call engineer and those that are handled during working hours. Metric collection and alert rules were moved to Prometheus and Grafana, and every nighttime signal is reviewed during the on-call shift.

A separate effect is shift predictability: the on-call engineer knows that a nighttime signal means real budget spend, not noise. Outages began to be found before customers reached out, and reviewing a trigger became a short procedure instead of an investigation.

How does SLO differ from SLA and what should be configured first?

The difference is simple, but it is confused more often than anything else. SLA is an obligation to the customer: fixed in a contract, with a measurable formulation and specified consequences for violation. SLO is an internal target level by which the team makes decisions about releases and priorities.

The setup order is the same: internal targets first, then contractual ones. SLOs are set stricter than SLAs so that a reserve remains between them — deviations are absorbed within the team and do not turn into a breach of obligations.

The reverse order gives a promise without data: it is unclear what reserve actually exists and whether the infrastructure can sustain it.

Error budget is the quantitative expression of an SLO: how much unavailability or share of failed requests may be spent over the window. While the budget is intact, releases proceed at their usual pace; when the budget approaches the boundary, priority shifts to reliability.

In Prometheus this is the rules for counting successful requests and the remaining budget; in Grafana it is the spend panel and alerts on burn rate, not on a single failure.

The consequences of violation are also different, and they must not be mixed. An SLO violation is an internal event: stopping releases, returning efforts to reliability, revising an unrealistic target. An SLA violation is external: obligations and everything secured behind them. Therefore you first measure yourself, and only then make promises.

Common technical doubts before implementation

Doubts before implementing an error budget usually concern not the tool, but how honestly the data reflects the service's reality. Below are the situations that occur most often and how we address them when setting up control in Prometheus and Grafana.

The metric is noisy: are spikes unrelated to real failures?

We suppress noise not with a threshold set "by eye" but at the level of the metric definition: we remove health checks, technical responses, and internal requests between services, and aggregate over a window that survives a single spike. Then budget spend reflects user losses, not background activity.

Several teams use the service — whose budget is it?

We tie the budget to the service owner, but mark spending by consumer: each team sees its own share of burn. The dispute over "who drained availability" is resolved with data, not in a general meeting.

Thresholds were set formally — will control turn into reporting?

We take levels from the service's actual behavior over the past period, not from what is desired. We revise them on a schedule — after major architecture changes or when the budget is consistently not spent at all.

Let's discuss your task and come back with a work plan

To make the estimate substantive rather than "by eye," we need a short review of your infrastructure: how metrics are collected, where they are stored, who responds to triggers and how.

Describe the task in free form — we will return with a scope estimate and a work plan broken down by stages.

What we clarify on the call:

• metric sources and the current collection scheme — Prometheus, exporters, agents on nodes;
• list of services and their owners: for whom we calculate the error budget and by which SLIs;
• windows and calculation policy: sliding window, burn thresholds, what we consider unavailability;
• current alerts and on-call rotations — which triggers are noisy, which are missed;
• access and constraints: environments, closed networks, rights to change rules and dashboards;
• reporting requirements — summaries for management, frequency, history depth;
• handover format: repository with configuration, runbook, training for the on-call shift.

Based on the results, we fix the team composition for the task — who is responsible for recording rules, who for dashboards and alerts, who for knowledge transfer. We agree on the plan before the start so that you see intermediate results at each stage, not only the final report.

Tell us what hurts right now: noisy alerts, no reliability budget, or fragmented dashboards. That is enough for us to come back with a plan and an estimate.