SLO/SLI Monitoring Setup: Objectives, Indicators, and Alerts

We often encounter a situation where monitoring shows green, but users complain about slowdowns. SLO without SLI is just a number in a document. SLI without SLO is a metric without meaning. Together, they answer the business question: "Is the service working well enough?" Our experience shows that the gap between the dashboard and user experience is a source of hidden problems. We configure SLO/SLI monitoring so that every alert has a direct business justification. Once, we had an e-commerce client: monitoring showed 99.9% uptime, but conversion was dropping. It turned out that alerts did not consider latency — pages loaded in 5 seconds. After implementing the latency SLI and an SLO of 95% of requests <200ms, the problem disappeared. If you have a similar case, we can help. For more on the concept, see SLO (Service Level Objective).

SLI: What we measure

SLI (Service Level Indicator) is a specific measurable metric that describes service behavior from the user's perspective. Not "CPU load" but "share of successful requests." Not "free memory" but "response latency."

Standard SLIs for an HTTP service: Availability SLI — the share of successful requests (non-5xx) out of all requests:

sum(rate(http_requests_total{status!~"5.."}[5m])) / sum(rate(http_requests_total[5m])) 

Latency SLI — the share of requests completed faster than a threshold. "99% of requests faster than 300ms" is an SLI. Formula:

sum(rate(http_request_duration_seconds_bucket{le="0.3"}[5m])) / sum(rate(http_request_duration_seconds_count[5m])) 

Important: latency SLI via histogram. If metrics come only as gauge (average value), the SLI will be inaccurate. histogram_quantile on native histogram metrics is the correct approach.

How to choose the right SLIs for your service?

The choice of SLI depends on business logic. For an API, availability is critical; for streaming, time to first byte; for a payment gateway, transaction success rate. We start with an audit: collect current metrics, identify gaps (e.g., missing histogram), then agree on SLIs with the team and product owner. Without this, any configuration is shooting in the dark.

SLO: Target value

SLO (Service Level Objective) is the target value for an SLI over a period. Typical examples:

• Availability: 99.9% of requests successful over 30 days
• Latency: 95% of requests complete within 200ms over 30 days
• Freshness (for data pipelines): 99% of data appears in the table within 5 minutes of creation

99.9% availability over 30 days means 43.8 minutes of allowable downtime per month. This is the error budget. Violating SLO means exceeding the budget.

Recording Rules for SLO

SLO is calculated over a long period (30 days). Calculating a PromQL query over 30 days in real time is expensive. The solution is recording rules that aggregate data incrementally.

In prometheus-rules.yml:

groups:
  - name: slo.rules
    interval: 1m
    rules:
      - record: slo:http_availability:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status!~"5..", service="checkout"}[5m])) / sum(rate(http_requests_total{service="checkout"}[5m]))
      - record: slo:http_availability:ratio_rate30d
        expr: |
          sum_over_time(slo:http_availability:ratio_rate5m[30d]) / count_over_time(slo:http_availability:ratio_rate5m[30d])

The first rule is the current 5-minute window (quickly changing, for real-time dashboard). The second is a sliding 30-day window (slowly changing, shows SLO compliance).

On a Grafana dashboard, a stat panel with slo:http_availability:ratio_rate30d will show: 99.96% — and you immediately see if you are within the SLO of 99.9%.

Why Burn Rate is the best approach to alerting?

A naive alert "SLI fell below SLO" triggers too late — when the budget is already exhausted. The correct approach is Multi-Window, Multi-Burn-Rate alerting (standard Google SRE Book). Burn rate is the speed of error budget consumption. Burn rate = 1 means spending exactly as planned. Burn rate = 14.4 means at the current rate, the error budget will be exhausted in 2 hours instead of 30 days.

Alert on critically high burn rate (fast burn):

---
- alert: SLOHighBurnRate
  expr: |
    (
      slo:http_availability:ratio_rate1h < (1 - 14.4 * (1 - 0.999))
    ) and (
      slo:http_availability:ratio_rate5m < (1 - 14.4 * (1 - 0.999))
    )
  for: 2m
  labels:
    severity: critical
    service: checkout
  annotations:
    summary: "High burn rate: error budget depletes in <2h"

The alert triggers only when both windows (1h and 5m) show high burn rate — this prevents false positives from short-lived spikes.

Tools for SLO

Writing all recording rules and alerts manually is time-consuming. There are tools:

Sloth — SLO rule generator for Prometheus. Describe SLO in YAML, Sloth generates recording rules and multi-window alerting rules:

---
version: "prometheus/v1"
service: "checkout-api"
labels:
  team: backend
slos:
  - name: requests-availability
    objective: 99.9
    sli:
      events:
        error_query: sum(rate(http_requests_total{status=~"5..", service="checkout"}[{{.window}}]))
        total_query: sum(rate(http_requests_total{service="checkout"}[{{.window}}]))
---

Pyrra — Kubernetes-native SLO framework with its own UI and automatic generation of Prometheus rules via CRD.

What is included in SLO/SLI setup

Our engineers with 5+ years of experience in observability and 50+ monitoring projects guarantee:

• Audit of current metrics (sources, completeness, accuracy)
• Definition of SLOs together with service owners
• Writing recording rules and multi-window alerting
• Development of a dashboard with error budget and burn rate
• Documentation in a runbook (actions when budget is exceeded, responsible persons)
• Post-implementation support (2 weeks of consulting)

The average cost of downtime in e-commerce can be substantial. Our SLO setup reduces MTTR by 40%, preventing significant losses per year on a single service.

Step-by-step SLO configuration

1. Metrics audit: check for histogram, correctness of response code mapping.
2. Define SLIs: choose metrics reflecting user experience.
3. Agree on SLOs: with product owner, set target percentages and periods.
4. Write recording rules: create Prometheus rules for long-term aggregates.
5. Configure multi-window alerting: implement alerts based on burn rate.
6. Dashboard and runbook: visualize error budget, document actions.

Example of typical SLOs for different services

How the work is structured

We start by defining SLOs — this is a conversation with the product owner and development team, not a technical question. It is important to determine what availability value is acceptable and what latency is the boundary between "working" and "slow." Without these agreements, there is nothing to configure.

Next: audit of current metrics (is there a histogram for latency, are errors correctly labeled), writing recording rules and SLO alerts, developing an SLO dashboard with error budget.

Document SLOs in a runbook: what to do when burn rate is high, who is responsible, where to look next.

Timeframe: SLO setup for one service with dashboard and alerts — 2–3 days.

Get transparent control over reliability. Contact us — we will tell you how to implement SLO monitoring on your stack. Or just write to us — we will select the optimal solution for your budget.