Golden Signals SRE Monitoring: Latency, Traffic, Errors, Saturation

When the service goes down but monitoring shows all green — that's a problem. Standard CPU and memory monitoring doesn't reflect user experience. Without golden signals, you won't know latency has risen to 10 seconds until you check logs. The four signals — Latency, Traffic, Errors, Saturation — answer the question "where to dig?". Google described them in the SRE Book SRE Book, and this framework remains one of the most practical approaches to monitoring. It focuses on user-facing metrics, not infrastructure.

We configure golden signals turnkey: from inventory of current metrics to dashboards and alerts with documentation. Contact us for a preliminary assessment of your project.

Latency: why p99 is more important than average

Latency — request execution time. The main mistake is monitoring the average. The average hides the problem: if 1% of requests take 30 seconds and the other 99% take 10ms, the average will look fine, but hundreds of users see a freeze every minute. p99 latency is 10 times more informative than the average for diagnosing performance issues.

Correct approach: p50, p95, p99 via histogram. Separate latency for successful and erroneous requests. A slow error is worse than a fast one — the user waited and still got a failure.

# P99 latency for successful requests
histogram_quantile(0.99, sum by (le, service) (
  rate(http_request_duration_seconds_bucket{service="$service", status_code!~"5.."}[5m])
))

On the dashboard — three lines: p50 (median), p95 (most users), p99 (unluckiest). Deviation of p99 from p50 signals an unstable tail of the distribution.

Traffic: baseline and anomalies

Traffic — load on the service. For HTTP services — requests per second. For queues — messages per second. For databases — transactions per second.

sum by (service) ( rate(http_requests_total{service="$service"}[1m]) ) 

Traffic provides context for other signals. High latency with low traffic — internal problem (memory leak, deadlock). High latency with anomalously high traffic — overload or DDoS.

Useful panel: traffic with deployment annotations. Sharp drop in traffic — possibly the service stopped accepting requests. Sharp increase — abnormal load.

Errors: classification and alerts

Errors — fraction of requests that ended with an error. Important to distinguish: HTTP 5xx (server errors), HTTP 4xx by code (400 = bad request, 404 = not found — different things), timeouts (may return 200 but be broken).

# Error rate 5xx
sum(rate(http_requests_total{service="$service", status_code=~"5.."}[5m])) / sum(rate(http_requests_total{service="$service"}[5m])) * 100

The alert on errors should consider the baseline. 0.1% errors for checkout is critical. 2% for search may be normal if search returns "not found".

Saturation: measurement for different components

Saturation — how close the service is to resource ceiling. This is a predictive signal: the service still works normally, but resources are low, and at the next spike degradation is inevitable.

For different components, saturation is measured differently:

# CPU saturation for pod
rate(container_cpu_usage_seconds_total{pod=~"$service.*"}[5m]) / on(pod) kube_pod_container_resource_limits{resource="cpu"}
# Saturation connection pool (for apps with DB connection pool)
db_pool_active_connections{service="$service"} / db_pool_max_connections{service="$service"}
# Queue saturation
rabbitmq_queue_messages{queue="$queue"} / rabbitmq_queue_consumer_capacity{queue="$queue"}

Saturation > 80% — yellow alert. > 95% — red, scale before degradation.

Dashboard of four signals

One dashboard, four rows, each for one signal. In each row — several panels with different aspects.

Variable $service at the top — dashboard works for any service from discovery.

How to implement golden signals in 3 steps

1. Inventory current metrics and instrumentation. Check if there are latency histograms and error metrics.
2. Configure recording rules for histogram, error rate, saturation.
3. Develop dashboard with four rows and a set of alerts with yellow/red thresholds.

Get a consultation on monitoring setup — we help at every stage.

Comparison: average vs percentiles

Percentiles give 100 times more accurate picture for latency diagnosis.

What is included in the work

• Inventory of current metrics and instrumentation
• Configuration of recording rules for histogram, error rate, saturation
• Development of dashboard with four rows and $service variable
• Configuration of alerts with yellow/red thresholds
• Documentation for the team: what each signal means, how to respond
• Handover and training (optional)

How the work is structured

We start with inventory: which of the four signals are already measured. Latency via histogram — is it present? Or only gauge with average? If gauge — we fix application instrumentation, without histogram percentiles cannot be calculated.

Errors — are 5xx logged as metrics? Traffic — are there Prometheus request metrics, or only from nginx/loadbalancer logs.

After inventory: development of recording rules, dashboard, set of alerts for each signal with reasonable thresholds. The result — documentation with signal descriptions and runbook for alerts.

Our engineers have extensive SRE experience, Prometheus and Kubernetes certifications. We guarantee compliance with MTTR SLA.

Timelines and how to order

Dashboard of four signals for one service — 1-2 days. Standardized template for fleet of microservices — 2-3 days. Cost is calculated individually.

We will assess your project. Contact us — we'll tell you how to improve monitoring.