Configuring RED Metrics (Rate, Errors, Duration) for Microservices

Note: when you have 50+ microservices in production and monitoring for each is cobbled together hastily — incidents are detected late. We've faced situations where 5xx were growing for half an hour, but the dashboard showed p99 latency under 200 ms — simply because errors didn't make it into the metrics. RED metrics (Rate, Errors, Duration) solve this: they provide a unified template for any service and clearly show what went wrong. They are 2 times easier to set up than the full set of Golden Signals and cover 90% of monitoring needs for client experience. Setting up RED metrics is standardizing microservices monitoring, which reduces MTTR by an average of 40%.

Why RED Metrics Are Critical for Microservices

RED is a compact version of Golden Signals focused on what the client sees from the service. Unlike the USE approach, RED does not require customization per service. For a fleet of 20–100 services, this is the only way to keep monitoring under control without manual effort. In practice, we implement RED on projects with Istio service mesh — metrics are collected automatically without code modification. Time savings on instrumentation — up to 80%.

Rate: Requests Per Second

Rate — the current incoming traffic of the service. In PromQL:

sum by (service, method) ( rate(http_requests_total{job="microservices"}[1m]) ) 

The label method (GET/POST/PUT) grouping helps in debugging: sometimes a specific HTTP method starts generating load — mass PUTs from a runaway process, atypical number of POSTs from repeated retries. For gRPC services, use grpc_server_handled_total with labels grpc_code and grpc_method.

Errors: Breakdown by Type

In microservice architecture, errors are divided into two classes:

• Client errors (4xx) — incorrect request from the client. A rise in 4xx in a downstream service while 4xx are stable in upstream — someone changed a contract without notification. This is a common scenario with independent deployment of services.
• Server errors (5xx) — problem inside the service. These are the ones that enter the error budget.

sum by (service) ( rate(http_requests_total{status_code=~"5.."}[5m]) ) / sum by (service) ( rate(http_requests_total[5m]) ) 

An additional useful metric: http_requests_total by upstream_service label. If errors in payment-service grow synchronously with errors in database-service — it's not two independent problems, it's one problem with a cascading effect.

Duration: Not Only p99

Duration — request processing time from receipt to sending a response. Standard set for a dashboard: p50, p90, p99. But for microservices, another slice is important — duration breakdown by downstream dependencies. A service may respond slowly not because of itself, but due to a slow response from a database or external API. This requires distributed tracing (Jaeger, Tempo) in addition to metrics. RED metrics show the symptom, traces show the cause. Without traces, you can use indirect metrics: if the application exports duration for outgoing requests, use histogram_quantile(0.99, sum by (le, target_service) (rate(outgoing_request_duration_seconds_bucket[5m]))).

How to Automate Collecting RED Metrics Without Code Modification?

Istio and Linkerd automatically collect RED metrics for each service without code changes. istio_requests_total and istio_request_duration_milliseconds are available immediately after installing the service mesh. Example PromQL for Istio:

sum by (destination_service_name) (
  rate(istio_requests_total{ reporter="destination", response_code=~"5.." }[5m])
) / sum by (destination_service_name) (
  rate(istio_requests_total{ reporter="destination" }[5m])
)

reporter="destination" — metrics from the receiver side of the request. This excludes network jitter and shows the real service performance.

Standardized Dashboard for the Fleet

For 50+ microservices, manual dashboard development for each is unrealistic. The correct approach: one template dashboard with a $service variable that works for any service. Variable $service from label values: label_values(http_requests_total{job="microservices"}, service). Three main panels on the dashboard:

1. Rate — time series with rate(http_requests_total{service="$service"}[1m]), breakdown by status_code.
2. Errors — time series error rate in percent + stat panel with current value.
3. Duration — multi-line time series with p50/p90/p99.

One dashboard — any of the 50 services via dropdown. This scales.

Alerts

Standard set of RED alerts applicable to any service:

---
- alert: HighErrorRate
  expr: |
    sum by (service) (rate(http_requests_total{status_code=~"5.."}[5m])) / sum by (service) (rate(http_requests_total[5m])) > 0.05
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Service {{ $labels.service }}: error rate > 5%"
- alert: HighLatencyP99
  expr: |
    histogram_quantile(0.99, sum by (le, service) ( rate(http_request_duration_seconds_bucket[5m]) ) ) > 2
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Service {{ $labels.service }}: p99 latency > 2s"
---

Thresholds need to be adapted for the specific service — checkout and background-worker have different expected latencies. Our experience shows: for correct alert configuration, it's enough to analyze SLO data for a month. Guaranteed you won't miss critical incidents. Alternatively, if your stack does not use service mesh, you can export RED metrics from the application via Prometheus client, but this requires 5 times more manual work. RED metrics are collected 3 times faster than custom metrics for each service.

How the Work Is Done

1. Instrumentation audit: what metrics each service exports, whether there is a histogram for latency, whether errors are correctly tagged. If using service mesh — check Istio/Linkerd configuration.
2. Design: develop a unified dashboard with variables, a set of recording rules to speed up queries, alert templates.
3. Implementation: configure metric export, create the dashboard and alerts, test on several services.
4. Testing: verify that metrics display correctly, alerts trigger when thresholds are exceeded, and there is no alert noise.
5. Deployment: roll out configurations to production, document the process for adding a new service to monitoring.

Timeline: standardized RED template for a fleet of microservices — 2-3 days. Including recording rules, alerts, and documentation. Cost is calculated individually — contact us to estimate your project.

What Is Included in Turnkey Work

• Full audit of current monitoring and instrumentation.
• Ready-made dashboards in Grafana for any number of services.
• Set of alerts and recording rules adapted to your SLOs.
• Documentation for team onboarding.
• Support during first rollout (up to 2 weeks).

Our experience — 10+ years in observability and microservices monitoring, 50+ RED metric implementations in production. Book a consultation to discuss details. Get a ready-made solution in 2-3 days and see RED metrics in action.