Setting up USE metrics for infrastructure: Utilization, Saturation, Errors

USE metrics: where infrastructure loses stability

Nighttime traffic growth looks normal on graphs: CPU stays within the usual corridor, memory grows smoothly.

And then one node in the group stops responding — the disk request queue went up, the handler failed to return the connection, and the load balancer shifted the load to its neighbors.

If the node was at its limit, they go down next: a node failure under load is almost never isolated. Recovery takes hours, because you have to investigate after the fact, using indirect signs.

The cause is usually blind spots in monitoring. We collect CPU load, used memory, and service response time — but each sensor lives on its own.

From such graphs you cannot tell which resource is exhausted: resource saturation is visible not in the average value, but in the dynamics of queues and failures against growing demand.

Adding ten more separate sensors means getting ten more graphs, between which the engineer reconstructs the relationship manually, already during an incident.

USE method removes this manual work: for each resource — CPU, memory, disk, network, connection pool — three values are captured.

Utilization shows what share of the resource is busy, Saturation — how much work is queued and waiting to be released, Errors — how many operations failed. Only the combination of three signals identifies an infrastructure bottleneck before it becomes a failure.

Next we break down how this is handled in practice: which metrics we capture for each resource, where we set warning thresholds, and how we integrate them into on-call.

What implementation delivers: response speed and capacity headroom

USE metrics break down node state into three questions: how loaded the resource is, where the queue is accumulating, what is already returning errors.

We collect them with standard exporters and aggregate them in Prometheus — without custom agents that themselves become a point of failure. Then the effect of implementation is measured in measurable quantities: time, share, headroom.

Incident detection time

Saturation thresholds trigger before users notice degradation. The on-call engineer receives an alert about a growing queue or errors on a specific node, rather than a generic “service unavailable” signal — detection time drops from tens of minutes to single-digit minutes.

Service recovery time

The cause is localized to a resource and a specific metric, so diagnostics don’t start with reading logs on all nodes. Typical scenarios — degraded disk operations or connection exhaustion — are recognized from the graph, and recovery is reduced by a factor.

Failure class and downtime impact

Some incidents never occur: saturation is visible in advance, and adding a replica or a limit resolves the problem before failure. The share of prevented failures grows, and with it downtime and its business impact fall.

Node capacity headroom

Load graphs show how much resource remains before the threshold at peak load. Scaling is planned based on facts, not gut feeling: the amount of manual diagnostics stops growing, and the node does not keep idle capacity “just in case.”

Methods of collecting USE metrics: agent, exporter, polling

The collection method determines which signals reach storage at all and with what delay. Three basic schemes — an agent on the node, an external exporter, and scheduled polling — differ in connection point, permissions, and behavior when communication breaks.

The mode is fixed once: changing it on a running environment is more disruptive than immediately separating two independent streams.

In practice, the schemes are combined: the agent on the node covers the basic triad, the exporter adds application-layer metrics, and polling covers network and peripheral devices.

When the environment is isolated, collection is separated from delivery — a channel break then does not become a loss of history, and accumulated data catches up to storage after communication is restored. Next we break down how these streams are consolidated into common rules and alerts.

Implementation stages: from audit to handover of policies

The order of work matters more than speed here: if you start with thresholds, half of the alerts will be false, and trust in them will disappear before any benefit appears. So we go from inventory to policies, recording at each stage the result that the next stage relies on.

1. Coverage audit. We inventory nodes, services, and queues; for each resource we note which of the three signals — utilization, saturation, errors — are already visible, and where the blind spots are: connection pools, disk queues, network interfaces.
2. Data collection scheme. We define collection points, polling frequency, label set, and retention period; we limit cardinality so storage volume does not grow along with the number of containers.
3. Threshold values. For utilization, we take the threshold from the sustained level over a window; for saturation, from queue length and wait time; for errors, from the share per interval. A threshold without an averaging window either misses degradation or is noisy.
4. Alert validation. We run a drill failure: load the resource and make sure the signal arrives once, with clear context, and clears on recovery, rather than as an avalanche across all three metrics of one node.
5. Handover to operations. We hand over monitoring dashboards, alert rules as configuration, and the on-call policy: what each signal means and in what order to act when it fires.
6. Post-launch review. A few weeks later, we review false and missed triggers and adjust thresholds to the real load profile — without this, the configuration quickly diverges from reality.

What we hand over: schemes, rules, documentation

We hand over not a set of graphs but a working monitoring loop: an engineer who sees the system for the first time brings up collection from configurations, finds the bottleneck from the data flow diagram, and receives an alert before degradation reaches the user. Everything is in the repository — changes go through review, and rollback takes one revision.

• Collection configurations: list of objects (nodes, disks, network interfaces, application queues) and, for each, three signals — utilization, saturation, errors.
• Data flow diagram: from exporter to storage and alerting — where we aggregate, how long we keep raw series, and what retention period different metric classes have.
• Alert rules: thresholds for saturation (queue length, wait time), error rate, and utilization; duplicate suppression and severity separation.
• On-call dashboard: a single screen with the state of all services, transitioning from the overall view to a specific node in two clicks.
• Operational documentation: what each signal means, what actions to take when it fires, how to test a hypothesis, and how to roll back a collection change.
• On-call schedule: who responds to critical alerts, first-response times, escalation path, and shift handover rules.
• Expansion policy: how to connect a new service to collection in one commit, so metric and alert templates are not written anew.

All this is accompanied by review notes: which thresholds were revised and why — so that six months later the on-call engineer understands the logic instead of adjusting values at random.

Case study: how blind spots were closed and downtime was reduced

Before implementation

An infrastructure of several dozen nodes relied on availability checks: an alert arrived when the service was already unresponsive. Out of ten incidents per quarter, one or two were detected in advance.

Investigating each took hours — logs were used to reconstruct what overflowed first. Four failures in the same quarter were repeat incidents: the symptom was fixed, not the resource that led to it. The on-call engineer lived in a mode of nighttime wake-ups and manual data collection.

What was changed

We broke down each node along three axes — utilization, saturation, errors — and configured collection so that saturation is caught before failure: queue length, processing wait time, failure rate per request.

Metrics were consolidated onto one screen in Grafana, and thresholds were set from the observed peak. An action was tied to each signal: what to check, what to restart, when to involve a senior engineer.

Implementation result: eight out of ten problems began to be seen in advance, and response time dropped from forty minutes to five. Repeat failures — one per quarter instead of four, because investigation now ends with eliminating the root-cause resource.

The on-call engineer’s nighttime wake-ups were halved, and incident investigation takes about twenty minutes using the ready-made dashboard.

How does continuous metric collection differ from scheduled polling?

The difference is in what the collector sees between measurements. With scheduled polling, the agent takes the current value once per interval — everything that happened between points does not exist for the system.

Continuous collection maintains a stream at a short interval, so the picture is closer to the real behavior of the service rather than to the state at a fortunate moment.

A short load spike is the scenario where the discrepancy matters most. Queue saturation or a series of errors lasts tens of seconds: with a rare interval, such spikes either do not fall into any point or are blurred into a single average.

Detection accuracy does not fall linearly — the shorter the spike relative to the interval, the higher the chance of missing it.

A separate effect is smoothing. Scheduled polling almost always returns an average or delta over the interval, and average utilization easily hides saturation: the resource is free almost all the time, a short section is at the limit — and the graph looks calm.

Continuous collection preserves the signal shape and allows calculating percentiles, which show whether the connection pool or disk hit the ceiling.

The trade-off is collection overhead. A continuous stream produces an order of magnitude more points: storage volume and load on the agent and receiver grow, and on weak nodes collection begins to affect what it measures.

Therefore the choice is rarely binary: the USE metrics critical for failure — queue saturation and errors — are collected continuously, while the rest is left to infrequent polling.

We select the mode based on the service class and the question the metric must answer. If you need to prove that there were no short outages, infrequent polling is fundamentally unsuitable. Where a daily trend is sufficient, a schedule is enough, and a raw stream only bloats storage.

Technical doubts before starting work

USE metrics describe each node in three dimensions — utilization (Utilization), saturation (Saturation), errors (Errors). They are captured from what is already collected: agents on hosts, service exporters, time-series storage.

There is no need to replace current monitoring — calculation rules, thresholds, and dashboards are added to the existing environment.

How much will the load from data collection grow?

USE metrics are aggregates and shares, not raw counters with maximum detail. We choose the polling interval based on node criticality, store old data at reduced resolution, and limit volume with a retention policy. Resource consumption for collection itself remains within the margin of error.

How to avoid drowning in false alerts?

We calculate the saturation threshold from the baseline of a specific node, not from an absolute value; we set alerts with a confirmation window and suppression during planned maintenance.

For peak hours and different environments, thresholds are overridden separately — the on-call shift receives actionable signals, not noise.

The environments are different. Will one scheme work?

Yes. We make rules and templates common, and pass the environment as a label — test, pre-production, and production differ only in threshold values. This provides the same diagnostic logic across all environments without separate configuration sets.

What happens after handover?

We hand over documentation on metrics and thresholds, a response policy for each alert, and on-call shift training. During the warranty period, we support the environment and review false alerts — the configuration is refined for your load.

Let’s discuss the task: what to send for an estimate and plan

So that the scope estimate is based on facts rather than guesses, it is enough to send inputs on the current environment.

We will check which nodes already expose metrics, where a collector will need to be installed, and come back with a USE metrics implementation plan and a collection scheme tied to your topology rather than a generic template.

• Infrastructure diagram: nodes, network segments, clusters, external dependencies — it shows where to install collectors and where built-in tools are enough.
• List of services and their roles: what is critical for revenue and what can tolerate degradation — this determines the set of thresholds for Utilization, Saturation, and Errors for each level.
• Current monitoring: which agents and storages are already running, how alerts are configured, and how long data is retained — so we can integrate rather than break what exists.
• Access: a read-only account for inventory, access to the monitoring dashboard and nodes — we coordinate this once, and further work proceeds without your involvement.
• Target benchmarks: acceptable response time, critical failure scenarios, maintenance windows — from these we will assemble saturation thresholds and alert rules.
• Contact for discussion: an engineer or tech lead on your side who will confirm priorities and accept the result.

Describe the task — we will come back with an estimate and plan.