Why GPU monitoring lies: where metrics get lost
Accelerator utilization is almost always measured by averaging: one number per node per minute. Such a metric hides the main thing — the distribution.
An average load of 60% across eight cards describes equally well a steady mode and a picture where half the cards run at their limit while half idle waiting for tasks.
The second loss is memory. The memory occupied by a process and the actually available video memory diverge: fragmentation, un-freed buffers and kernel cache remain in the counters after a task crashes.
As a result, a new task doesn't start with a formally free resource, while the system keeps considering the node healthy.
A separate class is card degradation. Frequency reduction due to temperature and power, a growing number of correctable memory errors, falling throughput: the card responds more slowly but doesn't formally fail, so on an averaged graph its utilization looks normal.
Equipment downtime isn't recorded in this case, while the job queue and the waiting time of specialists grow.
This is how GPU monitoring turns into reporting rather than a tool. Lost metrics on individual cards mean invisible accelerator downtime, which is paid for with model training time and repeated runs.
This is solved by collecting indicators directly from each card via DCGM: core utilization, memory occupancy, frequency reduction and errors are visible for each accelerator separately — let's break down how this works.
What continuous accelerator telemetry collection gives
Continuous accelerator telemetry collection is not a dashboard for the sake of pretty graphs, but a set of thresholds by which the team makes decisions.
If metrics land in storage continuously rather than being sampled once a day, degradation is visible hours before it turns into a service failure.
We check against a checklist: each signal has a problem indicator and a pre-described action. Without that, monitoring turns into a stream of numbers that nobody reads.
| Signal | Problem indicator | Action |
|---|---|---|
| Compute unit utilization | Dip on some nodes while the overall norm holds | Redistribute the load between nodes |
| Video memory | Smooth growth of consumption without growth of requests | Find the leak, restart the worker before failure |
| Errors and restarts | Rare failures invisible in averages | Early notification to the on-duty engineer |
| Frequency under load | Drop in output during long operation | Drain tasks from the node, check cooling |
| Request queue | Growth of model response time | Expand the accelerator pool to saturation |
Telemetry collected by such rules changes reaction time: an incident is caught at the stage when a metric deviates, not by a user complaint.
Load balancing across the fleet raises average utilization and eliminates the situation where some accelerators idle while others run at their limit.
GPU monitoring modes: agent, collector, hybrid
The way utilization and video memory indicators are collected is determined by where the process reading accelerator counters runs.
If it lives on each node next to the workloads, the data is more precisely tied to specific processes; if a single shared collector polls the whole fleet over the network, operations are simpler. Hence three schemes that are usually chosen between.
| Mode | When it fits | Limitations |
|---|---|---|
| Local agent on each node | Different teams and workloads in one fleet, need breakdown by processes and devices | More points of failure and updates, the agent shares resources with the workload |
| Shared collector | Homogeneous fleet, single owner, simplicity of maintenance is the priority | Loss of network on a node looks like silence rather than a failure; no local detail, polling is rarer |
| Hybrid scheme | Large fleet where both detail and a single point of view are needed | Harder setup and debugging, consistency of labels between levels is required |
The choice is dictated not by the fleet size but by the required detail and the acceptable duration of the "blind zone".
The agent scheme ties a metric to a device and a process, so incident analysis is built on it; a shared collector is cheaper to maintain, but when a node restarts, its entire snapshot is lost.
The hybrid reads as a compromise: a lightweight collector stays on the nodes, handing aggregates upward, while detailed snapshots are requested on demand. The single point of view is preserved, and an investigation doesn't hit a wall of missing local data.
There is one limitation — consistency of node, device and process names between levels, otherwise metrics don't add up during analysis.
Implementation stages: from accelerator inventory to handover
Accelerator monitoring starts not with graphs but with inventory. As long as it's unknown which cards sit in which nodes, how they're interconnected and which teams share them, any metric hangs in a vacuum.
Our order of work is fixed — this way the result is reproducible rather than assembled from one-off scripts that work until the first reboot.
- Fleet inventory. We walk the nodes, record the accelerator model, memory size, link topology and distribution across teams. In parallel we reconcile the facts with the accounting systems — discrepancies are usually the source of blind zones where tasks fail without an explainable reason.
- Collection scheme. We deploy a hardware-level metric collector and define the set of signals: compute unit utilization, occupied memory, temperature, frequency reduction on overheating, memory errors. We choose the polling frequency as a compromise between detail and storage volume.
- Alerting rules. We tie each alert to an action: what exactly the on-duty engineer must do when it fires. We set thresholds from the observed load profile, not from documentation, otherwise the on-duty engineer drowns in noise and stops reacting.
- Testing under load. We run typical tasks, including peak ones, and see whether the system catches drops and memory leaks before they bring down training. Separately we test behavior when one node fails.
- Handover to operation. We hand over the response regulations, the list of responsible people and instructions for adding new nodes to monitoring. After handover, the customer's team expands coverage on its own, without coming back to us for every card.
The result is a transparent picture of the fleet: you can see who occupies accelerators and how much, where there's idling and where overload, and which failure has already begun.
Deliverables: configurations, alerting rules, regulations
The deliverables are not dashboard screenshots but a working framework that can be versioned, reviewed and deployed on a new node without the authors. Below is what exactly passes to your team along with the configured GPU monitoring.
- Collection configuration files. Which nodes and tasks metrics are taken from, the polling interval, the set of counters: compute unit utilization, occupied and free memory, temperature, throttling, driver errors.
- Alert rule templates. Thresholds with averaging windows, separation into warning and incident, repeat suppression and grouping — so the on-duty engineer isn't flooded with identical messages for a whole rack.
- Collection and storage scheme. The flow from nodes to the metric storage, retention period, labeling by nodes, tasks and teams: it shows whose process occupied memory, not just that memory ran out.
- Dashboards. Panels by nodes and by individual training tasks: utilization, memory, queue, bottlenecks — with history, not just the current snapshot.
- On-duty instructions. What to do when an alert fires: the order of checks, typical causes, when to restart a task and when to bring in a developer, and to what level to escalate.
- Maintenance regulations. Where configurations live in the version control system, who changes thresholds and how, how a change is verified before rollout to production nodes.
- Deployment scripts and acceptance protocol. Installing exporters on new nodes with a single command plus records of test firings: confirmation that every rule reaches the on-duty engineer.
We hand over everything listed in your code storage, not as an archive in a chat — the team continues working with the same set we do.
Case: unified metric collection for an accelerator fleet
An accelerator fleet of several dozen machines was serviced by an on-duty team: occupied memory and utilization were checked manually on a schedule, and they learned about overload from researchers — a run would fail at night and the node would idle until morning.
The head of the department named two costs of this mode: on-duty hours went into data collection instead of analysis, and failures were found after the fact, shifting model training deadlines.
What was changed
We built a unified telemetry collection based on DCGM: a service on each node sends utilization, occupied memory, temperature and process state to shared storage, and alerting rules were tied to thresholds rather than to scheduled walkthroughs.
Signals were separated by criticality: a process crash and overheating — immediate call, sustained memory growth — a warning with lead time to react.
We store history at different intervals: frequently for the last day, less often for months — that's enough for incident analysis, and the data volume doesn't grow uncontrollably.
Metrics before and after
The share of manual checks dropped from a daily walkthrough to selective control once a week — the on-duty engineer analyzes deviations rather than collecting readings.
Failure detection time went from hours (by user complaint) to minutes: the notification arrives on a deviation, not after a failed run.
A separate class of failures disappeared — invisible degradation, when processes were evicted due to memory and a task silently lost performance. Fault tolerance grew because nodes with an anomaly are removed from scheduling before new work lands on them.
What to do if DCGM is already deployed but metrics don't add up?
A discrepancy in readings almost never means the metric collector is broken. An existing scheme usually pulls data from several sources: accelerator counters, per-process metrics and aggregates already collected by the storage system.
Utilization is computed from compute block occupancy, while memory is computed from driver-side allocation, so these numbers describe different things.
We start by reconciling sources. We take values directly from the agent on the node, compare them with what reached the storage system, and with the sum across containers.
A discrepancy at the first step means agent configuration, at the second — losses during transmission and aggregation, at the third — different labeling by processes.
The most frequent culprit is the polling interval. If the collection period is longer than the cycle of memory redistribution within a task, peaks are smoothed out and the sum across the node stops matching the sum across processes.
The interval is aligned across all links, and detailed per-process counters are enabled selectively, otherwise collection starts competing with the task itself.
It happens that after a configuration edit the numbers don't change: the NVIDIA DCGM agent starts an old set of counters from the built image.
The agent needs to be rebuilt with the updated field description and the containers restarted — tasks are not stopped, collection runs in parallel with computation.
Then reconciliation on history: a day of data under the old and new schemes, separately for utilization and separately for memory. The residual difference between reserved and actually used memory volume is normal, everything else points to an unclosed source.
Technical concerns: overhead, access, noise
Engineers raise three objections before any others: how many resources the collection itself consumes, how to pull metrics out of an isolated environment, and whether alerting will turn into a stream of emails that people start ignoring. Below is how we close each one at the setup stage rather than after launch.
Will metric collection slow down model inference?
The agent takes aggregated accelerator counters — utilization, occupied memory, temperature — and doesn't read memory contents, so collection runs as a background process.
The polling frequency is configured separately: degradation is caught on a short interval, utilization reporting — once a minute. The share of time lost to collection is visible on the dashboard itself.
The environment is isolated, metrics can't be sent outside
The agent is installed on the hosts as a separate service with minimal privileges, and metrics go to a local collector inside the perimeter.
If outbound traffic is fully prohibited, we deploy the collector inside the perimeter and hand over the export via an already agreed channel — a file on a schedule or via a web interface. Credentials are not stored in plain text.
Alerts will be false and noisy
A threshold fires not on an instantaneous value but on sustained hold over a window — a single spike doesn't wake the on-duty engineer. Rules differ for memory, temperature and performance drop, and levels are warning, work shift and night call. For the first week we tune thresholds based on actuals so that few notifications remain.
We record all three points in the documentation along with the configuration — so the on-duty engineer can change thresholds and polling frequency on their own, without our help.Let's discuss the task: what to know before the start
To come back with an estimate rather than general words, a few inputs are enough for us. Describe where the accelerators live — your own cluster or rented capacity, how many nodes and cards, what tasks run on them and how they're divided between teams.
That's already enough to understand the scope: deploying a collection agent on each node, a time-series storage and monitoring panels differ in labor costs by factors depending on scale.
What's worth preparing for the conversation:
- list of nodes and accelerators: how many cards, which models, who occupies them and how;
- current state of observability: the vendor's standard metric collector or your own agent, where data is written, where graphs are viewed;
- access: what level of privileges you're ready to grant on nodes and the monitoring system, who approves changes;
- nature of the load: training or model inference, expected peaks, whether there are windows for restarting agents;
- alerting rules: to whom and how signals arrive, which thresholds are considered critical;
- target indicators: what needs to be seen first — compute unit utilization, video memory fill, compute errors or downtime.
In response you'll receive a staged implementation plan with the order of work and timelines, the composition of panels and alerting rules, a deployment scheme with access roles, as well as a list of metrics included in regular reporting.
If something is missing at the start, we clarify some of the inputs ourselves during a short review.
Describe the task in free form: configuration, bottlenecks, constraints on work windows. We'll come back with an estimate and an implementation plan.
