Configuring NVIDIA GPU Operator for Kubernetes

NVIDIA GPU Operator: where accelerator allocation is lost

An accelerator cluster rarely sits idle due to a lack of hardware — more often, accelerator idle time is caused by GPU allocation in the cluster not matching the actual state of nodes.

A node is labeled as containing a card, but the driver did not come up after a kernel update, the device plugin did not register the resource, and the scheduler no longer sees free cards. Meanwhile, jobs wait in the queue, and accelerators sit idle.

The root cause is usually manual driver installation on nodes. Some machines have a different set of libraries than the image; someone rebooted a server and did not restore the configuration; on one node the driver version is current, on another it lags several releases behind.

The more nodes there are, the faster the discrepancy accumulates, and the more often an engineer, instead of working with a model, has to figure out why a specific node stopped exposing its card.

The second bottleneck in job scheduling is resource accuracy. The scheduler allocates only what a node declares, so without correct card registration, workloads either do not start at all, or several processes share one card and fail on memory.

Some jobs then land on nodes without an accelerator and time out after queue time has already been spent.

NVIDIA GPU Operator moves drivers, card registration, and node labeling into the cluster's declarative state: a node reports what it has, and version discrepancies are resolved by a standard update rather than manual logins to the machine.

Below, we examine how this is structured and where the pitfalls remain — from driver compatibility with the kernel to shared access of multiple jobs to a single card.

What changes in the cluster after accelerator automation

After NVIDIA GPU Operator takes over the accelerator lifecycle, their allocation stops being a set of manual steps.

The driver, libraries, and scheduling components are installed on a node consistently, the cluster sees an accelerator as a regular resource, and allocates it among jobs.

Preparing a new node fits into a predictable window instead of a sequence that each engineer performed from memory.

Manual operations become noticeably fewer: driver updates and device access allocation are described declaratively and applied identically on all nodes.

This removes a separate class of failures — driver and library version discrepancies between machines, because of which a job either did not start or lost access to the accelerator while already running.

Accelerator observability stops being a separate story. Utilization, errors, and signs of overheating are collected in the same loop as the metrics of the rest of the cluster, so degradation is visible before jobs fail, not after.

The answer to the question “which node is behaving atypically” comes from a graph, not from the event logs of an individual machine.

Updates are also manageable: a new driver branch is rolled out to nodes in waves, the rest continue serving workloads, and reverting to the previous version does not require rebuilding the cluster.

In total, five measurable shifts — unified accelerator allocation, predictable node preparation time, fewer manual operations, an eliminated class of failures, and controlled updates.

Deployment options: manual configuration or managed layer

The choice between manual installation and a managed layer is determined not by cluster size, but by how often drivers change and how many different accelerators live in one pool.

NVIDIA GPU Operator combines the driver, device plugin, telemetry, and node labels into a single declarative object, but adds its own lifecycle that also needs to be maintained.

In practice, the choice is tested with one question: how many manual actions will remain when changing the driver version across the entire fleet. If the answer is “recreate the nodes,” we take the managed layer; if it is “change nothing,” manual setup is easier to maintain.

Hybrid mode is justified during migration: it allows moving one pool under management without touching the others, and rolling back at any step.

Implementation stages: from cluster audit to handover to you

The workflow is structured so that every change is verifiable and the cluster state is reversible. We start with inventory and finish with handing over configurations and metrics to your team.

1. Cluster audit. We collect the inventory of nodes with accelerators, labeling, installed drivers, and how the scheduler allocates devices. Here we also identify conflicts: old drivers, manual settings, forgotten restrictions.
2. Accelerator placement scheme. We agree on which jobs receive devices and how many, how we divide nodes between teams, which labels and restrictions prevent unauthorized pods from occupying an accelerator.
3. Operator configuration. We install drivers, device access, and telemetry declaratively — parameters are described in the repository. We enable sharing a single accelerator between jobs only where justified by the workload profile.
4. Validation on real workloads. We run not synthetic tests but a working model or service: time to readiness, utilization, behavior during pod eviction, and on several nodes simultaneously.
5. Rollback plan. We record the procedure for returning to the previous state: removing nodes from the pool, reverting changes, recovery points. We test rollback on a staging environment rather than describing it in words.
6. Handover to operations. We provide configurations, the placement scheme, metrics and alerts, instructions for standard operations and driver updates.

Each step ends with a verifiable artifact: a scheme, a configuration, a load test report. Therefore, the result does not depend on who remains on the team after handover.

Configurations, schemes, and policies that remain with you

The package is handed over not as a “folder with files” but as a working system: configurations in version control with a description of deviations from baseline values, schemes by which a new engineer understands within an hour how traffic and jobs reach the accelerators, and policies under which updates proceed without stopping services. Below is exactly what remains with you after acceptance.

• Configurations of nodes with accelerators — manifests of labels and node selection, accelerator sharing profiles, driver and device plugin parameters. We separately record which values were changed deliberately and why, so that the next update does not “overwrite” them with default settings.
• Component interaction schemes — from job scheduling to device allocation: which services are involved, in what order they start, where the point of failure is. The scheme is tied to object names in the cluster, not to abstract blocks.
• Access policies and role separation — roles for operations, development, and observability; who can change accelerator parameters, who only reads metrics. Permissions to change drivers are moved out of application accounts.
• Update policy — the procedure when a new branch is released: node preparation, draining, validation jobs, rollback criteria. Rollback is described separately and tested on a staging environment before handover.
• Operations documentation — typical failures (a job does not see the accelerator, a node is marked not ready, metrics do not match), diagnostic commands, and what to check first.
• Monitoring and alerting scheme — which metrics are collected, which thresholds are considered critical, who receives a notification and how.
• Acceptance and shift handover checklist — a short list of checks after any work in the cluster so that changes do not go unnoticed.

All of this is stored alongside the infrastructure code, not in chat messages — so updating the operator and updating drivers stop being events with an uncertain outcome.

Project case study: the cluster stopped sitting idle

We received a cluster for model training and batch data processing: dozens of jobs in the queue, heterogeneous nodes with accelerators of different generations, and a persistent feeling on the team that half of the runs fail “not because of the code.”

The bottleneck turned out to be not the scheduler but the layer between the hardware and the cluster.

Drivers, libraries, and the device plugin were installed manually on each node, versions diverged between machines, and after a reboot a node came back online without visible accelerators — a job failed at startup, consuming queue time and engineers' time.

We moved node preparation to NVIDIA GPU Operator.

The operator itself deploys drivers, the library set, the device plugin, and metric collection via a DaemonSet, labels machines based on the actual presence of an accelerator, and brings a new node to a working state without manual steps.

We additionally configured readiness checks: a node enters scheduling only when the accelerator is visible and has already been tested.

Metrics before and after

We also reduced the load on the on-call shift: typical failures are now resolved by recreating the node rather than troubleshooting a specific machine. This case is not about replacing hardware, but about a managed layer on top of it.

What to do if GPU is already deployed but jobs are failing

Jobs usually fail not because of the operator, but because two schemes diverge. On manually configured nodes, the driver was installed via packages, and labels there are set by hand.

Where NVIDIA GPU Operator runs, the same thing arrives in containers, and labels are managed by the operator.

While both schemes coexist, pods land on a node that does not accept them: the GPU resource is not published, the driver version does not match, labels do not match the selector.

Diagnostics begins with node status and events: you can see which nodes expose the resource and which do not, where the driver container restarts and drives the node into a not-ready state.

Then we compare three things per node — labels, tolerations, and driver version; discrepancies here account for almost all startup failures.

We migrate node by node, without stopping work: the node is closed to new pods, its workload moves to the others, then manual labels are removed from it and it connects to the common scheme.

After switching, a test pod verifies resource publication, and only then does the node return to service. Capacity does not drop — one node is taken out of service, not the entire pool.

Consolidating the schemes into one provides a single source of truth: driver versions, labels, tolerations, and policies come from one place, and manual edits are not made to them.

Partially configured nodes are not manually brought up to the others but are brought to the same form as managed ones — this eliminates the class of failures where a job fails because of the “wrong” node.

We preserve the original labels and tolerations: rollback is one operation, and the migration window is measured in minutes.

Technical concerns: compatibility, updates, rollback

Deploying the operator affects shared cluster nodes, so concerns about compatibility and returning to the previous scheme are natural.

Below are answers to what is most often checked before starting: what the current cluster is compatible with, in what order updates proceed, and what rollback looks like.

In what order do updates proceed, and can work continue during them?

Updates proceed in waves: first one node, where we validate the workload, then the rest. Jobs from the node being updated move to free ones, so queues and services remain available, and no downtime is planned.

The driver version is set in the cluster rather than edited manually on each node — no discrepancies between nodes arise.

What about the rollback plan?

The previous driver installation scheme is preserved until the end of the work, and the operator configuration is stored as a separate release with history — returning to the previous state takes one step. Removing labels returns nodes to their original mode. We record the rollback plan in writing before configuration begins.

How are team permissions separated?

Operator administration remains with the infrastructure team: updates, node labels, policies. Product teams work only in their own namespaces and request accelerators through a request or quota. They have no direct permissions on drivers and system components.

Let's discuss the task: we will return with an estimate and work plan

Estimating NVIDIA GPU Operator implementation rests on three things: which nodes we allocate for accelerators, how the cluster is currently structured, and what exactly will run on those accelerators.

Describe the task in free form — that will be enough for us to return with a plan, not with clarifying questions.

• cluster scheme: how many nodes and pools, where accelerators are located and how many on each
• how the cluster is managed and what constrains it: an isolated environment without external network, a private image registry, custom access policies
• which workloads will occupy the accelerators and whether they need to be shared between teams
• what has already been done: whether the driver is deployed manually, what attempts were made and where they stopped
• isolation and priority requirements: who can occupy an accelerator and what matters more when capacity is short
• work window: when rebooting nodes is acceptable and how sensitive downtime is

In response, you receive a working document that can be used to launch the work:

• a deployment scheme with nodes divided into pools and the role of each
• a breakdown of work by stages: node preparation, operator deployment, job scheduling validation, metric collection
• a rollout procedure with control points and a rollback plan to the current state
• implementation timelines and an acceptance checklist: which checks confirm that workloads actually went to the accelerators
• what will remain with your team: configurations, policy descriptions, maintenance instructions

Describe the task — we will return with an estimate and work plan for your cluster.