Why GPUs in Kubernetes sit idle and derail releases
The cluster reports full accelerator utilization, while half of training jobs have been queued for two days. The reason is that the scheduler sees the whole node, not an individual accelerator: a job monopolizes the card and holds it even where a fraction would suffice.
Then unevenness comes into play. One job requests a lot of accelerator memory and few compute cores, another the opposite, and they cannot be packed together onto one node: formally free resources exist, but in practice the node fails on one of the dimensions.
Some nodes sit idle while others are overloaded, accelerator idle time grows, and cluster throughput drops.
On top of that, training and inference compete for the same cards. Inference deals with bursts and short requests, while training deals with long tasks that are expensive to interrupt: losing a checkpoint means restarting from the last saved one and losing hours. Without priorities and quotas, whoever came first wins, not whoever is closer to the deadline.
Per-node utilization metrics do not show this picture — you need a breakdown by each accelerator and by job queue, otherwise the bottleneck is sought in the wrong place.
The result of the mistake is visible in the calendar: model training misses deadlines, engineers wait weeks for a card to free up, and the queue stops being manageable. We'll break down how this is addressed.
What managed accelerator scheduling provides
The default scheduler looks at free node resources and places a job on the first suitable one.
With mixed workloads — training, model inference, data preparation — this creates skew: some accelerators are loaded to the limit, others sit idle, and a job hangs in the queue waiting for the 'wrong' node to free up.
Managed scheduling starts with nodes getting clear labels: accelerator model, memory capacity, availability for specific teams.
Then resource requirements and queue priorities are set — a short model inference request does not queue behind many-hour training and does not block others.
The effect is measurable: accelerator utilization evens out across nodes, average job wait time drops, and cluster throughput grows on the same hardware configuration — without additional purchases.
The class of failures 'node busy with someone else's process' also goes away: a job no longer lands where memory is already taken by a neighbor and does not crash at startup.
The foundation of such a scheme is NVIDIA GPU Operator: it brings drivers, device labels, and resource accounting to a unified form, so the scheduler sees the real picture of accelerators, and priorities work in fact, not by rough estimate.
GPU allocation modes in Kubernetes: what to choose
The accelerator access mode is set not in the pod manifest, but at the level of the device plugin and scheduler. The scheduler sees an accelerator as an extended resource: as many units as the node advertises, that many pods it will place.
This directly determines packing density, latency spread, and how services behave when one process takes all the memory.
Below are four options that are actually chosen between. NVIDIA GPU Operator deploys drivers and the device plugin, but you set the access mode itself by configuration.
| Mode | How it works | When it fits | Limitations |
|---|---|---|---|
| Exclusive accelerator access | The node advertises whole devices, the scheduler assigns them one per pod | Training and fine-tuning: full memory and bandwidth are needed | Accelerators sit idle between tasks, the queue grows, density is low |
| Time-sliced GPU sharing | Multiple pods take turns sharing one device via plugin configuration | Light inference, interactive environments, debugging | No isolation by memory and compute blocks — a neighbor changes latency, a process failure affects others |
| Partitioning an accelerator into isolated instances | Each pod gets its own share of memory and compute blocks with hardware isolation | Dense inference where predictability under peak matters | Not all accelerator models support it, the set of profiles is fixed, large tasks need the whole card |
| Node-level isolation | Separate pools with labels and placement rules: training and serving do not overlap | Mixed clusters where training must not interfere with serving | The pool sits idle between runs, utilization drops, schedule control is needed |
Modes are combined: training exclusively in its own pool, inference shared or by instances. We verify the choice on a real workload profile, otherwise dense packing will turn into increased latency precisely under peak.
How we implement accelerator scheduling: stages
The workflow is arranged so that changes do not have to be rolled back after the first serious load. We start with inventory and finish with regulations for the team that will remain working with the cluster. Each step is closed with a verifiable artifact — a diagram, configuration, or run report.
- Cluster audit and workload inventory. We collect which jobs request accelerators, how long they live, how many devices they hold, and how many sit idle. We record peak windows and places where the queue is already growing.
- Job placement scheme. We divide the cluster into node groups for training, inference, and batch processing, set device profiles and quotas so that one team does not occupy all accelerators.
- Scheduler configuration. We install NVIDIA GPU Operator — driver, device plugin, and metrics update as a single whole. Then priorities, labels, and placement rules by which a job lands only on a suitable node.
- Fault tolerance testing. We take a node out of service one by one, crash the driver, hit accelerator memory limits — and see whether the job is rescheduled and whether it drags neighbors down with it.
- Run on real workload. We run the customer's scenarios on production volumes, compare estimated time with actual, and adjust limits where the scheduler keeps a job queued for no reason.
- Handover to operations. We hand over the cluster diagram, configurations, quota rules, alert set, and on-call instructions — and go through typical failures with the team.
Documentation, manifests, and regulations delivered
We deliver not only a working cluster, but also a complete set of materials your team can use to operate, extend, and update it without us.
Everything is stored in your repository next to the manifests — in the same branches and with the same change history as the working configurations.
- Scheduler configurations — manifests with job placement policies, accelerator reservation for specific workloads, and priorities for training and model inference queues.
- Cluster diagram — topology of nodes with accelerators, division into pools, failure zones, and connections between the scheduling service, storage, and monitoring system.
- Team access policies — namespace separation, accelerator quotas, and roles: who can launch training, who can change shared settings, who can only view metrics.
- On-call engineer instructions — what to check during degradation, how to find the job holding an accelerator, how to take a node for maintenance and return it to service.
- Update procedure — order of changing drivers and NVIDIA GPU Operator versions: first a test pool, then a limited one, then the whole cluster, with pre-described rollback criteria.
- Metric catalog and alerting rules — which accelerator, queue, and node indicators are considered normal and at what point the on-call engineer receives a signal rather than noise.
- Decision log — why this scheduling profile was chosen, what limitations it imposes, and what to change when load grows or the hardware fleet changes.
Each document is tied to a configuration version: when a manifest changes, the description is updated as well. A month later, your team reads not a chat thread but the current source — and makes edits itself.
Case study: job queue stopped waiting for an accelerator to free up
The customer kept model training and inference in one Kubernetes cluster, where an accelerator was allocated as a whole node per job. The scheduler did not see how much memory on the device was free, so a queue formed even when some accelerators sat idle. Jobs crashed at startup with a resource busy error and had to be restarted manually.
We separated device profiles for training and inference, enabled queue priorities and an accelerator availability check before container startup. Small tasks were moved to shareable profiles — they no longer occupy a whole device where a fraction is enough. High-priority jobs preempt low-priority ones instead of waiting for the node to free up entirely.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Accelerator utilization | 40–50% | 82–88% |
| Job wait time in queue | up to 6 hours | 15–20 minutes |
| Jobs failed due to busy node | every fifth | isolated cases |
We also set up observability: for each node, utilization, job status, and failure reasons are visible, so degradation is noticeable before the queue stalls. Rolling back to the previous profile takes minutes and does not require rebuilding the cluster.
How does shared accelerator access differ from dedicated access?
The access model answers one question: does a task get the entire accelerator or share it with neighbors? With exclusive ownership, memory and compute blocks are not reused by other tasks — response time depends only on the workload itself.
Shared access, on the contrary, raises utilization: an accelerator idling between processing steps is given to others.
The penalty for sharing manifests not in average performance, but in response time predictability. When several tasks share memory and the compute queue, contention arises: one heavy task raises response time for all neighbors, and tail percentiles grow faster than the average. Task isolation removes this effect — but at the cost of resource idleness between steps.
The choice is dictated by the workload profile, not the desire to save: for training and heavy inference — isolation, for a fleet of small services — sharing with a memory share limit.
In NVIDIA GPU Operator, the access mode is set by policy at the node pool level, so the same platform can be switched between models without rebuilding applications.
Technical doubts before starting work
Some questions arise even before the start: the cluster is already running, some nodes are configured manually, and no one is ready to change everything 'in one day.' Below are the things we most often discuss at the assessment stage.
The cluster is already partially configured — rebuild from scratch?
No. First, inventory: which nodes carry GPUs, which drivers and labels they have, who and how sets tasks. Then we bring nodes to a unified state one by one — drain from scheduling, migrate the workload, and return them already under operator management. The cluster keeps computing tasks the whole time.
Will model code need to be modified?
As a rule, no. The launch description changes: resource request, limits, priority, node type affinity. Training scripts and inference services remain the same — it is enough to repackage them and move them to the new launch rules.
How to limit access to expensive resources?
Through quotas and priorities: namespaces with limits on the number of GPUs, default policies, separate priority classes for production and experiments. A task without an explicit resource request will not land on GPU nodes, and a heavy experiment will not preempt a service.
What happens if a GPU node fails?
The node is marked unavailable, its pods are rescheduled to free GPU nodes with suitable labels, unfinished trainings restart from the last checkpoint.
Recovery takes minutes instead of manual investigation, and capacity is reserved so that production does not run out of resources.
Let's discuss the task: what to clarify before starting
Estimating such a project comes together quickly when the inputs are clear before the start: workload profile, how teams work, response time requirements, and the current state of the cluster.
Below is a list of what is worth clarifying in advance: with it, we come back with a work plan, not a string of clarifying questions.
- Workload profile. What exactly is being executed — training, additional training, or model inference; the ratio of modes, average and peak accelerator utilization, memory requirements per process.
- Job launch pattern. Who and how launches them: researchers manually, a scheduled service, or an external system. This determines whether team quotas and separate namespaces are needed.
- Response time requirements. Allowable queue wait time, how quickly a job gets onto an accelerator, priorities, readiness to preempt low-priority jobs for urgent ones.
- Current state. Kubernetes and its topology, node composition, deployed components for working with accelerators, driver installation method, and how reproducible the configuration is.
- Data and network. Where datasets, checkpoints, and model artifacts are stored, how fast they are read, and whether bandwidth between nodes is sufficient.
- Access and control. Who owns changes, how permissions are separated, what must be logged, and which checks are mandatory before rollout.
- Handover to the team. Who takes over operations — the platform team or researchers, whether regulations, diagrams, and on-call training are needed.
Then the workflow is usually as follows: review of inputs and current configuration, a pilot on one team, then expansion to the rest. Describe the task within these boundaries — we will come back with an estimate, timeline, and handover plan.
