Setting Up Autoscaling for GPU Infrastructure for AI Workloads

Why GPU autoscaling does not sustain peak load

Classic GPU autoscaling scheme relies on average accelerator utilization over a window of several minutes. While traffic is even, the metric has time to react and the pool grows along with demand.

But as soon as a queue of training tasks appears, average utilization stops reflecting the real deficit: jobs are waiting, some accelerators are busy preparing data, and thresholds simply do not trigger.

Hence the second source of losses — idle accelerators between peaks.

The pool is kept at the lower bound so resources don't sit idle, so during calm hours the queue clears almost instantly, while when several experiments start simultaneously it grows faster than the scheduler can add nodes. Scaling lags by exactly the interval where load is highest.

Manual pool expansion adds hours of delay on top of this: request, approval, node bring-up, warming up images and weight cache, checking accelerator availability in the required region.

While this is happening, peak inference load is already degrading — responses stretch out, some requests time out, and training shifts to the next cycle.

From the outside this looks like a capacity shortage, although nodes are available — they are simply not in the right place at the right time.

We break down how this is solved: which signals should drive the pool instead of average utilization, and how to connect the job queue, warm-up, and rollback into one manageable scheme.

What GPU autoscaling provides: time, capacity, resilience

GPU autoscaling starts with the queue. While there are more tasks than free accelerators, each new one waits, and the queue tail grows faster than average.

Therefore, the first thing we measure is task wait time before training starts and the distribution of accelerator utilization across the pool: with manual management, congestion is learned about from complaints, not from a graph.

Managed scaling keeps utilization within an operating corridor. For a peak, the pool scales out in advance based on the queue-length signal, and during calm hours it compresses to a minimum — peak capacity appears by the time the launch starts, not an hour after it.

The reserve does not warm up idly: the share of idle accelerators drops severalfold, and the cost of cluster idleness ceases to be a constant line item.

Pool fault tolerance is the second effect.

Automation removes a failed node from load balancing, restarts the job from the latest checkpoint on a healthy accelerator, and does not hand non-working resources to the scheduler, so stuck jobs stop occupying capacity until manual intervention. We take the same telemetry that native NVIDIA tools provide — no separate agent is needed for this.

At the output, the customer gets a predictable queue instead of manual duty shifts. Peak windows close without expansion requests, nightly runs do not wait for accelerators to free up, and cluster behavior stops depending on who is on shift today.

Comparison of compute pool scaling modes

Compute pool capacity is changed in four ways, and they are not interchangeable — each has its own peak reaction delay and its own cost of error. The choice depends on the load profile: a steady request flow, predictable windows, or sharp spikes from external events.

In practice, modes are combined: the schedule covers the baseline curve, the metric or queue threshold picks up the remainder, and the reserve pool holds a minimum hot capacity.

The key value when choosing is peak reaction delay: it consists of the metric averaging window, node launch time, and model warm-up.

Implementation stages: from load audit to handover of operational rules

A GPU load profile is rarely even: nightly batch calculations, daytime interactive requests, and short peaks fit into a day with different amplitudes. Rules set by eye produce either idle expensive cards or queues in which the user waits longer than they consider acceptable. That is why we go from measurements to the scheme and only then to thresholds.

1. Load profile audit — we collect metrics over a representative period: utilization per card, queue length, job wait time, peak shape, share of idle cards during calm hours.
2. Pool scheme — how many nodes we keep in constant reserve, how many go into the elastic part, how jobs are distributed across cards, and where the bottleneck is: card memory, inter-node communication, or data preparation.
3. Scaling rules — target metric, trigger thresholds, step for adding and removing nodes, stabilization time, queue priorities, headroom for urgent requests.
4. Load testing — running a real profile with peaks and failure of one node: does the system manage to add resources, how long does a new card take to launch, what happens to the queue at that moment.
5. Handover to operations — operational rules, monitoring dashboards, degradation alerts, manual intervention scenarios, and rollback of rules that is clear to the duty shift.

After handover, GPU autoscaling is not frozen: the profile changes, and thresholds are revised using the same metrics — otherwise, in a couple of months the setup starts working against the load again.

Configurations, schemes, and operational rules: document set

Working autoscaling is not only pools that grow under load. Without a document set, any configuration change becomes a risk: one engineer remembers the trigger thresholds, the duty shift does not know which alerts require intervention, and rollback is done from memory.

That is why we hand over to the customer a set of artifacts that allow the infrastructure to be maintained in-house.

• Pool configuration — files with trigger thresholds, minimum and maximum size, node warm-up and cool-down time, mapping of job classes to accelerator types. Edits go through a repository, not manually on a live system.
• Infrastructure scheme — placement of pools, zones, channels to model and data storage, failure points and their impact on scaling. Updated with every topology change.
• Access matrix — which roles change limits, bring up nodes, read metrics and logs. Tied to corporate groups: when an employee leaves, access is revoked in one operation.
• Duty regulations — what to check at the start of a shift, which alerts are critical, what can be done without approval, and where to escalate. Prohibited actions are listed separately, such as manually lowering limits “so it does not make noise.”
• Rollback instructions — step-by-step restoration of previous thresholds and pool sizes, verification that load is again following the old scenario, and control metrics confirming the result.
• Change log — what was changed, at whose request, and with what effect: after a month it is clear which threshold removed queues and which only added idleness.

The set is stored alongside the configuration, undergoes review, and is versioned — on the next edit it is clear what was changed and why.

Implementation case study: eliminating queues under peak loads

What it was

The accelerator pool was allocated for peak demand and stood at full size: at night idleness held at 60–70%, while during the day the training queue grew to 9 hours.

Researchers' jobs waited in a shared queue without priorities — a small single-accelerator experiment could sit behind a large training job for more than a day.

Capacity shortage incidents accumulated at 4–6 per month, and almost every one was closed manually: looking for free nodes, negotiating with a neighboring team, stopping other people's jobs.

What was done

We merged disparate pools into a shared one and exposed per-node utilization metrics to the autoscaler: scaling is driven by queue length and memory utilization, not by the average over a period.

Rules and priorities were described in KEDA: separate queues for training and inference, a node limit per team, and node release immediately after job completion.

We warm up images in advance — a new node enters operation without waiting for loading, so scaling does not hit environment preparation.

What became

Training queue wait dropped from 9 hours to 20–40 minutes at the same peaks, and the idle share fell from 65% to 18%. Capacity shortage incidents — 0–1 per quarter instead of 4–6 per month, and manual triage is no longer needed.

Rollback is provided for: autoscaling rules are removed and the pool returns to static mode in one step, without rebuilding jobs or losing running training sessions.

What to do if the accelerator pool is already deployed manually?

A manually deployed pool usually rests on fixed labels: specific jobs are pinned to specific machines, and growth and shrinkage are done by an engineer on request.

While the load is predictable, the scheme works; at the first spike, time passes between “we need more accelerators” and “accelerators appeared,” and all that time the job queue grows, and some requests do not wait for a response.

Such a fleet can be moved to managed scaling without stopping services. The existing pool is described declaratively — as is, with current labels, names, and job distribution: the description becomes the source of truth, but at this stage it changes nothing.

Backward compatibility is preserved: familiar ways of accessing nodes and manual commands continue to work, and nothing breaks at the moment of enablement.

Enablement is gradual. First — observation mode: the rules calculate the target number of nodes but do not apply it, and we compare the calculation with the actual pool behavior and refine thresholds based on real utilization.

Then the rules work on one pool or during non-peak hours, and only after that are they rolled out to the shared pools: the decisions an engineer made by hand — when to add a node, when to free an idle one — become formalized conditions.

Downtime-free migration is built on careful node removal: a machine leaves the pool only when it has no active jobs left, and new load is no longer scheduled there. Jobs are not restarted, and queues do not lose their positions. You can return to manual management at any moment without rebuilding the pool.

Technical doubts: queues, cold start, fault tolerance

Engineering questions before launch are usually the same: will capacity appear in time for a peak job, what happens to training when a node is removed, and will code need to be touched.

What if the capacity quota runs out during a peak?

We set the upper boundary in advance and assign priorities: interactive jobs get resources first, batch jobs wait or move to a separate pool. If the provider does not provide the required volume, part of the work goes to a reserve profile — slower, but without failure.

Preemptible nodes and maintenance removal — what about training?

Possible if the load can be restarted. The training cycle writes checkpoints on a schedule: a preemption or node removal rolls the job back to the latest point, not to the beginning. Before maintenance, the node stops receiving new jobs, saves state, and only then shuts down.

Will training code have to be rewritten?

As a rule, no — autoscaling lives at the infrastructure level. The edits are pinpoint: the checkpoint path, reaction to the node completion signal, wait timeouts. That is dozens of lines, not a model rework.

Let's discuss the task: we will design a scaling scheme for your profile

So that the assessment is based on your load profile rather than average figures, the review starts with data. Send what you already collect — we will assemble the pool scheme and work order.

• Load profile: daily and weekly peaks, ratio of long-running jobs to short requests, acceptable queue wait time.
• Current pool scheme: how nodes are grouped, what launches jobs, where the bottleneck is — accelerator utilization, memory, or network throughput.
• Metrics for a typical week: accelerator load, queue length, share of rejected jobs, time for a new node to enter operating mode.
• Environment constraints: placement regions, isolation requirements, available quotas, maintenance windows.
• Target indicators: the peak load that must be sustained, acceptable downtime when the number of nodes changes, the failure class that is unacceptable.
• Integration points: monitoring, scheduler, image registry, configuration management tools.

Then we work as follows. First, together with your engineers, we review the load profile and find what exactly limits scaling — node warm-up, lack of quotas, or absence of feedback from the queue.

Then we propose a pool scheme: which node groups to allocate, which metrics to use to decide on growth and reduction, how quickly a node should enter operation, and how to configure rollback.

Based on the review, we return with an assessment and plan: stages, what we do, what remains on your side, and which risks we close at each step.

Describe the task — what we are scaling, with what peaks, and under what constraints. If some metrics are not yet collected, say so: we will clarify during the review and look at the available logs.