Cost Optimization for ML Training: Spot and Preemptible VMs

Spot instances: why training hits the wall of accelerator idle time

Dedicated accelerators are usually ordered in advance and for a long interval, so the queue for accelerators is measured in days, not minutes of waiting.

While a resource is assigned to one experiment, the next one waits, even though the node itself is idle for half the window: data preparation, waiting for the previous stage, and analyzing results consume time that has already been paid for. Idle compute nodes in such a scheme are the norm, not a failure.

Interruptible instances (Spot) remove the price constraint but add their own: the platform reclaims the node based on an external signal, and the window for graceful completion is short.

If a task does not write checkpoints, training rolls back to the last successful state or starts over. In the worst case, unfinished training tasks are simply lost, and they are restarted manually, checking logs and restoring the environment.

The main cost item here is not the rate, but manual restart. An engineer is on duty at the console, the time to recovery depends on who noticed the failure first, and several parallel experiments compete for freed-up nodes.

The cost of model training grows due to idle time and rework: the same stage is computed twice, and the queue for accelerators does not get shorter.

We break down how this is addressed: queue and priority management between tasks, frequent state saving, automatic restart of interrupted tasks on free nodes, and resource sharing between experiments so that idle time is not paid for.

What changes after switching to interruptible capacity

The first change is visible within the first day: accelerators stop sitting idle waiting for a dedicated node to free up. The share of idle accelerators drops from a third to single-digit percentages — tasks queue and occupy freed capacity within seconds, not hours.

After a few days, the nature of failures changes. Node revocation stops being an emergency: state checkpoints are written every few minutes, and training continues from the last one, not from scratch.

The Kubernetes scheduler recreates the task on another machine, distributed training reassembles worker groups and picks up data from the last consistent iteration.

By the end of the second week, manual restarts stop being the on-duty engineer's job.

Automatic restart with checkpoint integrity verification and return to the queue handles typical failures without human involvement — the number of manual interventions in standard scenarios drops to zero, and the share of tasks that lose progress when a node is revoked tends to zero.

By the first month, the main effect becomes apparent: training throughput increases severalfold on the same fleet of accelerators.

Time from hypothesis to metric shrinks from weeks to days, and the number of concurrent experiments is now limited by task formulation, not by available infrastructure.

Spot instances, dedicated nodes, and hybrid pool

Model training involves multi-hour runs, where the cost of idle time consists of lost progress and re-paid time.

The key parameter when choosing a platform is node revocation frequency: for interruptible ones it is an order of magnitude higher, for dedicated ones it is close to zero. Then the only question is how to combine cheap capacity with predictable run completion.

The Kubernetes cluster labels pools: nodes with revocation labels accept only tasks capable of saving checkpoints, dedicated ones carry the stable training core, and the queue itself migrates evicted work.

The application receives a revocation signal in advance and has time to flush state to object storage, so no modification of the model code is required — only launch manifests and eviction policies.

The main savings come from the hybrid: the rough part of the run goes on interruptible nodes, and the amount of lost hours is limited by the interval between checkpoints.

A dedicated peak reserve holds the final stretch and prevents missing the deadline, even if node revocations come in waves.

How we deploy training on interruptible nodes

An interruptible node can go away at any moment — in a Spot pool this is a standard event, not an emergency. The engineering challenge is to ensure that node loss does not turn into lost training progress and repeated runs.

We proceed step by step: first we measure the workload profile, then we design placement, then we test how the system survives real interruptions.

1. Workload profile audit. We look at step duration, memory consumption, share of time spent on data exchange, and training tolerance to stoppage. The output is a map of sections that can be given to interruptible capacity without risking convergence.

2. Task placement scheme. We separate roles: coordination, checkpoint storage, and critical services remain on stable nodes, heavy computation goes on interruptible ones. Then node loss does not stop the entire experiment.

3. Checkpoint mechanism. We select the save frequency based on step duration, write states atomically and outside the node itself, and version them. After an interruption, training continues from the nearest valid checkpoint, not from the beginning.

4. Interruption response procedure. We determine how long to wait for capacity to return, how to reuse freed nodes, and when to switch to the stable pool so that the experiment queue does not stall.

5. Recovery testing. We intentionally shut down nodes during a run and measure progress loss, time to return to the previous pace, and the share of tasks completed without manual intervention.

6. Training maintenance. We hand over on-duty procedures and policy settings, then adjust them based on actual results — as the model profile and pool behavior change.

Configurations, policies, and procedures you receive

The output is not a set of disparate scripts, but a described training loop.

It can be handed over to another team, and they will continue work without verbal explanations: the artifacts record what is considered normal and how to act when interruptible nodes are reclaimed in a batch.

• Pool scheme — how many pools, what node types in each, proportions of interruptible and permanent machines, distribution across zones. It is clear which stages will survive simultaneous node revocation.
• Node selection policies — rules by which a task lands on Spot machines while critical steps remain on permanent ones: priorities, accelerator type constraints, queue wait time.
• Access policies — roles of engineers and service accounts, separation of rights to change configurations and to read logs, key rotation procedure.
• Checkpoint storage scheme — save interval, retention depth, separation of fast local and long object storage layers, cleanup rule by count and age.
• Recovery procedure — how training comes back after node revocation, from which checkpoint, how many steps of progress are lost, and who confirms return to work.
• Infrastructure documentation — descriptions of pools, quotas, limits, and failure points, change procedure; stored in the same repository as run descriptions.
• Observability — dashboard for interruption share, idle time, and pool utilization, so the scheme is changed based on data, not gut feeling.

Quarterly, these documents are reviewed along with changes in the provider's node type availability — so the configuration does not diverge from reality.

Case study: fine-tuning a ranking model on interruptible nodes

Fine-tuning of the ranking model proceeded in cycles: new interaction log, feature rebuild, run, validation. The main losses were not in the computation itself — accelerators sat idle between runs, and node revocation resulted in lost progress and manual troubleshooting.

We moved the runs to interruptible nodes: checkpoints are written to object storage every few hundred steps, the scheduler tracks revoked Spot instances and brings up a replacement from the same pool, and training continues from the last saved checkpoint.

Final validation and artifact packaging were left on a small stable reserve — interruption is unacceptable there, and the amount of work is small. In Google Cloud, such a pool is assembled from interruptible and regular machines in one group.

Node revocation stopped being an event: the run returns to work on its own, without an on-duty engineer. The share of idle time fell fivefold, time to model release more than halved, and manual interventions were reduced to checks during final validation.

What to do if part of the capacity is already occupied by permanent load?

Permanent services — request handlers, queues, cache, databases — keep utilization around the clock, while training comes in peaks: sometimes it needs the entire fleet, sometimes half sits idle.

If everything lives in one pool, the benefit of preemptible machines turns into a seesaw: either training kicks service processes off nodes, or hardware is reserved for it that sits idle most of the day.

The working scheme is to split pools into three parts. A stable pool for permanent load covers it with peak reserve, a preemptible pool is given to training and does not affect services, and a buffer pool switches between them on a schedule.

If nodes are still shared, task priorities and eviction classes are introduced: service processes are marked as non-interruptible, training ones as yielding first.

Scheduler labels and tolerations prevent training from landing on nodes with critical services, so sharing does not turn into a fight for memory and disk I/O.

Capacity planning is built on measurements, not intuition: they take the upper percentile of service utilization over a long period, add peak reserve, and only the remainder is declared available for training.

Training starts in low-load windows and is conducted with checkpoints — an interrupted process continues from the last checkpoint, not from scratch.

This way permanent load remains in predictable conditions, and training takes everything that is truly free: fleet utilization grows without risk to services.

Technical doubts and answers on interruptible nodes

Node revocation is not a failure, but a standard event with a warning: the environment receives a signal, has time to complete the current step, and unload state.

Revocation frequency depends on the resource class and region — in practice it is single-digit percentages of the day, not 'the node goes down every hour'.

The scheduler restarts the task on a new node and picks up the last checkpoint, so the run continues rather than starting over. It is precisely on this mechanics that the calculation of whether Spot instances are beneficial for a specific task is built.

Progress loss is limited by the interval between checkpoints, not the length of the run: with saving every few minutes, a rollback costs minutes of computation.

We write checkpoints to object storage with versioning and lifecycle policy — fresh versions remain for rollback, old ones are deleted, and storage volume does not grow indefinitely.

What about fine-tuning a large model?

It is critical what exactly goes into the checkpoint. If only weights are saved without the optimizer state, convergence degrades after restart. We write the full state and verify recovery on a test run before starting.

Let's discuss the training task and come back with a plan and estimate

To make the conversation substantive, it is useful to come with workload inputs.

The current training profile is sufficient: which models and on what dataset are being trained, how many epochs, how often tasks are launched, what share of time goes to waiting for accelerators, and what happens when a machine is suddenly revoked.

• Task statement. Which training jobs you want to accelerate first and where the bottleneck currently is — queue, accelerator idle time, or excess machine reserve.
• Workload inputs. Task profile, memory requirements, duration of a single run, readiness for interruption and restart from the last checkpoint.
• Current setup. How launches are organized, where data and model artifacts are stored, whether there is observability and per-team consumption accounting.
• Implementation plan. We will show how part of the fleet is moved to Spot instances with automatic return to the stable pool when capacity is insufficient.
• Scope of work. Tagging jobs by interruption eligibility, capacity selection policy, checkpoint saving, automatic queue scaling.
• Timeline estimate. Range by stages — from a trial migration of one task type to full rollout with a rollback plan.

Next is a short business conversation: describe the task and current training profile, and we will come back with a plan, scope of work, and timeline.