Spot instances are interrupted mid-batch calculation
Preemptible instances are the backbone of batch processing: the platform provides them for computation at a significant discount and has the right to take them back at any moment.
This is exactly where progress loss comes from: a monolithic recalculation keeps intermediate results in process memory, and resource revocation kills that process entirely. A job that had been running for six hours starts from the first line after a restart.
The revocation notice arrives tens of seconds in advance — enough to correctly finish the current step, but not enough to save all state.
Worse, resources are not taken back one node at a time: if a group is assembled in a single pool, the revocation comes in a batch, and half the fleet drops out at once. The queue stalls, and some tasks remain in an uncertain state — some checks have passed, some have not.
In day-to-day operations, it looks like this: the nightly recalculation does not fit into the window because every interruption rolls work back by several hours; repeated runs duplicate records if operations are not idempotent; computation downtime grows, and reports arrive late.
Engineers start keeping critical tasks on permanent resources, and the gain from preemptible instances is eaten away.
We break down how this is addressed: we split the recalculation into short steps, write checkpoints to object storage, make operations idempotent, and limit the share of preemptible nodes so that the queue survives resource revocation without manual intervention.
Effect of managed interruptions in batch calculations
Preemptible resource revocation is not an accident but a normal mode: a node is removed on a signal, and without preparation the process dies with it.
We build processing so that the task catches the revocation signal, has time to write a checkpoint to object storage, and restarts on another node from the last saved state. Between checkpoints, one segment of computation is lost, not the entire batch.
We measure the effect by three values. Restart downtime is reduced to minutes — source data is not reread from scratch, only the failed subtask is brought up.
The share of tasks completed reaches 99% and above even with frequent revocations, because progress is recorded continuously rather than at the end of the run.
Capacity utilization grows many times over: preemptible resources are cheaper than idle ones, and the queue no longer waits for a dedicated node to free up.
A whole class of failures is also eliminated — “the resource was revoked, the batch failed, we recalculate from scratch.” Checkpoints are written asynchronously and do not slow down computation, reruns are idempotent, and an orchestrator such as Kubernetes reschedules the subtask onto available capacity on its own.
Batch execution time becomes predictable: the restart budget is known in advance, and deadlines do not shift because some nodes were revoked mid-run.
Spot instances vs dedicated and reserved
Placement of batch tasks rarely comes down to a single mode: the bottleneck is not the instance type but the distribution of load across pools.
The difference between pools is exactly one thing: what happens when a resource is revoked, and who is responsible for the restart — you or the provider.
| Option | When it fits | Limitations |
|---|---|---|
| Preemptible instances | Tasks with checkpoints: rendering, indexing, data mart recalculation, model training | Revocation with short notice; during peak hours capacity may be unavailable altogether |
| Dedicated capacity | Critical path: data ingestion, production services, scheduled reporting | Downtime is paid for; capacity is taken for peak rather than average load |
| Reserved resources | Steady load profile for months ahead, predictable volume | Commitment on volume and term: when the task profile changes, part of the capacity sits idle |
| Hybrid scheme | Mixed flow — a stable baseline plus bursts and one-off runs | Requires an eviction policy and queue separation; capacity calculation is more complex |
They combine them like this: the baseline runs on reserved capacity, critical stages on dedicated, and the tail and peak runs on preemptible. The scheduler stores checkpoints and reruns, so instance revocation loses the last step, not the entire run.
The share of preemptible instances in computation reaches two thirds — provided that tasks are idempotent and can continue from the last checkpoint.
Implementation sequence: from load analysis to handover
We build the work sequence from the task profile, not from the cloud configuration: job duration, presence of checkpoints, and sensitivity to launch delay determine what share of the fleet can work on Spot instances at all and what will have to be kept on regular machines. Next come the placement scheme, selection rules, and testing against real node revocations.
- Load audit. We count the queue, the distribution by duration and resources, and mark jobs that restart without loss and those that need a checkpoint.
- Placement scheme. We determine the ratio of preemptible to regular machines, distribute them across availability zones, and set a limit on simultaneous revocation.
- Rule configuration. We describe which machine types we take, how we respond to capacity shortages and cost growth, and how quickly we return a job to the queue.
- Failure testing. We deliberately revoke nodes under load: whether a job fits within its state-saving limit, and whether there are duplicates or lost results.
- Observability. We expose metrics — share of interrupted jobs, time to restart, volume of recalculations. They show where the rules work against us.
- Handover to operations. Instructions for the on-call engineer on typical failures, alerts on anomalies, a procedure for changing limits, and shift training.
For the first weeks, we compare actual metrics with calculated ones and fine-tune thresholds — after that, the configuration runs without constant engineer intervention.
Set of configurations and procedures delivered
What the customer is left with is not “configured preemptible machines” but a repeatable loop: it can be handed over to the on-call shift, redeployed, and verified. Below is the set we deliver along with the working system; each item addresses its own class of failures.
- Pool and job configurations — descriptions of preemptible and permanent nodes plus batch job manifests with resources, priorities, and maximum runtime. They immediately show which load can tolerate interruption at all.
- Placement scheme — a map: which jobs go to temporary capacity, which remain on permanent capacity, and labeling rules that prevent critical services from landing on nodes that may disappear.
- Access policies — roles for the on-call engineer and developers: who changes pool size, who removes the reserve, who reads the change log. Mass node release becomes a controlled action rather than an accident.
- Restart procedure — handling the interruption notice: how much time is allowed for completion, where intermediate results are written, and from which step the job continues after a rerun.
- Documentation — how to rebuild the loop from scratch, what to do when nodes in one zone are released simultaneously, and how to quickly return to permanent capacity.
- Control points — metrics and thresholds: share of interrupted jobs, time to restart, share completed on the first attempt. They show that the scheme holds up after your changes as well.
The set lives together with the infrastructure: when placement rules change, the procedure is updated too. Describe your task — we will come back with an estimate and a plan.
Case: calculations on preemptible resources without losses
What it was
Calculations ran as a batch of several hundred tasks in a nightly window, and the result was needed by morning.
Nodes were kept dedicated: they sat idle between runs, but any failure was handled manually — an engineer figured out which steps had managed to write results and restarted the branch from the beginning.
One-off runs on Spot instances ended the same way: resource revocation hit the middle of the batch, and up to 15% of tasks lost their results.
What changed
The batch was split into idempotent steps with checkpoints in object storage, and the build queue was moved to Kubernetes with a separate pool of preemptible nodes.
A control component intercepts resource revocation: the task returns to the queue from the last checkpoint, and a step that is revoked too often automatically moves to a regular node. Manual handling remained only for two steps where an intermediate result cannot be saved.
What became
- Share of tasks that lost results on resource revocation: 15% → 0.
- Restart time: 25–30 minutes of manual work → 40 seconds automatically.
- Manual interventions: 3–5 per night → none over a quarter of observations.
- The nightly window shrank by a third: preemptible nodes absorb the peak that previously waited for dedicated nodes to free up.
How a preemptible instance differs from a reserved one
A reserved resource is a capacity agreement: the instance lives until you turn it off yourself, and the scheduler will not take it for other customers.
A preemptible one is the opposite: the provider gives idle capacity at a lower cost but reserves the right to revoke it at any moment. The difference is not in configuration but in the ownership model.
Availability guarantees and revocation priority are the two key differences. When capacity is scarce, the provider first frees preemptible machines and only then looks at reserved ones.
Revocation comes with a short warning — at Amazon Web Services about two minutes — during which the application must save its result and terminate correctly.
| Property | Preemptible instance | Reserved resource |
|---|---|---|
| Availability guarantees | Revocation at any moment | Lives until you shut it down |
| Revocation priority | Taken first when capacity is scarce | Not affected |
| Long-running calculations | Only with intermediate saving | Suitable directly |
A calculation that runs for hours without intermediate saves loses progress on a preemptible instance with every revocation.
Batch tasks rely on a different logic: work is divided into short independent units, each unit’s result goes to external storage, and the queue reissues interrupted units to other nodes.
In practice, a mixed scheme is used: control nodes and everything that holds state run on reserved machines, while mass computation runs on preemptible ones with auto-restart.
Common technical doubts before starting
Before starting, a tech lead checks not savings but predictability: how the task will behave when the machine is taken away, and what will have to be changed in the code. Below are five questions we discuss before the first run.
What happens to a task when a Spot instance is revoked?
The platform warns in advance, and the window for completion is about two minutes. On the signal, the handler finishes writing the intermediate result to storage and releases the queue lock, while a new instance picks up the unfinished fragment.
Will task code have to be rewritten?
Usually not. You need revocation signal handling, progress save points, and reruns that do not duplicate the result. These are local changes in launch and save logic; the business logic remains the same.
Is this compatible with the current stack and storage?
Launch uses the same images as on permanent machines, and there are no language restrictions. We separately check access to storage and secrets, network rules, and the availability of places to save intermediate data.
Who is responsible for execution control?
We handle configuration, queue policy, and monitoring; you handle the task logic. We keep control on three metrics: interruption share, time to restart, and share of lost fragments.
What if there are too many revocations and the run does not fit into the window?
Then we leave part of the load on permanent machines, and a rule based on the preemption share moves the rest. Critical steps run on guaranteed resources — the run fits within the deadline.
Let's discuss your task and we will come back with an estimate and plan
To start the conversation, a description of the load is enough: what launches the batch tasks, how many there are per day, what the maximum deadline is for each, and what happens when a virtual machine is revoked.
Logs from previous runs, the cluster configuration, and the list of services the tasks exchange data with reduce the discussion to a single meeting — they show where the load breaks under interruption.
Then the work proceeds step by step:
- We analyze the task profile: duration, sensitivity to losing intermediate results, and checkpoint requirements.
- We divide the load into what can safely go to preemptible machines and what must remain on permanent ones.
- We assemble the scope of work: queues, restart policies, log collection, and notifications about mass machine revocation.
- We agree on acceptance criteria — the share of jobs completed without manual intervention.
- We fix the rollout order: first one queue, then expansion to the rest.
- We review the result on a real stream and, if needed, adjust the machine selection rules.
- We hand over documentation and access to your team so that launching new queues does not require us.
Let's discuss the task — send a description of the load and what you already have. We will come back with an estimate and plan: which steps we take on, what will be required from your side, and when the first result will be visible.