Why AWS Spot Fleet interrupts batch processing
AWS Spot Fleet assembles a pool from free Spot capacity and holds it until the price rises above the threshold specified in the request.
As soon as the provider has fewer free machines, the capacity is revoked: the node is given a short window, about two minutes, and is taken away.
For a service that responds to requests, this is enough to drain connections, but for batch processing, where a single task is computed for tens of minutes, such a window gives nothing.
A task that was being computed on the node at the moment of revocation is interrupted midway.
Intermediate results reside in memory or on a local volume and disappear together with the instance if the processing does not write checkpoints to external storage — and by default it does not. The work does not continue from the interruption point but starts over.
Then the failure unfolds as a cascade.
Jobs have already been dispatched to worker nodes that no longer exist, the queue broker does not receive acknowledgment and holds them until the timeout expires — the stuck task queue grows, new jobs wait, and the scheduler meanwhile tries to replenish the pool and falls under the next revocation.
The loss of intermediate results is multiplied by repeated passes: part of the batch is computed two or three times, and the total processing time for the set drifts far beyond the estimate.
Hence the typical picture: the nightly recalculation does not finish by morning, reports arrive late, and the on-call engineer repeatedly tops up capacity manually. The problem is not Spot capacity itself, but that the job is not designed for its revocation. We'll break down how this is addressed.
What a managed Spot capacity pool provides
A managed Spot capacity pool in Amazon Web Services is not “turn on Spot and wait,” but a set of rules: how many node families and Availability Zones participate, by what strategy free capacity is acquired, and what to do at the moment of its revocation.
We fix the target share of Spot in the overall pool, and keep the remainder on regular nodes — the processing window does not stall entirely, even if Spot machines leave almost simultaneously.
The first measurable effect is the processing window completion time.
When the pool is tied to one zone and one node type, a significant part of the window is spent waiting for freed capacity; after distribution across several zones, the same queue volume is closed noticeably faster, and the processing schedule stops drifting toward morning.
The second is the share of successfully completed tasks. A node revoked in the middle of computation used to mean a restart from scratch or a corrupted result; now the task receives a revocation warning, takes a checkpoint, and hands the unfinished part to another machine.
The class of failures “sudden node revocation” simply disappears from the failure log: instead of analyzing nightly incidents, the team looks at one metric — the share of completed tasks.
“Previously, every morning began with analyzing which windows did not finish,” says our engineer. “Now the pool redistributes tasks on its own, and the report consistently shows the same success share — any discrepancy is immediately visible.”
Options for distributing Spot capacity in AWS
Spot capacity cannot be requested with a single pool in one Availability Zone: if the required instance type is scarce, the scheduler will interrupt some tasks, and the queue will stall.
The working scheme is to distribute the request across several zones and instance families at once: scarcity arrives in them at different times, so the aggregate capacity availability is higher than that of any single pool.
The choice between three approaches comes down to one question: what share of the work must reach completion, and what may be interrupted and restarted. Below is a comparison by this criterion.
| Option | When it fits | Limitations |
|---|---|---|
| Fully Spot pool | Interruptible processing: log parsing, index recalculation, model runs, batch jobs without a hard deadline | No capacity guarantee — during a demand spike, tasks wait in the queue. Interruption mid-step requires checkpoints and a restart that does not corrupt the result |
| Mixed pool with a baseline share of regular resources | There is a stage that must always run: queue coordinator, metadata database, result intake and assembly | The regular part sits idle during quiet hours; you need to ensure the Spot share grows and the baseline does not inflate “just in case” |
| Pool with a reserve across several Availability Zones | A hard completion window and a fault-tolerance requirement: if one zone is unavailable, computation continues in the others | More complex setup: the distribution strategy and zone weights are selected for the load profile, otherwise the reserve sits idle |
In practice, a mixed pool with a reserve across two or three zones is more often used: the baseline share maintains coordination and job completion, while Spot capacity takes the bulk of the work. Next, we'll break down how to lay this out across request parameters.
Implementation stages: from load audit to handover
The stage order is structured so that before the first run it becomes clear where the pool will hit limits and what will happen when the provider reclaims instances. We start not with configuration, but with calculating what the infrastructure must withstand.
-
Load audit. We collect actual consumption: task duration, peak concurrency, memory and disk requirements per unit of work. The result is the target pool size and headroom for overprovisioning according to an individual calculation Interruption profile.
We analyze the history of resource revocation by instance types and Availability Zones: how often the notification arrives, how much time remains before shutdown, and whether this coincides with demand peaks.
From this it is clear which instance families are acceptable and which will cause constant restarts.
-
Pool design. Spot Fleet configuration: several pools by types and zones, distribution strategy, share of stable instances as insurance against scarcity. We fix which tasks can be interrupted and which cannot.
-
Restart rules. We configure notification handling: saving the intermediate result, graceful process termination, and returning the task to the queue. We verify that a repeat run does not duplicate results.
-
Failure handling validation. We simulate instance revocation under load: whether processing fits within the allotted window, whether data is lost, and whether the pool recovers to the target size.
-
Handover of procedures. We deliver configuration as code, the diagnostics procedure, metrics and alert thresholds, and actions for capacity scarcity.
What the team retains after implementation
After implementation, the logic for working with interruptible instances remains in the repository and documentation, not in the head of the engineer who configured the pool. Below is a list of artifacts that the team receives and can verify during acceptance.
- Pool configurations — declarative files with target capacity, distribution across instance types and Availability Zones, selection strategy, and replacement rules. Changes go through review; the staging environment and production environment are assembled from a single source.
- Access policies — roles for nodes, the scheduler, and operators with the minimum required permissions, separate credentials for manual interventions, secrets moved out of configs and launch templates.
- Infrastructure diagrams — queue, pool, result storage, network boundaries, and the points where the instance interruption notification is processed, indicating the responsible component.
- Queue control rules — job priorities, concurrency limits, splitting large jobs into short portions with intermediate result saving, and the threshold at which the pool scales out instead of losing tasks.
- Failure procedure — a step-by-step order for mass instance withdrawal: how the unfinished portion is restarted, how the integrity of already computed results is checked, and who receives the notification.
- Monitoring — a dashboard and alerts for queue depth, share of successfully completed jobs, queue wait time, and interruption frequency.
- Documentation and handover — an on-call guide, the procedure for changing configurations, and analysis of real incidents on a call with the team.
The team can change the pool size, priorities, and thresholds independently, without coming back to us for every edit.
Case study: a batch pipeline that crashed every half hour
A Spot Fleet pipeline processed tens of thousands of tasks per day and relied on a pool of one machine type in one Availability Zone.
When the cloud reclaimed capacity, all nodes were evicted at once: the queue stalled, some tasks were lost midway through a step, and the rest were restarted manually.
The on-call engineer spent 25–40 minutes on recovery, and such outages occurred every half hour during peak hours.
What was rebuilt
The pool was spread across several machine types and zones — eviction of one group no longer brings down the entire pipeline.
Long jobs were split into short steps with state saving: an interrupted step returns to the queue and continues from the last checkpoint, not from the beginning.
The capacity selection rule considers both availability and price, and tasks sensitive to interruption go to the reserve pool.
| Metric | Before | After |
|---|---|---|
| Share of successful runs | 71% | 99.2% |
| Lost tasks per day | up to 400 | a few, and those from the retry queue |
| Restart time after eviction | 25–40 minutes manually | 2–3 minutes automatically |
| On-call interventions per day | more than 30 | 0–1 |
| Full pipeline run | 6–9 hours with downtime | about 4 hours without stops |
SeverTrans Logistik: “Previously, we sat on call by the monitor and restarted the queue by hand. Now we learn about a failure from the report, not from an adjacent department — the pipeline finishes on its own.”
How does a Spot pool differ from a mixed pool and autoscaling?
These are not three replacements for each other, but three levels of one scheme. A Spot pool in AWS is a capacity selection rule within a group, a mixed pool is the same group but with several instance types, and autoscaling is managing the number of nodes. They are often presented as alternatives, although in practice one is configured on top of another.
A pool with a single capacity selection strategy works as long as its instance type is available: upon revocation, the entire reserve drops at once, and the task queue stalls.
A mixed pool spreads the risk — several types and Availability Zones with weighted capacity, so revocation of some nodes does not bring down the cluster. The price for this is heterogeneous nodes, and the scheduler must pack tasks with different disk and memory requirements.
Autoscaling solves a different task: it keeps the number of workers aligned with demand based on the queue length metric, with step scaling and cooldown.
It does not cancel capacity revocation — the notification arrives within minutes, and a task without a checkpoint restarts from scratch. A Spot pool without a notification handler and graceful node draining simply turns the gain into rework.
The applicability boundary runs along the nature of the load. A batch with idempotent steps and checkpoints can live on Spot, while a database, a singleton service, and a long computation without state saving cannot.
Autoscaling does not help where the bottleneck is not the number of workers but an external dependency: a request limit to a source or storage speed — new nodes will only add conflicts for one resource.
Limitations of Spot pools: interruptions, quotas, capacity
Spot capacity costs less and there is usually plenty of it, but pools have physics that cannot be bypassed with settings.
Instance revocation comes with short notice, the quota per family limits simultaneous acquisition, and free capacity flows between Availability Zones during the day. Below are common doubts before moving batch processing to Spot pools.
What happens to tasks when an instance is revoked?
The revocation notification arrives in advance: the handler on the node puts it into shutdown mode, writes the step checkpoint to shared storage, and returns the task to the queue — another node picks it up.
This works if the step is idempotent and split into short parts: then one fragment of computation is lost, not the entire job.
Can interruptions be completely eliminated?
No — Spot capacity is always returned to the provider on demand; the share of revocations can only be reduced. Choosing pools with the largest remaining balance, distributing across several instance families and zones, and having headroom for peaks help.
For jobs with a hard deadline, a small on-demand reserve pool is kept, which turns on when Spot is scarce and covers the tail of the queue.
How should load be divided across Availability Zones?
The wider the pool across zones, the lower the chance that capacity scarcity in one zone will stop all processing. But the compute quota is calculated per region and per instance family, so a new zone hits the same limit — an increase is requested in advance. And the wider the geography, the more data exchange between zones when assembling results.
Let's discuss your load and interruption profile
To come back with an estimate and a plan, rather than generalities, we need a minimum of input data. Access and internal procedures are not required at this step — the load profile and deadline requirements are enough.
- Load profile: how many batch jobs are launched per day, average and peak concurrency, volume of processed data.
- Deadline requirements: whether there is a hard processing completion window or the remainder can be moved to the next window.
- Interruption profile: what share of tasks can be restarted, and where restart is unacceptable and a checkpoint is needed.
- Current state: whether Spot capacity is already used, which instance families and Availability Zones are involved.
- Data handling: where the task reads input data from and where it writes the result, and whether there is a dependency on shared storage.
- Constraints: account quotas, isolation requirements, maintenance windows, resource tagging rules.
- Acceptance criteria: by which metrics you will understand that the queue is draining and the cost of processing has stopped growing.
Describe the task — we will analyze how the load fits a Spot capacity pool and propose an order of work.
After the review, we will come back with an estimate and a plan: a capacity distribution scheme across zones, a strategy for returning interrupted tasks, implementation stages, and control points at each of them.