Spot VM GCP: where batch processing breaks
Batch window on permanent nodes looks fine: nightly recalculation, report, export — and then a day of idle while hardware waits for next run.
Moving to Spot VM GCP removes idle time, but exposes second problem: batch processing tolerates preemptible machines only under one condition — job can die without losing result.
In practice condition is not met. Scheduler hands off task as whole, executor keeps state in memory and writes output as single file at end, and notification of node preemption arrives with window of minutes and does not always reach child processes.
Interruption in middle of such write yields either empty artifact or partially overwritten — and rerun accepts it as valid.
Loss of job result is more expensive than saved idle time. One failed executor means rerun of entire chain, including already computed steps, manual reconciliation of discrepancies in aggregates, and missed window until next cycle.
More dangerous when losses are not immediately visible: report goes out with a hole, and it is found later from complaint about incorrect numbers.
Next, we break down how this is closed: splitting work into replayable units, checkpoints in object storage, idempotent write by task key, and preemption configuration where node departure costs seconds, not an entire window.
What a pool of Spot machines gives batch processing
A pool of preemptible machines gives batch processing more parallel executors and predictable degradation instead of lost results.
The price is that notification of node release arrives in 30 seconds, and all calculation logic is built around making those seconds not cost a task recomputation.
Google Cloud publishes statistics on Spot node preemptions: on average, single-digit percentages of capacity per day, but during peak demand hours the share grows — so the pool is designed not for "usually no preemption" but for "there is always preemption."
Reducing the processing window
The duration of the window is determined not by pool size but by the speed at which it adapts to queue length.
We tie autoscaling of the machine group to queue depth, not CPU load: a batch of hundreds of thousands of independent tasks consumes free capacity in minutes, and after queue drains the pool shrinks to baseline size.
The window fits into two to three hours instead of eight, and idle capacity consumption between runs disappears.
Why node preemption no longer kills what has been computed
The failure class that must be eliminated looks like this: node is taken away midway through a step, the unfinished step and everything computed before it is lost, task starts over.
This is cured by splitting calculation into idempotent steps with checkpoints in object storage and correct handling of termination signal: upon notification, executor finishes writing intermediate result, releases task lease, and only then exits. Rerun continues from last checkpoint, and missing steps are caught up by steady pool.
The share of recomputed work drops from the whole step to the last minutes — preemption stops being an incident and becomes a routine event, kept under control by preemption share and queue depth on dashboard.
Resource allocation options: when each fits
The batch processing profile is almost never uniform: some jobs can be interrupted and restarted, while others hold state and coordinate others.
Therefore, choice between preemptible and permanent nodes is decided not by taste but by load profile — and often correct answer is their combination. Below are three schemes that are actually chosen between when deploying a pool in Google Cloud.
| Option | When it fits | Constraints |
|---|---|---|
| Only preemptible nodes (Spot VM) | Queues without hard deadline; tasks are idempotent, split into short portions, and can restart | Node is revoked with short warning; checkpoints, work splitting, and re-entry without side effects are required |
| Only permanent nodes | Jobs with promised deadline, in-memory state, stream computing, shared access to one resource | You pay for idle time between peaks; pool scaling is manual, peak keeps extra machines running continuously |
| Hybrid pool scheme | Core holds state and coordination, while mass peak tasks go to preemptible | More complex to manage: placement rules, queue separation, and image version control across groups are required |
Choosing mode for load profile
The decisive sign is what happens to job when node is revoked. If task is idempotent and cut into short portions, revocation costs only restart; if it writes to one shared file or keeps state in memory, revocation breaks result entirely.
The second sign is queue predictability: where promised deadline matters, preemptible nodes fail precisely at peak hour.
Hybrid scheme with core on permanent nodes
We move pool core to permanent nodes: scheduler, job coordinator, queue broker, storage gateway. Preemptible nodes connect as workers and live exactly as long as needed.
We calculate automatic pool scaling by queue length and number of in-flight tasks, not average load — otherwise peak spills into permanent fleet.
Pool implementation procedure: from audit to handover
Work proceeds in stages: each next step relies on data from previous one, so pool does not have to be redone after launch. Below is implementation procedure with points where we stop and agree on result with your team.
Load audit and preemption profile
First we capture actual picture: daily job schedule, duration of one task, sensitivity to rerun. Separately we collect preemption profile — how often cloud takes away Spot nodes and what happens to queue. Without this data, calculating number of nodes will be guesswork.
- Load audit — analyze queues, job types, and processing time requirements.
- Preemption profile — calculate what share of tasks tolerates node stop and how much time restart needs.
- Pool scheme — node composition, reserve, job distribution rules, intermediate result storage.
- Pool setup — configuration, node labeling, automatic restart of interrupted tasks, queue binding.
- Fault tolerance testing — deliberately shut down nodes and see how tasks return to queue and recover.
- Procedures and handover to operations — metrics, alerts, instructions for on-call shift, team training.
Acceptance checks before handover
Acceptance is not a report but a series of controlled failures: we mass-drain nodes and check that tasks are not lost and are not counted twice, and queue is processed within expected time. We hand over pool together with procedures, metrics, and clear action order for on-call shift.
Delivery composition: configurations, policies, procedures
Delivery for a pool on GCP is not only working batch processing but also a set of artifacts by which customer's team itself changes configuration, troubleshoots failures, and hands over on-call duty.
Everything is in customer's repository and updated together with code — without verbal agreements and knowledge held in one engineer's head.
- Pool configuration templates — parameters of machine set, queue-length scaling rules, preemption limits, and procedure for returning to regular machines during resource shortage.
- Node placement scheme — which zones and machine types are allocated to different job classes and how load is redistributed when platform takes away some machines.
- Access policies — service account roles, minimum required permissions, environment separation, and ban on manual pool edits during working hours.
- Job processing scheme — splitting queue into independent parts, rerun of unfinished fragments, requirements for repeatability of write operations.
- Response procedures — alert thresholds, on-call action order, criteria for switching processing to backup pool.
- Instructions for on-call shift — step-by-step scenarios: restarting processing, clearing stuck jobs, checking queue and pool state.
- Description of typical failures — "symptom → cause → action" analysis, collected from test results and first weeks of operation.
- Pool documentation — resource map, service owners, configuration change procedure, and decision log.
The kit is considered accepted when customer's on-call engineer goes through failure scenario using instructions without our prompting, and pool changes are made through same templates used in main environment.
Case: batch processing on preemptible nodes
Project: nightly recalculation of analytical data marts — tens of thousands of short independent jobs, result needed by start of workday. Pool was kept on permanent nodes sized for peak: at night they ran at limit, and during day they idled.
We moved batch processing to preemptible Google Cloud nodes: a Spot VM pool with autoscaling and distribution across multiple zones.
Before and after
- Processing window: 9–11 hours → 4–5 hours.
- Simultaneously disabled nodes: up to 40% of pool → 8–10% due to diversification across zones and machine types.
- Jobs for rerun: about one third → 2–3%.
The key change is not node type itself but readiness for their disappearance. Each step became idempotent, intermediate results are written to object storage as checkpoints, and node receives stop signal in advance and has time to correctly finish current portion.
Preemptible nodes in operation stopped being a risk: job rerun continues from last checkpoint, not from scratch.
"Before checkpoints appeared, half the night went to restarts — now it is a few percent of jobs, and they are not visible in morning report," — engineer who led implementation.
The recalculation window reliably fits into the night, and queue itself picks up freed nodes instead of waiting for a new pool.
How preemptible machines differ from permanent nodes
A permanent node is shut down by your signal: traffic is removed, connections are drained, process stops.
A preemptible machine is revoked by provider itself as soon as capacity is freed for regular instances — signal arrives in advance, but window for orderly completion is measured in tens of seconds.
Therefore modes are compared by behavior during revocation and by what job manages to save, not by cost of ownership.
| Criterion | Permanent nodes | Preemptible machines |
|---|---|---|
| Shutdown moment | by your decision | by provider decision |
| Window for completion | unlimited | tens of seconds after signal |
| Suitable load | user services, message brokers, state stores | batch jobs, recalculations, log parsing |
| Result protection | session and state on disk | checkpoints, external storage |
| Return to pool | manual provisioning and decommissioning | recreation from template |
Planning takeaway: one-off and recalculation tasks are given to preemptible nodes — everything that is cut into short independent portions and repeated without side effects.
Long transactions, state stores, and interactive services are not placed on them: revocation midway through such operation costs more than any gain from the mode.
Protection is built on three levels. Signal handler writes checkpoint and removes machine from pool, queue stores completed portion marker, and rerun is idempotent — recalculating same portion does not produce duplicates.
In a mixed pool, permanent nodes hold orchestrator and queue tail, while preemptible nodes process bulk of portions and are recreated from template by labels.
The mechanics of revocation, notification window, and automatic recreation rules are described by provider: Google Cloud publishes official description of preemptible machines — that is what handler timings are checked against.
Limitations, quotas, and typical pool failures
A pool on Spot nodes breaks not "on average" but through specific scenarios: node revocation in the middle of writing a part, hitting quota for machines in region, rerun of already processed parts after restart. Below are common questions that surface when analyzing pool limitations and typical failures.
What happens to a job when a node is shut down?
The handler catches stop signal, finishes writing current part atomically — restart takes only unfinished work. Parts that failed to close within window before shutdown are recalculated.
Can regular and Spot nodes be kept in one pool?
Yes, but better in separate pools or with node labels: otherwise scheduler will place a critical step on a node that is about to be revoked.
How to avoid reprocessing completed parts?
Idempotent key per part and a log of completed tasks: handler checks against it before start, and repeat pass costs only a read.
What load should not be moved to Spot?
Everything that holds state longer than node revocation: databases, message brokers, coordinators, and streaming data writes — they are placed on permanent nodes, while only recalculated and one-off portions of work are given to preemptible ones.