Kubernetes CronJob: configuration, best practices, and monitoring

Kubernetes CronJob: why jobs don't run on schedule

CronJob looks like the simplest entity in a cluster: schedule, image, command — then the environment starts the job itself.

That is why schedule failure is noticed late, when a missed job run or duplicate routine task surfaces in reports: the same export, reconciliation, or mailing runs twice.

The scheduler does not wake a job exactly on schedule — it checks time in a loop. If the control plane restarted, a node was unavailable, or the schedule is set more frequently than the check interval, the launch window passes by.

The startingDeadlineSeconds parameter decides how much delay is still acceptable, and concurrencyPolicy: Forbid prevents a new instance from starting on top of a running one.

Duplication most often comes from the Allow policy and a long job: the next run begins while the previous one is still writing to the same table.

The same effect comes from several control plane replicas without an elected leader and manual "catch-ups" after an incident.

Stuck processes in the cluster are a separate class. Without activeDeadlineSeconds and a retry limit, a job can hang for days: the instance does not leave Pending due to quotas, the image does not pull, the container does not terminate on signal. Such a process holds resources and blocks the next one under a forbid policy.

Finally, lost results: a successful instance is deleted, logs go with it, and history stores only the last few runs. Hence the picture "the job ran but there is no result." We analyze how each failure class is closed by configuration, not by on-call duty.

Managed scheduled jobs: effect for the team

Routine jobs stop being a risk point when the schedule, limits, and restart policy are described in Kubernetes manifests, not restored manually after every failure. The effect is measurable: share of lost runs, recovery time, and number of manual on-call interventions.

Reducing manual restarts

Execution timeout and retry policy keep a job from hanging and blocking the next window. The cause of failure is visible in run history, so a typical failure is closed by editing a manifest, not by restarting and hoping. The number of manual investigations drops from several per week to isolated ones per quarter.

Predictable cluster load

Schedules are spaced apart in time, resource requests and limits are set for jobs, and parallel runs are limited. Peaks stop falling during working hours: nightly jobs do not compete with services, and the cluster does not go into a service queue.

Fast recovery after failure

Returning to work is rolling back a manifest and rerunning the missed window, not restoring from backup. Recovery time drops from tens of minutes to single digits: the procedure is known in advance and does not depend on who is on call.

A missed window stops being invisible — the discrepancy is visible in monitoring the same day. The schedule becomes a reproducible artifact: it can be handed to another team or deployed in a new environment without verbal instructions.

Kubernetes CronJob: launch options and their limitations

A routine job in a cluster is rarely alone: report export, key rotation, cleanup of stale records, aggregate recalculation.

CronJob covers the schedule but does not answer two questions — how many copies of a job are allowed simultaneously and what to do if the previous pass has not finished. Hence four modes are actually chosen among.

It is not about "which tool is better" but where the lock lives and who is responsible for a missed run. Applicability boundaries of each option:

In practice, modes are mixed: the schedule and launch are left to the scheduler, while locking and overlap protection are moved into the handler itself — then moving a job outside the cluster does not break the logic.

Track not the fact "the pod ran" but the result: the metric of successful passes, the time of the last completion, and the number of missed slots.

Implementation order: from schedule audit to handover

Implementation starts not with manifests but with inventory: cron on servers, timers, the scheduler in the build system, and jobs already living in the cluster.

Already here you can see which of them compete for resources, where schedules overlap in time, and which failures no one noticed for months.

1. Audit current schedules. Compile jobs into a table: frequency, actual duration, data dependencies, owner. Mark duplicates and those running more often than needed.
2. Job launch scheme. For each one, fix the concurrency policy, maximum execution time, resource limits, and retention depth for the history of completed runs.
3. Configure retries. Set the number of retries, the delay between them, and the container restart policy; separately decide what to do with a job that did not fit the allotted window.
4. Fault tolerance testing. Kill a pod, disable an external dependency, saturate the node with load — the job must restart, not create duplicates, and not block the next one on schedule.
5. Observability. Configure metrics for successful and failed runs, missed window alerting, and a log by which the cause of failure is visible without manual investigation.
6. Handover of support. Hand over the schedule scheme, configurations, typical failure instructions, and the change procedure.

Each stage ends with a verifiable artifact: an updated schedule, configuration, or report on the fault test run. This removes the main risk — when jobs live on their own and you learn about a failure from an adjacent team, not from an alert.

Artifact kit: manifests, policies, procedures

Work ends not with a "configured cluster" but with a kit that can be handed to another team and reproduced from scratch.

Everything lies in a repository with change history: manifests are applied declaratively, differences between environments are described explicitly, not edited by hand on production.

• Job configuration description — manifests with explicit concurrencyPolicy, backoffLimit, resource limits, and retention policy for completed pods; separately recorded why exactly these values were chosen.
• Access policies and roles — service accounts per job, permissions within namespace boundaries, network policies, secrets from an external store instead of values in manifests.
• Environment scheme — separation of development, pre-production, and production environments, per-environment value overrides, change promotion and rollback procedure.
• Monitoring and alerts — run metrics and their durations, alerting rules for a missed schedule, repeated failures, and stuck jobs.
• Failure handling procedure — what an on-call engineer does when a job fails, how to pause the schedule via suspend and rerun a missed launch without duplicates.
• Support documentation — purpose of each schedule, dependencies on external systems, typical operations, and points requiring human intervention.

We hand over the kit together with a walkthrough: the customer's engineer goes through it and repeats deployment in a test environment, after which support can be maintained in-house.

Practice: how the share of lost runs was reduced

The client came with a platform team where about 120 jobs ran on schedule: nightly exports, balance reconciliations, condition recalculations, mailings. Some runs disappeared — and they learned about it from business complaints, not from a monitoring signal.

What metrics showed before

The share of lost runs held at 8–9% per day. About half of them were silent misses: the job did not start or did not finish, but no alert fired. Average reaction time to failure was several hours, because the only signal was a call from the department waiting for the report.

What changed

Consolidated schedules into one control layer on Kubernetes: set startingDeadlineSeconds so a job raised too late would not overlap the next window, concurrencyPolicy: Forbid for overlapping jobs, and backoffLimit with activeDeadlineSeconds instead of infinite retries.

The alert was set not on failure but on absence of successful completion within the window, plus TTL for completed job history. The jobs themselves were made idempotent — a retry after failure no longer duplicated charges and sends.

What became

The share of lost runs dropped from 8–9% to 0.2–0.4%, and reaction time to failure from several hours to 1–2 minutes. The failure class "missed run no one learned about" is closed: the missed window itself became an event of job execution control.

How scheduled jobs differ from work queues?

The boundary runs along the source of truth. A scheduled job is started by time: the scheduler itself decides the moment has arrived and starts the process without checking whether there is work for it.

A queue is started by an event: until a new task arrives, the handler waits and does not consume resources.

Hence different failure classes. For a schedule — overlaps and misses: two runs of the same recalculation overlap, a long pass consumes the next one, and a scheduler stop remains unnoticed.

This is closed by concurrency policy, an acceptable lateness window, and run history retention showing both success and failure. For a queue — redelivery and "stuck" tasks that failed and did not return to work without a separate check.

In both, idempotency of operations is required.

A repeated nightly recalculation or redelivered event must not charge twice, send a second email, or create a duplicate order — an idempotency key and reconciliation with the current state in the database solve this regardless of the launch method.

The choice is simple: a routine without an external trigger — reports, reference data sync, cleanup and archiving — goes on a schedule. An event stream — orders, webhooks, files — goes through a queue, where the number of workers grows with queue length.

A working compromise is a hybrid: a job in Kubernetes does not perform the work itself but only fills the queue, while processing and retries are handled by separate workers.

Technical questions asked before start

Before start, the same doubts almost always surface: how the schedule will survive a cluster upgrade, what to do with a long-running job, and where double runs come from. Below are four questions that most often affect the placement scheme.

What to do if a job hangs?

We catch hangs with the job's own timeout: activeDeadlineSeconds forcibly removes the pod, backoffLimit limits retries, ttlSecondsAfterFinished removes completed objects. Additionally, monitor the time of the last successful run — it grows before the timeout fires.

Will a job survive a cluster upgrade?

During an upgrade, nodes are drained and the pod is interrupted. Short jobs survive this via retries; long ones are moved to maintenance-free windows and made idempotent so a retry does not corrupt data.

Keep the job description in the current version — otherwise the upgraded cluster will not accept it.

How to limit execution time?

A time limit prevents a job from hanging forever and overlapping the next run. If the work does not fit the schedule interval, move it to a separate loop or lengthen the period — otherwise some runs will be skipped.

Let's discuss the task: review schedules and failure points

To come back with an estimate and work plan for schedules, we need a snapshot of the environment, not the phrasing "periodic tasks sometimes don't fire." Describe the task — we will compare it with the scheme and propose a configuration that can be implemented without stopping services.

What to send as inputs for the estimate:

• current environment scheme: clusters and environments, how launch parameters differ between them;
• list of periodic jobs: schedule, typical duration, dependencies between steps, acceptable delay window;
• how launch is arranged now: scheduler inside the application, system timer, or manual script;
• requirements for behavior after failure: what happens if the same job is executed twice;
• how long to store the history of successful and unsuccessful attempts and whether rollback to the previous build is needed;
• existing alerting rules and known failure points: where signals arrive and who reads them outside working hours;
• environment constraints: network policies, access to external resources, maintenance windows.

As output you will receive a task map with risk assessment, a staged work plan for schedules, and a set of checks proving that the launch actually happened and the result was recorded. Send the inputs — we will come back with an estimate and plan.