Velero Kubernetes backup: where recovery breaks
Backups are taken on schedule, reports are green, copies sit in object storage.
But during an incident, recovery completes without a single error—and the cluster comes up "empty": pods crash on startup, applications cannot find their data, and some objects are simply missing.
The reason is that Velero by default does not copy everything needed for a working state, and the recovery order does not follow the dependency order.
The first things to go missing are resources that are not in the application manifests: custom resource definitions and their controllers, secrets and configuration parameters, access control bindings, webhooks, and volume plumbing.
By object count, the backup looks complete, but cluster recovery after a failure yields not a single working link between them—and this is an incomplete backup that is discovered only in a live environment.
The second break point is backup consistency.
A volume snapshot is taken at the storage level, while part of the application state lives in memory and in the transaction log: without first stopping writes, the snapshot and the data diverge, and after rollback we get a consistent file system with inconsistent contents. The application starts, the database opens—and goes into integrity errors.
The third is order. Objects are applied in waves, and if custom types, secrets, or operators come after the applications that depend on them, controllers create resources into the void.
We will look at how this is addressed: what to include in the backup, in what sequence to restore, and how to verify that the backup actually brings up a working cluster, not just formally completes successfully.
What managed cluster backup provides
Backup stops being a just-in-case insurance policy and becomes a managed process with a measurable recovery time objective.
Schedules capture the state of resources and volumes to object storage, and recovery is performed from a specific point: a separate namespace or the entire cluster.
The recovery time objective is set in advance—tens of minutes for a typical scenario instead of hours of manual assembly.
An entire class of failures that previously meant state loss is covered: corruption of the cluster state store, accidental deletion of a namespace together with volumes, a failed update of resource definitions, and a node failure with local data.
In each of these cases, recovery comes from the last verified backup, not from attempts to reconstruct the configuration from memory.
Manual work is taken off the on-call shift: Velero creates backups on schedule, applies the retention policy, and on the same schedule verifies recovery in a separate namespace.
The on-call engineer does not search for a fresh backup or assemble resources manually—they run a proven scenario and see the result in a report.
Predictability comes from regular verification: you can see how long recovery took, which resources did not come up, and what lives outside the cluster—certificates, secrets, the state of external systems.
Service downtime becomes a planned window rather than an incident, and the share of successful drill recoveries is a metric that shows the protection actually works.
Velero, snapshots, and replication: when each fits
Protecting cluster state splits into two different tasks: quickly restore the workload after a failure and move it to another site. A volume snapshot covers the first but knows nothing about API objects—manifests, secrets, role bindings, and configurations.
Velero works specifically with cluster resources and can complement them with volume data backup, but it is slower to roll back than a snapshot.
Below are the backup options that are actually chosen between, and their limitations, which are usually remembered at the recovery stage.
| Option | When it fits | Limitations |
|---|---|---|
| Volume snapshots (CSI or cloud tools) | Cluster in one zone, disks support snapshots, need rollback in minutes | Does not transfer API object state, tied to provider and zone |
| Velero backups with file-based volume data collection | Different sites and clouds, need workload migration | Slower on large volumes, agent on nodes, databases require a consistent stop |
| Replication between clusters | Need low recovery time and service continuity | Does not protect against logical errors: data deletion is replicated just the same |
| Incremental backups | Large volumes, limited backup window | Dependency chain: all segments are needed for recovery |
In practice, the approaches are combined: a snapshot provides fast rollback, Velero provides migration and long-term storage, and replication covers a site failure. What to take as the foundation is determined by the recovery time you promise the business.
Implementation stages: from storage audit to handover
Implementation starts not with installation but with an inventory: which namespaces are critical, which volumes need to be snapshotted consistently, and which are restored from external systems.
Without this picture, schedules are made "for everything in a row," storage grows, and no one has tested rollback.
- Audit of workloads and recovery objectives. We classify resources by criticality and record two numbers for each group: acceptable data loss and acceptable downtime. They set the depth of schedules and determine what goes into backups.
- Backup scheme. We choose the target object storage, container layout, and retention rules: how many backups to keep and when to clean up old ones. Separately, we determine how to snapshot persistent volumes—with disk snapshots or a file-based method.
- Schedule configuration. We describe Velero schedules declaratively, separate them by criticality, and supplement them with scenarios for consistently capturing database state. The configuration is stored in a version control system and goes through review as code.
- Access isolation and control. Accounts with the minimum sufficient permissions, backup encryption, and notifications about failed jobs—a missed backup is visible the same day, not during an incident.
- Verification by recovery. We deploy backups in a separate test environment, measure rollback time, and verify data integrity. We feed discrepancies back into the scheme: adjust the schedule, the volume snapshot order, or the set of resources.
- Documentation handover. We hand over schemes, configuration files, the rollback procedure, and instructions for the on-call engineer, and walk the customer's team through the process. After that, the procedure lives as a regular operational process.
What your team has after implementation
Work is considered delivered not when the backup schedule is running, but when the on-call shift can restore the cluster without the person who configured it.
Therefore, along with the working configuration, the team receives a package of accesses, schemes, and procedures that guide step-by-step recovery.
- Backup configurations — Velero schedules, templates, include and exclude resource lists, and retention periods for recovery points.
- Backup scheme — a map: which namespaces, disks, and objects go into the backup, which storage the data goes to, and how full and incremental backups differ.
- Access policies and roles for storage — separate accounts for writing backups and for recovery, the minimum required permissions, and key rotation procedure.
- Recovery procedure — a step-by-step scenario with branching points: choosing a point, bringing up a separate namespace or the entire cluster, and checking the result.
- Instructions for the on-call shift — what to do during a failure, how to check the freshness of the last backup, when to escalate, and whom to involve.
- Regular verification procedure — how often test recovery is performed in a separate environment, what counts as a successful result, and where the report is stored.
- Monitoring and alerts — a run log, alerts for failed backups and for recovery time overruns.
Every artifact is verified in a training recovery: if, according to the instructions, the backup does not come up in the test environment, the document is revised.
Case study: recovery in minutes instead of a day
Before implementation, cluster recovery was a manual operation: etcd snapshots were taken on schedule, but volumes and objects inside the cluster were not included in the backup at all.
When, after a failed controller update, the production environment needed to be brought up again, engineers assembled it piece by piece from old descriptions—downtime stretched to almost a day.
No backup verification had been done before, so no one could say whether the archive contained anything suitable for recovery.
What was changed
Backup and recovery were moved to Velero: schedules were split across namespaces with different intervals—more frequent for critical services, less frequent for test ones.
An additional agent backs up volumes at the file level, so the archive includes not only object definitions but the data itself. Once a week, a backup is automatically deployed in a separate namespace: verification no longer depends on whether the on-call engineer remembers it.
What changed in numbers
Recovery time for a typical service dropped from a day to 15–20 minutes, and the acceptable data loss window—from several hours to 15 minutes on critical environments. The share of successful automatic verifications exceeded 90%, and each takes minutes without human involvement.
Rollback stopped being an event: an engineer brings up the needed data slice with one command and receives a report on exactly what was restored, while schedule intervals are selected so that archive volume grows predictably.
What to do if the cluster is already deployed and split
The cluster is already running—and that is no reason to stop it for backup. Velero is installed inside the cluster as a controller and communicates with it only through the API, while storing backups in object storage outside the cluster.
Applications, operators, access rules, and build pipelines remain untouched—only schedule and retention policy definitions are added.
Compatibility with the current scheme is ensured at the object selection level: you can back up by namespaces, by labels, or by specific resource types. For stateful applications, pre- and post-commands are added that pause writes and take a consistent snapshot.
If the platform already has a disk snapshot mechanism, it is connected as is; if not, file-based collection works, and there is no need to rebuild the storage.
Multiple environments are separated by storage areas: each has its own set of credentials and its own area, so that short retention policies on test environments do not evict production environment backups.
The management instance can be shared across all environments or run separately in each one: a shared one is easier to maintain, while a separate one isolates failures and access rights.
The phased transition starts with a pilot in one environment, then recovery is verified in a separate namespace—there you can see whether data and resource links come up in full.
After that, schedules are enabled on the remaining environments one by one, with monitoring of backup size and creation time. There is no downtime: backup runs in parallel with production traffic, and verification recovery does not affect production services.
Technical doubts and limitations of the approach
Backups are not made for a checkbox but for recovery verification. Below are the objections most often raised before implementing Velero: how complete the backup is, how the process affects the production environment, and where the limits of applicability end.
Will the backup be consistent if the snapshot coincided with a write to the database?
A volume snapshot captures the state of the disk, but not the order of operations in the application. For databases and queues, the state at the moment of the snapshot can be inconsistent—hooks are added to the configuration that flush buffers or temporarily stop writes.
Backup completeness depends not on the tool, but on whether rules have been described for every stateful component.
Will the production environment degrade during backup?
The main cost is reading the volume and uploading to object storage. The snapshot is taken by the storage itself, so the application does not notice the process, and the transfer runs in the background with limits on the number of streams and bandwidth.
Backup frequency is set by schedule: incremental snapshots once a day, full ones less often and outside peak windows.
Who is responsible if recovery does not work?
The tool returns what was put into it: cluster objects and volumes. Secrets in third-party storage, DNS records, load balancers, and state outside the cluster do not go into the backup—that is the customer's area.
We describe policies, hooks, and schedules and verify recovery in a separate environment; the customer confirms the list of systems and the order in which they are brought up.
Where does the approach stop working?
On clusters with thousands of volumes and a high rate of change, a full pass becomes expensive in time. Where loss of the latest transactions is unacceptable, backups do not replace streaming replication.
Recovery is possible at the namespace, cluster, or individual volume level, and the target time is measured in hours.
Let's discuss your task and come back with a plan
To come back with the scope of work and timelines rather than general words, we need your input data.
Send the cluster configuration and the list of namespaces, mark the services that store state, and indicate where the backups currently reside—in object storage or on disks next to the cluster.
Separately, we need target metrics: acceptable downtime and acceptable data loss for each group of services. Both the scope of work and the order of stages depend on them—what we cover first and what can be deferred.
Then we put together an implementation plan:
- an analysis of how backups are arranged now—using the cluster's built-in tools or via Velero—and at which steps they stop working
- resource inventory: what is backed up in full, what is restored in parts, and what requires a separate rule
- storage scheme: schedule, version lifetimes, and protection of the backups themselves from deletion
- recovery verification in a test environment—at least one service from each group
- a procedure for the on-call shift: how to start recovery and by what signs to tell that it succeeded
- acceptance criteria by which you can verify the result without our involvement
The output is an estimate of timelines by stage, the team's scope of work, and a list of what will be required from your side: access, a window for test runs, and the person responsible for infrastructure.
Some data may be missing—come with what you have, and we will clarify the rest in conversation. Describe the task, and we will come back with an estimate and an implementation plan.