Kubernetes Backup with Velero – Setup, Pitfalls, and Timelines

Setting Up Kubernetes Backup with Velero – Experience and Timelines

Data loss in a StatefulSet after a failed deployment is a common scenario in production clusters. Without namespace-level backups, recovery takes hours: recreating the StatefulSet, searching for the last etcd snapshot, manually restoring PersistentVolumes. We've encountered this dozens of times and know how to avoid such scenarios. Velero closes this gap: namespace-level and resource-level backup with PV support via CSI snapshot or Kopia (formerly Restic). In a test restore to an isolated namespace, data recovers in 5–15 minutes – ten times faster than manual recovery, which takes 2–3 hours.

Why Velero Over etcd Snapshots?

etcd stores the cluster state, but restoring from it means rolling back the entire cluster. If you need to revert a single namespace or one PersistentVolume after a faulty deployment, etcd won't help. Velero provides granularity at the resource level and supports flexible retention policies.

Additionally, Velero integrates with cloud object storage (S3, GCS, Azure Blob), letting you store backups in a different region or account – a key element of disaster recovery. By our measurements, recovery time for a single application is reduced tenfold compared to manual processes. On a typical cluster with 100 PVCs and 50 workloads, a full backup takes about 20 minutes.

How to Avoid Common Mistakes When Setting Up Velero

The first issue: a CSI driver that doesn't support the VolumeSnapshot API. If such a driver hasn't registered a VolumeSnapshotClass, Velero silently skips volume data. Logs show Included Volumes: 0. Solution: enable the Node Agent (Kopia) for file-level backup or add the annotation velero.io/csi-volumesnapshot-class to the PVC.

The second mistake: an IAM policy for object storage without write permissions for objects. On AWS, it's common: the service account has access to the bucket but not to the /* path. Metadata backup passes, but data fails. The error in logs: AccessDenied on s3:PutObject for backups/*. Fix by adding a separate statement on arn:aws:s3:::velero-bucket/*.

The third: restoring to a cluster with a different StorageClass. Velero won't automatically rename the StorageClass. You need --namespace-mappings or a ConfigMap with a resource modifier for name mapping. Without this, PVCs will remain in Pending state.

Proper Backup Architecture

For a production cluster, the scheme looks like this: Object storage: a dedicated S3 bucket (or GCS/Azure Blob) in a different region or account.

Backup policy: two schedules via Schedule CR:

• Daily full backup of the entire cluster with a TTL of 30 days
• Hourly backup of critical namespaces (databases, stateful workloads) with a TTL of 7 days

---
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: hourly-databases
  namespace: velero
spec:
  schedule: "0 * * * *"
  template:
    includedNamespaces:
      - postgres
      - redis
    storageLocation: default
    volumeSnapshotLocations:
      - default
    ttl: 168h0m0s
---