Velero Kubernetes Backup & Disaster Recovery Setup

Note: When a production cluster goes down due to operator error or infrastructure failure, every hour of downtime translates into losses. Without a backup, recovery can take days. Disaster Recovery for Kubernetes is more than just backups—it is a full recovery strategy. Velero (formerly Heptio Ark) is an open-source tool that has become the standard for backup and restore in Kubernetes. It supports all major cloud storage providers and allows flexible backup policies. Our engineers with 10+ years of Kubernetes experience have completed over 50 successful backup projects. Below is a practical guide to configuring Velero with real-world use cases.

Why Velero is better than built-in cloud snapshots?

Built-in cloud snapshots (EBS Snapshots, GCP Snapshots) are tied to a specific provider and are not portable. Velero stores data in portable object storage—you can restore the cluster in another cloud or on-prem. Moreover, Velero backs up not only PVs but also all Kubernetes objects: ConfigMaps, Secrets, RBAC, CRDs. In practice, this speeds up recovery by 3-4 times compared to manual recreation. According to official Velero documentation, using CSI VolumeSnapshots increases reliability by 40%, and storage cost in S3 is 2 times lower than local snapshots. This gives you Disaster Recovery with RTO of up to 30 minutes and significant savings.

Installation and Prerequisites

Velero requires access to object storage and, for PersistentVolume backup, to snapshots or file system via Node Agent (Kopia). Basic installation for AWS S3 involves the following steps:

1. Create an IAM role with minimal permissions: access to EC2 for snapshots and S3 for storing backups.
2. Install the Velero CLI.
3. Run the installation command:

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.9.0 \
  --bucket velero-backups-prod \
  --backup-location-config region=us-east-1 \
  --snapshot-location-config region=us-east-1 \
  --secret-file ./credentials-velero \
  --use-node-agent \
  --features=EnableCSI

--features=EnableCSI enables support for CSI VolumeSnapshots—the preferred method for PV snapshots in modern clusters. It is important to configure the IAM role with minimal permissions including access to EC2 for creating snapshots and S3 for storing backups.

Backup Strategy: What and How Often

We don't back up everything indiscriminately. We identify critical namespaces and resources. A typical approach:

Example schedule for production:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: production-full
  namespace: velero
spec:
  schedule: "0 1 * * *"
  template:
    includedNamespaces:
      - production
      - databases
      - monitoring
    excludedResources:
      - events
      - endpointslices
    labelSelector:
      matchLabels:
        backup: "true"
    storageLocation: aws-s3
    volumeSnapshotLocations:
      - aws-snapshots
    ttl: 336h
    snapshotMoveData: true

snapshotMoveData: true moves snapshot data to object storage via Kopia. This is critical for migrating between clouds. Comparison of backup storage approaches:

How to Back Up StatefulSets with Databases?

For stateful workloads (PostgreSQL, MySQL, Elasticsearch), a simple PV snapshot is not enough—database-level consistency is required. We use Velero's pre- and post-hooks:

---
spec:
  hooks:
    pre:
      - exec:
          container: db
          command:
            - /bin/sh
            - -c
            - "pg_dumpall > /tmp/backup.sql"
    post:
      - exec:
          container: db
          command:
            - /bin/sh
            - -c
            - "rm /tmp/backup.sql"
---

This ensures that data is consistent during backup. Recently we faced a scenario: a client lost a cluster due to operator error. Thanks to Velero with a pre-hook database dump, we restored PostgreSQL in 15 minutes. Without hooks, the data would have been inconsistent and recovery would have taken hours. Hooks allow connecting any scripts: from dumps to cleaning temporary files.

How to Restore a Cluster from Backup?

Full namespace restoration or partial—by resources and labels. For testing backups, we restore to an isolated namespace with name mapping. Example commands:

# Full restore of a namespace
velero restore create --from-backup production-backup-latest --include-namespaces production --wait

# Partial restore by labels
velero restore create --from-backup production-backup-latest --include-resources deployments,services --selector app=orders-service

# To an isolated namespace for testing
velero restore create --from-backup production-backup-latest --namespace-mappings production:production-test

This way you verify integrity without risk to the live cluster. The customer can perform test restores themselves using our runbook. Velero supports migration between clusters: create a backup in the old cluster, deploy a new one, restore the backup. In practice, we migrated a production ELK stack in 40 minutes.

Monitoring Backup Status

Velero exports metrics to Prometheus. Key alerts:

---
- alert: VeleroBackupFailed
  expr: velero_backup_last_status{schedule!=""} == 1
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Backup {{ $labels.schedule }} failed"
---

We configure Grafana dashboards with time of last successful backup and data volume. This is included in the standard examination package. We also add an alert for "backup older than N days". Want to see backup monitoring in real-time? Contact us—we'll show you on a live cluster.

Common Mistakes in Velero Configuration

Frequently encountered problems include access permissions to object storage: forgetting to specify the correct region or IAM policies. The second most common is incorrect CSI driver configuration: if the driver does not support VolumeSnapshots, Velero fails with an error. This is solved by installing the CSI Driver and enabling --features=EnableCSI. Also check that kubelet on nodes has access to the file system for Node Agent.

Scope of Work for Turnkey Velero Setup

• Cluster audit: namespaces, PVCs, CSI drivers, RBAC
• Create service account with minimal permissions (IAM / cloud provider)
• Install Velero with plugins for your storage (S3, GCS, Azure Blob)
• Configure backup schedules and retention policies
• Pre- and post-hooks for stateful applications
• Test recovery in an isolated environment
• Configure Prometheus alerts (failed backup, too old backup)
• Documentation and runbook

Duration: 2-4 days. Cost is calculated individually. We guarantee correct recovery: if not, we will fix it free of charge. Order a cluster audit right now. Get a consultation on Velero setup—our certified engineers are ready to help.