Data Backup and Disaster Recovery Services

Why is data backup often unreliable?

A backup exists. Last run — three weeks ago. Retention removed everything except the last two copies, one of which is unverified. The RTO in the documentation says "4 hours", but no one has checked since it was written a year ago on different infrastructure. This is not a hypothetical scenario. Order an audit of your current scheme — it takes two hours. We constantly encounter the illusion of protection: clients come with the task of setting up backups, but in reality, they get a scheme unsuitable for recovery. Our team of DevOps engineers designs and implements resilient schemes, from file backups to Kubernetes with Velero and hybrid environments with Veeam. With 10+ years of experience and over 50 DR projects, we know where failures hide.

Why are most backups unreliable?

Problem #1 — lack of verification. Set up a cron with restic backup and forget it. Six months later, it turns out the S3 bucket was deleted, the repository is damaged, or the password is lost. The second typical mistake — incorrect retention: a policy of "keep everything forever" leads to overflow, while "keep the last 7 days" doesn't protect against ransomware that encrypted data a month ago. The third — inconsistent copies. Backing up PostgreSQL via cp without pg_dump gives a snapshot unsuitable for recovery. We solve these problems comprehensively: audit of the current scheme, selection of a stack for the specific infrastructure, automation with notifications, and regular recovery testing.

According to the 3-2-1 rule (Wikipedia), the minimum at which a single failure doesn't destroy all copies is three copies on two different media, one of which is offsite. In modern implementation, this includes immutable storage to protect against malicious actors.

In practice, the scheme looks like this:

• Copy 1 — on NAS in the same data center (fast restore)
• Copy 2 — object storage in another region (S3, Backblaze B2, Wasabi) with retention policy
• Copy 3 — immutable storage with WORM (Write Once Read Many)

How to set up immutable backup?

Immutable copies are a key point in current conditions. S3 Object Lock in Compliance mode prevents object deletion even by the bucket owner for a specified period. This is the only protection if an attacker gains access to AWS credentials: they cannot delete backups. Without Object Lock, rotational ransomware can delete all copies in S3 with a single delete-objects API call.

# Enable S3 Object Lock on bucket (only at creation) aws s3api create-bucket \ --bucket my-backups-immutable \ --region eu-central-1 \ --create-bucket-configuration LocationConstraint=eu-central-1 aws s3api put-object-lock-configuration \ --bucket my-backups-immutable \ --object-lock-configuration \ '{"ObjectLockEnabled":"Enabled","Rule":{"DefaultRetention":{"Mode":"GOVERNANCE","Days":30}}}' 

According to the official AWS docs, Object Lock in Compliance mode prevents any user, including the root account, from deleting or overwriting objects for a fixed duration. GOVERNANCE mode allows users with s3:BypassGovernanceRetention to modify early. The choice depends on compliance requirements.

How do we implement 3-2-1?

We don't offer template solutions. For each project, we conduct an audit and select a combination of tools. Below is a real case: recovery after a ransomware attack on an infrastructure of 20 servers and three Kubernetes clusters.

Case: ransomware attack — recovery in 8 hours

A company reached out after ransomware encrypted all production and NAS. Local backups were also encrypted. Only an immutable copy in S3 with Object Lock, configured by us a month before the incident, survived. The scheme included:

• Daily restic backup of critical servers to S3 with 30-day retention
• Velero for Kubernetes: backups of all namespaces with a pre-hook for PostgreSQL
• S3 Object Lock in Compliance mode for 14 days (minimum period sufficient for detection)

After the attack:

1. Stopped infected VMs and clusters.
2. Deployed a new Velero cluster with the same StorageClass.
3. Restored PVC from snapshots — took 2 hours (5 TB of data).
4. Started pods, verified database consistency via pg_checksum.
5. Simultaneously restored file servers via restic restore to new instances.

Final RTO — 8 hours, RPO — 1 hour. Without the immutable copy, all data from the last month would be lost — an estimated $250,000 in data value. The actual recovery cost, including engineering time and cloud resources, was approximately $50,000. Regular recovery tests (quarterly) ensure the runbook works. Contact us to set up similar protection for your infrastructure.

Restic and BorgBackup: file-level backup for servers

Restic stores data in a repository as deduplicated, encrypted blocks. Incremental snapshots via chunk-level deduplication: if 1 MB changes in a 10 GB file, only the changed chunk is copied.

Typical Restic configuration with upload to S3:

# Initialize, backup, retention and check export RESTIC_REPOSITORY="s3:s3.eu-central-1.amazonaws.com/my-backups/server1" export RESTIC_PASSWORD="$(cat /etc/restic/password)" restic init # Backup with exclude patterns restic backup /var/lib/postgresql /etc /home \ --exclude="*.log" \ --exclude="/var/lib/postgresql/*/pg_wal" \ --tag "daily" --verbose # Retention: 7 daily, 4 weekly, 12 monthly restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune # Weekly verification (10% blocks) restic check --read-data-subset=10% 

BorgBackup is faster than Restic on large volumes (Cython vs Go), but the repository is tied to a single host and doesn't natively support S3. For local NAS backups, BorgBackup with borgmatic is an excellent choice.

Velero for Kubernetes: deep dive

Velero backs up Kubernetes resources via API + snapshots PVC via CSI. Installation via Helm with S3 backend:

velero install \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.8.0 \ --bucket velero-backups \ --secret-file ./credentials-velero \ --backup-location-config region=eu-central-1 \ --snapshot-location-config region=eu-central-1 \ --use-volume-snapshots=true 

credentials-velero — minimum permissions: s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket, ec2:CreateSnapshot, ec2:DeleteSnapshot, ec2:DescribeSnapshots.

Schedule with TTL and pre-hook for PostgreSQL:

apiVersion: velero.io/v1 kind: Schedule metadata: name: daily-production namespace: velero spec: schedule: "0 2 * * *" # every night at 2:00 template: ttl: 720h # 30 days includedNamespaces: - production - monitoring excludedResources: - events - events.events.k8s.io hooks: resources: - name: postgres-backup-hook includedNamespaces: - production labelSelector: matchLabels: app: postgresql pre: - exec: container: postgresql command: - /bin/sh - -c - pg_dump -U postgres mydb > /tmp/pre-backup.sql timeout: 5m 

Restoration pitfalls

Velero restore restores resources + PVC, but does not touch existing StorageClass and CRD. If the target cluster doesn't have a Standard StorageClass, restore will fail. The second pitfall: ImagePullBackOff due to a private registry. We recommend replicating critical images to a registry in another region.

Veeam for VMware and hybrid environments

Veeam remains the standard for VMware vSphere. Changed Block Tracking (CBT) makes incremental VM-level backups without an agent. For Windows Server, 1C, MS SQL — practically no alternative. In hybrid environments (VMware + cloud) — single console Veeam Backup & Replication.

Process

We implement end-to-end backup in 2–4 weeks, depending on complexity.

1. Audit (1–2 days): collect current scheme, check retention, verification, access.
2. Design (1–3 days): choose stack (Restic/Velero/Veeam), define RTO/RPO, storage architecture.
3. Implementation (3–10 days): configure agents, schedules, S3/Local storage, immutable policies.
4. Recovery testing (2–3 days): perform restore in an isolated environment, record actual time.
5. Documentation and training (1 day): deliver runbook, configs, access, contact scheme.

What's included

• Documentation: DR plan with step-by-step runbooks, backup map, responsible contacts.
• Access: creation of service accounts with minimal rights, key storage in Vault.
• Training: session for on-call engineers — how to perform recovery via runbook.
• Support: assistance during the first incident, consultations on retention settings.

Backup audit checklist

• All critical services included in the schedule.
• Retention policy matches RPO.
• At least one immutable copy exists.
• Test recovery performed in the last 3 months.
• Backup accesses protected with MFA.
• Notifications for backup errors configured.

Estimated timelines

Cost is calculated individually after infrastructure audit. Data loss can cost millions — investing in backup is an order of magnitude less. A typical downtime runs $10,000 per hour, making a DR plan a direct business decision.

Get a consultation from an engineer: contact us for an audit of your current setup. We will show weak points and offer a turnkey solution with a recovery guarantee. Schedule a call to discuss your specific infrastructure and RTO/RPO targets.