Disaster Recovery Plan: RPO, RTO, and Automation

Disaster Recovery Plan: RPO, RTO, and Automation

Imagine this: primary datacenter goes down at night, clients can't place orders. Without a proven DR plan, every incident is a lottery. We've seen cases where a supposed DR plan turned out to be a year outdated and runbooks didn't work. That's why we build procedures that actually execute, with automated steps and regular drills. Our engineers with 10+ years of DevOps and SRE experience ensure your DR plan fires on time when it matters most. Automating DR can save up to 80% of recovery time compared to manual processes.

Why RPO and RTO Must Be Aligned with Business?

Before designing DR, two metrics need to be agreed upon with the business, not guessed by engineers. RPO (Recovery Point Objective) is the maximum acceptable data loss in time. If RPO = 1 hour, you can lose at most one hour of transactions. This dictates backup frequency and the need for synchronous replication. RTO (Recovery Time Objective) is the maximum acceptable downtime. RTO = 4 hours means services must be restored within 4 hours. This dictates the DR site model (hot/warm/cold) and the degree of automation in failover.

A common mistake: engineers assign RPO = 0 and RTO = 15 minutes without realizing this requires real-time synchronous replication and hot standby — essentially doubling infrastructure cost. Business might be fine with RPO = 24 hours and RTO = 8 hours at significantly lower cost. Aligning these numbers is the first step to a realistic plan.

How to Choose a DR Site Model?

Architecture choice depends on RPO/RTO and budget. For minimal RTO (< 1 minute), you need active hot standby with synchronous replication. For RTO up to 4 hours, warm standby with asynchronous replication suffices. If the business can tolerate a day of downtime, cold standby with recovery from backups works. Compare options in the table:

Inventory of Critical Systems

A DR plan starts with classifying systems by criticality tier:

For each Tier 1 system, record dependencies (what fails with it), owner, recovery procedure, and responsible contacts.

Architectural Solutions for RPO/RTO

PostgreSQL with replication and Patroni. For RPO < 5 minutes and RTO < 10 minutes, use streaming replication with automatic failover via Patroni:

# patroni.yml — config for DR cluster
scope: pg-production
namespace: /service/
restapi:
  listen: 0.0.0.0:8008
  connect_address: 10.0.1.10:8008
etcd3:
  hosts:
    - 10.0.0.1:2379
    - 10.0.0.2:2379
    - 10.0.0.3:2379
bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576 # 1 MB — replication lag
postgresql:
  use_pg_rewind: true
  use_slots: true
  parameters:
    wal_level: replica
    hot_standby: "on"
    max_wal_senders: 10
    max_replication_slots: 10
    synchronous_commit: "on"
    synchronous_standby_names: "ANY 1 (replica-dr)"
postgresql:
  listen: 0.0.0.0:5432
  connect_address: 10.0.1.10:5432
  data_dir: /var/lib/postgresql/data
  pg_hba:
    - host replication replicator 10.0.0.0/24 md5
    - host all all 0.0.0.0/0 md5

Terraform for infrastructure as code. DR infrastructure should be described in Terraform and stored in Git. Terraform is 10x faster than manual deployment for repeatable runs — the only way to guarantee predictable recovery a year later:

# dr-site/main.tf
module "dr_vpc" {
  source = "../modules/vpc"
  cidr_block = "10.10.0.0/16"
  availability_zones = ["eu-central-1a", "eu-central-1b"]
  tags = {
    Environment = "dr"
    Tier = "1"
  }
}

module "dr_rds" {
  source = "../modules/rds"
  identifier = "prod-dr-replica"
  source_db_identifier = "prod-primary"
  # Read replica in DR region — base for failover
  replicate_source_db = "arn:aws:rds:us-east-1:123456789:db:prod-primary"
}

Nginx and load balancers. DNS failover via Route53 or Cloudflare: when the primary endpoint is unhealthy, traffic switches to DR. For RTO < 15 minutes, active health checks with TTL ≤ 60 seconds are required.

Structure of the DR Plan: What the Document Should Contain

A good DR plan is not an architecture description. It's step-by-step instructions for the person getting a call at 3 AM after an alert. Structure:

1. DR activation triggers — specific conditions that start the DR procedure. Not "significant problems" but "primary site unreachable for more than 15 minutes according to Prometheus Alertmanager alert DatabaseDown severity:critical."
2. Responsibility matrix — who decides to activate DR, who executes, who notifies the business.
3. Step-by-step runbooks — separate runbook for each Tier 1 service with commands, expected results, and verification points.
4. Failback procedure — returning to primary after recovery. Often forgotten, then DR site becomes the new "primary" without documentation.
5. Communication plan — notification templates for clients, SLA commitments, vendor contacts (cloud, hosting, CDN).

Why Regular Drills Are Critical?

Without regular drills, the DR plan quickly becomes obsolete. Even a perfectly written document doesn't guarantee the team can execute it under stress. We recommend drills at least once per quarter for Tier 1, monthly for the most critical services. Drills reveal outdated runbooks, infrastructure changes, and knowledge gaps. After each drill, the plan is updated.

What's Included in the Work

When we develop a turnkey DR plan, we provide:

• Audit of current infrastructure and tier classification
• RPO/RTO alignment with business and documentation
• DR site architecture design (hot/warm/cold selection)
• Replication setup, failover automation, and monitoring
• Runbooks for all critical services
• Procedure documentation (triggers, responsibility matrix, failback)
• Initial drills and plan adjustments based on results
• Team training and access handover
• Support during the warranty period post-deployment

Plan Development Process

Work starts with a workshop with key stakeholders: audit of critical systems, gathering RPO/RTO requirements, inventory of dependencies. Typical duration is 1-2 days for simple infrastructure, a week for complex.

Next step is gap analysis: what is already in place for DR and what is missing. Often backups exist but no restore procedure, or replication is configured but never tested.

After that, we develop and document procedures, configure or enhance technical components, and conduct the first drill.

Timeline Estimates

Cost is estimated individually after infrastructure audit. To evaluate your project, contact us — we'll analyze your infrastructure and propose the optimal solution. Get a consultation from an engineer with 10+ years of experience in DevOps and SRE.

Official Patroni documentation