Disaster Recovery Runbook Development for Critical Systems

We develop Disaster Recovery Runbooks for critical systems — not as theory, but as a tool that actually works at 3 AM when the business is down and the director is calling every 10 minutes. In such conditions, only one format helps: specific step → specific command → expected result → what to do if something goes wrong. For more on the concept, see Disaster Recovery.

Generic phrases like "restore the database from the last backup" are not a runbook. That is a script for improvisation, which in a stressful situation almost guarantees an error. Our runbooks, on the contrary, contain ready-made commands: from pg_ctl promote to Ansible commands for switching the application.

Why is a DR Runbook critical?

Without disaster recovery documentation, every incident turns into chaos: engineers try to remember what was done last time, search for credentials in different chats, and the recovery time (MTTR) grows. A good runbook reduces MTTR by 3–5 times — this is confirmed by our projects. In one case, a runbook helped restore a 50 GB database in 40 minutes instead of the usual 4 hours. The client noted: "After implementing the runbook, we reduced MTTR from 6 hours to 40 minutes." Over 5 years of work, we have conducted more than 100 DR drills, and 95% of them were successful with an RTO of less than 15 minutes for critical services. Average downtime is reduced by 70%, allowing savings of up to $900–1.3k per year.

How we develop DR Runbooks

We do not write template instructions. Each runbook is created for a specific infrastructure: stack, versions, topology, cloud provider. Our engineers with seven years of DevOps experience work through all failure scenarios — from a single node failure to the loss of an entire availability zone. Testing has shown that runbooks written to our standard ensure recovery twice as fast as typical instructions. Order a runbook development and reduce your recovery time.

Work process:

1. Infrastructure audit — we collect the dependency diagram, identify critical services, clarify RPO/RTO.
2. Scenario design — we describe typical failures (DB master crash, storage unavailability, configuration loss).
3. Runbook development — for each scenario, we write atomic steps with commands, checks, and expected results.
4. Validation on a DR drill — we simulate a real failure, fix errors in the documentation.
5. Handover and training — we upload the runbook to Git (Markdown), connect CI/CD for automatic PDF generation, train on-call engineers.

What is included in deliverables?

• DR Runbook for each critical service (up to 5 systems in the standard set).
• Failback procedure — return to the original architecture after recovery.
• Dependency matrix and escalation contacts.
• Integration with PagerDuty or Opsgenie.
• Runbook review after 3 months included.

Runbook structure for one service

Each critical service gets its own runbook. Structure:

Service overview

• Purpose: what the service does, who depends on it
• Owner: name + on-call engineer contact
• RPO / RTO: specific numbers
• Dependencies: list of services without which this one does not work

Incident indicators

Specific symptoms that indicate this runbook is needed, not another:

• alert: PostgreSQLDown{job="prod-db"} in Alertmanager
• HTTP 502 from nginx on /api/v1/health for more than 5 minutes
• CPU on primary DB > 95% for more than 10 minutes

Diagnostics (before starting DR procedures)

Checks to perform before activating DR — maybe the problem can be solved simpler:

# Check PostgreSQL status
systemctl status postgresql-15
pg_isready -h localhost -p 5432

# View active connections
psql -U postgres -c "SELECT count(*), state FROM pg_stat_activity GROUP BY state;"

# Check disk space
df -h /var/lib/postgresql

# Check logs
journalctl -u postgresql-15 -n 100 --no-pager
tail -100 /var/log/postgresql/postgresql-15-main.log

DR procedure steps

This is the main part. Each step is an atomic action with a result check.

# STEP 1: Confirm that primary is unavailable (not a false alarm)
# Execute from jump-host or from DR network
ping -c 3 10.0.1.10 # primary DB IP
ssh -o ConnectTimeout=5 [email protected] "echo ok"
# Expected result in case of actual failure: connection timeout

# STEP 2: Check DR replica status
ssh [email protected] "pg_controldata /var/lib/postgresql/data | grep 'state'"
# Expected result: "Database cluster state: in archive recovery"

# STEP 3: Promote DR replica (IRREVERSIBLE OPERATION — ensure primary is dead)
ssh [email protected] pg_ctl promote -D /var/lib/postgresql/data
# Expected result after 30 sec:
# pg_controldata /var/lib/postgresql/data | grep state
# "Database cluster state: in production"

# STEP 4: Verify that promote succeeded
psql -h 10.10.1.5 -U postgres -c "\dt"
psql -h 10.10.1.5 -U postgres -c "SELECT count(*) FROM users;"
# Expected result: tables exist, row count is reasonable

# STEP 5: Switch application to DR database
# Update DATABASE_URL in application configuration
export DR_DB_HOST=10.10.1.5
ansible-playbook -i inventory/dr \
  playbooks/update-db-config.yml \
  -e "db_host=$DR_DB_HOST"

# STEP 6: Restart application
ansible-playbook -i inventory/dr \
  playbooks/restart-app.yml

# STEP 7: Verify functionality
curl -f https://app-dr.company.com/api/v1/health
# Expected result: HTTP 200, {"status":"ok","db":"connected"}

Rollback / failback procedure

After the primary is restored, everything needs to be switched back. This is a separate procedure that is often forgotten to document:

# After primary is restored:
# 1. Set up primary as a replica of DR (pg_basebackup)
# 2. Wait for synchronization
# 3. Switch application back
# 4. Demote DR to replica

Contacts and escalation

How to store and update a runbook?

A runbook should live next to the code — in a Git repository, not in Confluence, where it cannot be accessed if Confluence goes down along with production. Markdown in Git, export to PDF in case of complete system outage. We guarantee that the runbook will be synchronized with the infrastructure: we use a GitOps approach where any configuration change triggers a CI pipeline to verify documentation freshness.

Every 3 months — review: check that commands are still valid, IP addresses have not changed, contacts are correct. The best way to check a runbook is to conduct a DR drill using it. We organize such drills with simulation of real failures — contact us.

Timeline estimates

Cost is calculated individually after an infrastructure audit. Get a consultation — we will assess the scope and prepare a commercial offer. Contact us to discuss details and request runbook examples for your systems.