Website Recovery: Runbook, Backups, and Timelines

Website Recovery After Failure

You discovered that the site is not opening and SSH access is lost? Or the last backup turned out to be corrupt? In 10 years we have restored more than 50 projects: from WordPress hacks to complete disk array failure. The key to fast recovery is not heroism, but a prepared runbook and regularly tested backups. Without this, recovery can take days instead of hours. In this article you will find ready-made diagnostic and recovery scripts, as well as tips for automating backups. Our experience confirms: proper preparation reduces downtime by 5 times.

Main Failure Scenarios and Diagnostics

Diagnostics is the first step. Here is a standard set of commands we run when the site is unavailable:

# Check services and logs
systemctl status nginx php8.2-fpm mysql
journalctl -u nginx -n 100 --no-pager
tail -100 /var/log/php8.2-fpm.log

# Disk and memory diagnostics
df -h
du -sh /var/log/* | sort -rh | head -10
dmesg | grep -i "out of memory"
free -m

# If database is not responding
sudo systemctl restart postgresql
tail -50 /var/log/postgresql/postgresql-14-main.log

Typical causes: full disk (clear logs, temporary Docker files), insufficient memory (add swap), corrupted database tables. After diagnostics, choose a recovery scenario. If no fresh backup is available, don't panic — often a rollback to the last stable dump from cloud storage helps. Make sure the DBMS version matches, otherwise recovery will fail.

Why Is Backup Testing Critical?

30% of backups we check in production turn out to be useless: archive corrupted, dump incomplete, DBMS version mismatch. We test recovery on a staging environment before relying on a backup. Testing takes 1-2 hours but can save days of downtime. Regular tests are the only way to ensure your Disaster Recovery Plan works. According to Wikipedia, testing is a mandatory part of the plan.

We recommend a combination: weekly full + daily incremental. Recovery from an incremental backup takes 2-3 times longer than from a full one, but saves space.

Recovery from Backup: Step-by-Step (Our Case)

Let's break down a real situation: a client updated a plugin, the site went down, and the backup was two weeks old. Here's what we did:

# Identify the last working backup
aws s3 ls s3://my-backups/database/ | tail -5

# Download
aws s3 cp s3://my-backups/database/mysite_<backup_date>.dump.gz /tmp/

# Create a new DB (leave old one untouched — test first)
createdb mysite_restored
gunzip < /tmp/mysite_<backup_date>.dump.gz | pg_restore -d mysite_restored --no-owner

# Check integrity
psql -d mysite_restored -c "SELECT COUNT(*) FROM users;"
psql -d mysite_restored -c "SELECT MAX(created_at) FROM orders;"

# Switch application to restored DB (change DB_NAME in .env)
# If all good — rename database
# ALTER DATABASE mysite RENAME TO mysite_broken;
# ALTER DATABASE mysite_restored RENAME TO mysite;

After recovery, we configured daily incremental backups to Amazon S3 and automated testing via pgBackRest.

How to Automate Backup Testing?

Manual testing is expensive. We use a script that once a week deploys a backup to an isolated staging environment and checks data integrity. If the test fails, the team gets an alert in Slack. Automating testing reduces the risk of discovering a broken backup during an emergency by 80%.

File Recovery and Code Rollback

If only files are damaged (e.g., media), use cloud synchronization:

aws s3 sync s3://my-backups/files/ /var/www/mysite/storage/app/public/ \
--exact-timestamps
chown -R www-data:www-data /var/www/mysite/storage/
chmod -R 755 /var/www/mysite/storage/

For code rollback, use Git or Docker:

# Via Git
git log --oneline -10
git checkout <commit-hash>
# or
git revert <bad-commit>

# Via Docker
docker pull myregistry/myapp:previous-tag
docker stop myapp
docker run -d --name myapp myregistry/myapp:previous-tag

How to Prepare a Runbook for Fast Recovery?

A runbook is a step-by-step guide for the team. Without it, you waste up to 3 times more time in a stressful situation. We include in the runbook:

• Service check scripts
• Recovery scenarios for each failure type
• Contacts of responsible persons and notification channels
• Post-mortem template

Example runbook table:

What's Included in the Recovery Service

• Audit of current backup scheme — we check what is saved and restored.
• Runbook preparation — detailed instructions for your stack.
• Recovery testing on a staging environment — we prove the process works.
• Backup optimization — incremental, off-site, encrypted.
• Team training — we hold a session to practice failure scenario.
• Post-recovery support — we guarantee stability for 30 days.

Timelines and Cost

Website recovery with a ready runbook — from 1 to 3 days depending on complexity. Full audit and Disaster Recovery Plan setup — from 5 to 10 business days. Cost is calculated individually after the audit.

We are confident in our quality — we guarantee results. Assess your recovery plan now: contact us for an audit. Get a readiness checklist and recommendations. Order recovery today — our engineers will contact you within an hour.

Our numbers: 10+ years of experience, 50+ restored projects, average recovery time 45 minutes. Recovery from a tested backup is 5 times faster than from an untested one.