Website Disaster Recovery Plan: Where Outages Catch You Off Guard
Outages rarely come from where you expect them.
The server may be duplicated, the load balancer configured, yet the site still goes down: disk space for logs ran out, a certificate was renewed and didn't get picked up, the DNS provider served the zone with an error, a database migration locked a table.
Each of these failure points looks like a trifle, but they account for the bulk of downtime — not "the server burned down," but everyday failures that no one ever rehearsed.
The cost of such a failure has two components. The first is downtime: while engineers figure out what exactly went down and hunt for access credentials, visitors see an error, and inquiries and orders don't come through.
The second is data loss: if a backup is taken once a day and sits on the same disk as the database, a rollback throws the site back, and some transactions vanish irretrievably.
Recovery after a failure without a plan almost always proceeds by trial and error: first they guess, then they fix, then they discover the backup is corrupted.
A disaster recovery plan turns this process from improvisation into a procedure. It is fixed in advance which nodes are critical, how often copies are taken and where they are stored, who brings the service up and in what sequence, and within what time they must finish.
Then recovery after a failure is executing familiar steps against a timer, not searching for someone to blame and for passwords.
Let's break down how this is covered: which website failure points should be addressed first, how to choose the target downtime and the depth of data recovery, and what must be in the document so it can be used during an incident, not after.
What a well-built disaster recovery plan delivers
A disaster recovery plan moves continuity from the realm of hopes into the realm of numbers.
Two parameters define everything else: the recovery time objective — how long a service can be down before losses become irreversible — and the acceptable data loss — how much transaction volume may be lost during a failover. Until these values are fixed, any talk of fault tolerance remains a matter of taste.
What changes once the parameters are set
When the recovery time objective is defined, the architecture adapts to it, not the other way around.
A database replica in another availability zone, cache warm-up, a traffic failover procedure — every decision is tested with the question "do we fit within the window." Backup stops being a ritual: the frequency of snapshots and the retention of transaction logs are derived from the acceptable data loss rather than chosen "by eye."
Next comes controlled data loss. Instead of "there's a backup, but it's unclear how fresh it is," the team knows: during a failure, no more than a certain write interval is lost, and that interval is confirmed by a drill failover.
This eliminates an entire class of failures — the ones where recovery formally succeeded but the data turned out to be inconsistent.
Verification instead of assumptions
Netflix brought this principle to the level of an engineering norm: fault tolerance is confirmed by regular drills, not by diagrams.
We follow the same logic — a plan is considered working only after failover has been performed on a staging environment, measured by time, and documented. A procedure that has never been run does not work during a real incident.
The result is measurable: recovery time is reduced from hours to minutes, data loss is limited to a defined interval, and the class of failures known as "we recovered, but not the right thing" falls out of practice. Continuity stops depending on who exactly is on duty at the moment of the failure.
Approaches to redundancy: cold, warm, hot
Choosing a redundancy mode is a trade-off between recovery time and the cost of continuously maintaining a second site.
Cold standby holds only copies of data and configurations: the infrastructure is brought up from scratch according to a runbook, so recovery time is measured in hours, but idle resources consume almost nothing.
Warm standby keeps a minimal contour running at all times — the database is replicated, application images are ready — and recovery comes down to scaling and switching traffic.
Hot standby is an active contour that takes a share of the traffic, with synchronous or near-synchronous replication. Switching takes minutes or seconds, but requires constantly doubling capacity and discipline in schema migrations.
In practice, modes are combined: the critical contour is kept hot, analytics and internal services are kept warm, and the archive is kept cold.
| Option | When it fits | Limitations |
|---|---|---|
| Cold standby | Non-critical services, a long recovery window is acceptable | Recovery takes hours, requires proven runbooks and regular drills |
| Warm standby | A primary site with an acceptable window of tens of minutes | Requires data replication and scaling automation |
| Hot standby | A payment contour, services with a strict service level agreement | More expensive to maintain, more complex schema and migration synchronization |
The recovery mode is fixed in advance: who makes the decision to switch, based on which metrics, how data is rolled back. Without this, even hot standby turns into hours of downtime.
Stages of implementing a disaster recovery plan
Implementing a disaster recovery plan is not the purchase of a ready-made solution, but a sequence of engineering steps where each subsequent one relies on the data from the previous one.
Below is what the process looks like from the first infrastructure assessment to handing over the procedures to your team.
-
Infrastructure audit. We build a map of services: what is hosted where, what dependencies exist between the database, cache, queues, and external integrations, and where the points of failure are.
We record the current metrics — downtime during recent incidents, the presence and age of backups, who checks them and how. The output is a list of critical nodes with an assessment of the acceptable downtime for each.
-
Defining target metrics. For each service, we agree on how much time it may be unavailable and how much data may be lost. This defines the architecture: what is hot-standby, what is cold-standby, and what is restored from a backup manually.
-
Designing the redundancy scheme. For the chosen metrics, we build a scheme: database replication, application replicas in different zones, backups with the required frequency and retention depth, fault isolation so that the failure of one node does not drag down the rest.
-
Implementation and automation. We set up replication, backup schedules, monitoring, and failure alerts. We bring recovery scripts to the point of launching with a single command — manual steps during an incident are performed slowly and with errors.
-
Recovery testing. We conduct drills: we deliberately take a node out of service and restore it according to the procedure, measuring the actual time. Drills reveal what is not visible on the diagrams — incomplete backups, forgotten dependencies, incorrect access permissions.
-
Action procedures. We describe who does what during an incident: the notification order, the person responsible for the failover decision, the sequence of steps, and the criteria for returning to normal operation. The procedure should be readable in minutes, not studied at the moment of failure.
-
Handover and support. We hand over to your team the documentation, access credentials, and drill results, and we agree on the frequency of repeat tests. Infrastructure changes — a plan without regular drills becomes outdated within a few months.
Describe your task — we will come back with an estimate and a work plan following this process.
Output artifacts: configurations, diagrams, runbooks
After implementing the disaster recovery plan, the customer is left not with a presentation, but with a working set: the settings that bring the system up, and the documents the on-call shift acts on.
Everything is stored in the repository alongside the infrastructure code — when the environment changes, the plan is updated too, otherwise within six months the document diverges from reality and recovery happens from memory.
What the set includes
- Redundancy configurations — descriptions of replicas, backup schedules, and retention rules: how many copies we keep, how often, and for how long. Separately — integrity check settings, so that a backup doesn't turn out to be empty at the crucial moment.
- Recovery diagrams — a map of components with the target recovery time and acceptable data loss for each: database, file storage, queues, external integrations. It shows what gets restored first and what can wait.
- Access policies — who has rights to what in emergency mode, where keys and credentials are stored, and how they are rotated. Roles are separated so that one person cannot both break and restore without a trace.
- Action runbook — a step-by-step procedure for the on-call engineer: how to record the failure, whom to notify, at what point to declare an incident and switch traffic to the standby circuit.
- Notification matrix — who receives the signal and through which channel, who makes the switchover decision, who is responsible for communication with the customer until the incident is closed.
- Drill log — records of scheduled recovery runs with actual recovery times and a list of discrepancies that were fixed in the configurations as a result.
- Rollback instructions — how to return to the previous version of the application or data schema if recovery did not go according to plan.
We hand the set over to the customer's repository with a change history: it is visible who edited the plan, when, and why. From there it can be maintained by their own team — the documents are written so that they can be read without the author.
Case study: reducing recovery time by four times
The customer — a chain of retail outlets with an online store and a customer account area — came with a typical pain point: the site "sometimes goes down," but no one knows how long it will take to come back up.
Incidents were handled manually: the on-duty engineer looked for what was running on which server, restored the database from the latest copy, and only then brought the application up.
Backups were taken once a day, stored on the same disk as the production data, and no one remembered to test recovery.
The customer already had GitLab, so the disaster recovery plan was built around it, without disrupting familiar development processes.
What it was before
A full recovery after a database failure took an average of 6 hours 40 minutes, and that was without counting diagnostic time. The recovery point was up to 24 hours: orders and catalog edits from the past day were lost.
Once, the disk with the copies failed along with the primary one — recovery was done from a week-old export.
What was changed
- Database backups were switched to continuous mode with a change log: the recovery point was reduced to 5 minutes.
- Copies are stored in a separate storage in another region, accessible only to a service account.
- A recovery playbook was put together: bringing up the application, database, and load balancer with a single command, without manually searching for configurations.
- Once a month we run a training recovery on a separate environment and compare the metrics against the targets.
What it became
A full recovery now fits within 1 hour 35 minutes — four times faster. The recovery point is 5 minutes instead of a day: only a handful of orders are lost, not a full workday.
The training runs revealed another problem — some checks didn't cover queue recovery, and it was fixed before a real failure occurred.
The main result isn't in the numbers but in predictability: the team now has an answer to the question "how long will it take us to come back up," and it's backed by practice, not a promise.
How does a disaster recovery plan differ from backup?
Backup answers the question "what do we have," while a recovery plan answers the question "what do we do and in what order." A backup is an artifact: copies of the database, files, and configurations stored in a separate repository on a schedule.
By itself, it is passive: the copies exist, but no one knows who will bring them up, where they will be deployed, and what counts as success. A plan is a set of procedures with roles, priorities, and decision points that turns a set of copies into a working service.
The difference shows up at the moment of failure.
If an application server goes down, a backup will not tell you whether to restore the database or the load balancer first, who holds access to the production environment, and how many minutes it takes for management to get involved.
The plan describes the sequence: assess the scope, shift traffic to the standby environment, bring up data from the last consistent point, verify key scenarios, and only then return users. Each step has an owner and a readiness criterion.
Another difference lies in the target metrics. For copies, frequency and retention depth are measured; for the plan, recovery time and acceptable data loss are measured.
These are different metrics, and they are achieved by different means: backup frequency through scheduling and retention, recovery speed through drills and a pre-prepared environment.
Copies without a plan give false peace of mind: the data is intact, but the service is down for hours because the sequence of actions has not been practiced.
Finally, the plan lives and changes: a new integration, a change of vendor, a move to another environment — all of this requires revisiting roles and contacts. Backup is updated automatically; procedures only through revision.
That is why a mature practice looks like this: copies are tested for recoverability, and the plan is tested for executability, up to and including a training bring-up of the service in the standby environment.
Common technical concerns
A disaster recovery plan is verified not by reading the document, but by drills.
Once a quarter we run a controlled failure: we shut down a node, disable the primary availability zone, and watch how long it takes for traffic to move to the standby circuit, whether the databases come up from the latest copy, and whether the data state converges.
The discrepancy between the estimated and actual recovery time is the main signal that the plan is outdated.
Integrating the plan with the current circuit is a separate effort. The redundancy scheme must account for real dependencies: queues, cache, external services, DNS, and certificates. If even one component fails to switch according to the scenario, recovery stalls on it.
That is why we identify failure points and define the order in which services come up, rather than relying on the general slogan "we restore from backup."
Maintaining the plan is a regular review, not a one-time setup. The architecture changes, services are added, copy retention policies are updated — the plan must follow them. Below are the questions most often asked at the start.
How often should the recovery plan be tested?
At least once a quarter — a full drill with a real switchover, and monthly — a spot check of individual scenarios: database recovery, node startup, release rollback. Between drills we monitor that copies are created and checked for integrity. This cadence catches degradation before it shows up in a real incident.
What will happen to the current infrastructure during the plan integration?
We break nothing: the plan is layered on top of the existing circuit. We describe what already works, add the missing redundancy mechanisms, and verify that the switchover does not conflict with current deployment processes. Changes are introduced in stages, with the ability to roll back at every step.
Who is responsible for maintaining the plan after launch?
The plan is handed over to your team along with the documentation and the drill regulations. We can stay on for support: keeping the scenarios up to date, updating them as the architecture changes, and running checks. You choose the format — full support or periodic reviews.
Let's discuss the task and come back with an estimate and a plan
We gather the inputs about your environment: where the site is hosted, which external services are tied to it, how backups are set up, and who has access to the panels. That is enough to assess the current recovery time and points of failure. Then we prepare a proposal — with the scope of work, timelines, and what you get as a result.
What the next steps include:
- a 40-60 minute call: we go over the architecture, failure scenarios, and downtime requirements;
- an audit of the current state: where backups are stored, whether they are tested, whether replication and monitoring are in place;
- defining target metrics: how much data loss is acceptable and how quickly the service needs to be brought back up;
- a draft disaster recovery plan: roles, sequence of actions, notification channels;
- an estimate and schedule: stages, responsible parties, acceptance milestones;
- approval and kickoff: after the plan is approved, we move to implementation.
Based on the discussion, you receive a document with an estimate and a work plan — it can also be used for internal approval. Describe the task in any convenient format: the current stack, what is already set up, what incidents have occurred before. We will come back with a proposal and timelines.
