Disaster recovery drill: what breaks if the plan has not been tested
A disaster recovery plan most often exists as a document: approved, signed, sitting in the knowledge base. It is usually tested piecemeal — restored a backup, brought up the database, confirmed the service responds.
As a whole, with traffic switching and traversal of all dependencies, the procedure has never been executed once.
The first thing that diverges in a real failure is the startup order.
In the document it looks linear, but in a running environment dependencies turn out to be mutual: the database will not come up without the authentication service, and that service will not start without the database.
The on-call engineer spends precious time figuring out something that is simply not in the procedure.
The second is access rights. Keys and service accounts were issued for the previous team composition, some have already been revoked or have expired, and the person who came on shift does not have rights to switch over.
This is discovered at the moment of the incident, when every minute of service downtime is visible to the business and customers.
Disaster recovery drills expose exactly these gaps: inconsistent startup order, expired access, unaccounted-for connections between services.
After a run-through, the procedure turns from a description of intentions into a sequence of actions that can be executed under pressure and without hints.
Below we break down how this is closed and what exactly is checked during a drill.
What regular testing of the recovery plan provides
A recovery plan that has never been executed in full is a hypothesis.
Drills turn it into a measurable value: we go through the scenario from the failure signal to service restoration and record the actual RTO and RPO — not the ones written in the document, but the ones obtained from measurement.
Testing of the recovery plan proceeds by failure classes, not by a single "general" scenario: loss of a node, failure of an entire zone, logical data corruption, human error, unavailability of an external service.
Each class has its own acceptable downtime limit and its own set of steps — restoring a node and rolling data back from a backup require different times and different decisions.
Regularity matters more than a one-time run: it removes variance. The first drill usually shows that actual recovery time is many times greater than the target — because of manual steps, waiting for access, and unclear role assignment.
By the third or fourth run, these steps are automated or formalized, and the plan is updated after each drill — this is also how service continuity practice works in ITIL.
The result is predictable recovery. Restore time stops being a surprise, variance between runs shrinks, dependence on a specific on-call engineer falls, and bottlenecks are visible before an incident rather than during one.
A failure class that once meant hours of downtime now follows a proven scenario with a known outcome.
Drill formats: tabletop run, partial and full failure
The formats differ in two things: how many people and how much time they take, and which failure classes they actually expose. A tabletop run follows the document — the team goes through the steps sequentially, clarifying roles and decision points.
This is a cheap way to catch discrepancies in the plan, outdated contacts, and ambiguous wording, but this format does not test a single real switchover.
A partial failure brings up one node — a standby environment, a replica, restoration from a copy — and leaves the rest of the system running. A full switchover tests the chain end to end: the order of operations, hidden dependencies between services, actual recovery time. An unannounced drill adds to this a check of the on-call staff and real response time.
| Format | When it fits | Limitations |
|---|---|---|
| Tabletop run | Testing the plan and roles after architecture changes, documentation regression | Does not reveal configuration errors or service unavailability: everything remains on paper |
| Partial failure | Testing individual nodes — standby environment, restoration from a copy | Adjacent services are not involved, dependencies between them remain unnoticed |
| Full switchover | Before major changes and according to procedure: confirmation of target recovery time | Requires moving the primary environment to standby and high team readiness |
| Unannounced drill | Checking on-call readiness and real response time | Risk of false triggers, requires a narrow perimeter and pre-agreed boundaries |
Choosing a format is choosing the risks you are willing to leave untested: a quarterly tabletop run and an annual full switchover close different scenarios.
Stages of conducting a drill: from audit to results review
A drill without a pre-fixed order turns into improvisation: the team closes problems in the order it discovered them, not in the order the business needs. We break down the drill stages so that each step ends with an artifact — from a dependency map to an updated procedure.
- Inventory and infrastructure audit. We gather the actual picture: which services are running, where they are hosted, who owns them, what data they store. We record discrepancies with documentation immediately — usually it is these that break recovery.
- Dependency map. We determine what depends on what and in what order to bring things up: storage, queues, integrations, access. Without this, recovery proceeds by guesswork and runs into "the service came up, but it is not serving data."
- Scenario and success criteria. Before the start, we agree on what counts as failure: target recovery time, acceptable data loss, responsibility zones. Then there will be no need to argue about the result after the fact.
- Backup verification. We look not at the presence of copies but at their restorability: restoration to an isolated environment, integrity verification, rollback to the previous snapshot. A copy that does not restore is not a backup.
- Scenario run-through with measurements. We keep time tracking for each step and an action log: who did what and when. Measurements show where minutes are actually lost, not where the team feels "everything is slow."
- Review of results and discrepancies. We compare the plan with the facts: which steps did not match, which assumptions were not confirmed, where we had to invent workarounds. Each discrepancy becomes a task with an owner.
- Plan update and handover. We reissue the procedure, checklists, and dependency diagram, and assign responsible people and the date of the next drill. The plan lives in documentation, not in one engineer's head.
After such a cycle, the disaster recovery plan stops being a formality: it has been tested on live infrastructure, and roles in the team are assigned without hints.
What the customer receives after the drill
A drill ends not with a presentation but with a working set: using it, the on-call shift restores the service without calls, messages, or searching for "the one who remembers." All documents are reconciled with the actual state of the infrastructure, so the discrepancy between the diagram and reality has already been recorded rather than discovered at the moment of an incident.
- Infrastructure and dependency diagrams: service topology, bottlenecks and failure points, traversal order when a site is lost.
- Configurations and backups: what is included in backups, at what frequency, where they are stored, and how integrity is verified before recovery.
- Recovery procedure: step-by-step order of actions with actual switchover times, not calculated ones.
- Access policies: who has rights to what in emergency mode, how extended rights are issued, and when they are revoked.
- Responsibility matrix and notification channels: who makes the switchover decision, who to wake first, and who confirms restoration.
- Drill report: scenario, timing, share of completed steps, and every discrepancy found with its cause.
- Discrepancy remediation plan: priorities, owners, and deadlines for each item in the report.
Documents go into the repository alongside configuration, not into personal folders: they are governed by the same change procedure as the infrastructure itself.
The date of the next test is set immediately — otherwise the set will again become outdated by the first real incident.
Case: how a drill exposed an unworkable switchover scenario
The customer was a service with a constant flow of orders and a standby environment in a second availability zone.
The recovery procedure had existed for more than a year but had never been run in full: only the bringing up of database copies on PostgreSQL was tested; it never reached traffic switching.
The drill was conducted on a business day, without warning the on-call shift, with the primary environment actually taken out of service.
The run exposed a discrepancy between the document and the system's real behavior.
Restoration to operation took 3 hours 40 minutes instead of the stated 30: an hour and a half went to manual approvals, another forty minutes to waiting for a lagging replica whose state no one had checked before switchover.
The step "transfer traffic" in the procedure came after application startup, so some records went to the already disconnected environment.
The order of operations was rewritten: first data, then applications, then traffic.
Before switchover — a mandatory check of replica lag with a threshold at which the operation stops; traffic is transferred through a managed load balancer with a single confirmation point instead of a chain of approvals. Manual steps became 6 instead of 31; the rest were covered by scripts with result verification.
Metrics before and after:
| Indicator | Before the drill | After the fixes |
|---|---|---|
| Restoration to operation | 3 h 40 min | 22 min |
| Manual steps | 31 | 6 |
| Record loss during switchover | some orders | 0 |
| Rollback to the original environment | not practiced | 9 min |
The failure class "switchover from memory, without checking replica state" was eliminated: the scenario is run in full during drills twice a year, and discrepancies between the document and reality are closed before the next run.
How a drill differs from a real switchover
A drill reproduces an incident on a separate environment: a copy of the production environment, databases brought up from backups, the startup order of services checked, and how the on-call shift follows the recovery instructions.
This provides honest testing boundaries — backup integrity, deployment time, completeness of documentation, availability of access for the shift. Everything that breaks at this stage breaks without risk of downtime for customers, so such findings are the cheapest.
Outside drill mode remains what the stand fundamentally does not reproduce: the real load profile, behavior of external integrations, state of caches and queues, discrepancy between the data schema version and application code.
Therefore, successful drills do not guarantee that a real switchover will go just as smoothly — they only remove controllable causes of failure.
Actual numbers are provided only by a production switchover: how long unavailability really lasts, how quickly external addresses are updated, how many customers remain on the old environment, what happens to unfinished operations and open sessions.
A drill shows a lower-bound estimate — what part of the time depends on the team — but does not close the risks created by live traffic.
Hence the division of goals: a drill checks the readiness of people and artifacts, while a planned switchover checks system readiness and data consistency between environments.
We conduct drills regularly, with rotation of on-call staff and review of every discrepancy against the instructions, and switchover — in an agreed window, with a rollback order and pre-described rollback criteria.
If a drill has never exposed a failed step or forgotten access, it is too soft for an emergency scenario.
Technical doubts before the first drills
Before the first drills, the same questions are asked: will the service go down, how often should this be repeated, should all systems be checked at once, and how many people should be pulled away from work. Below are short answers that usually remove doubts before the start.
Will the service have to be stopped during testing?
No. Copies are deployed to an isolated environment whose network does not intersect with production, and user traffic continues to go to the primary.
A consistent data snapshot is taken by a standard mechanism — for example, in PostgreSQL and analogues it does not require stopping writes. Only the window for reading from the primary storage is planned in advance.
How often should drills be repeated?
Critical services — once a quarter, the rest — once every six months. We launch an unscheduled run after a change in the backup scheme, a major release, or edits to the topology.
Frequency matters more than depth: a short check once a quarter is more useful than one long run once a year, because the team's skill fades.
Can we start with only some systems?
Yes, that is usually how it starts: we take two or three critical paths — authorization, payments, queue processing — and check recovery only for them. Coverage expands as findings are resolved, so drills do not turn into a multi-month project.
How many people are needed from the customer side?
Two or three: the service owner, an on-call engineer, and someone who has access to storage and management panels. Their role is to confirm access, align expectations on target recovery time, and evaluate the result. We take on the rest of the work.
Let's discuss the task: where to start and what to prepare
A drill begins not with launching a scenario but with inventory: which services we consider critical, where the boundaries between them run, who is responsible for switchover, and in what window it is acceptable to take the environment down.
These inputs determine which failure scenarios make sense to test and what level of stand isolation is needed so that the drill does not affect production.
With this data, we return with a timeline estimate, scope of work, and implementation plan — by stages, with points where the customer accepts the result. To start, the following is enough.
- A list of services and which of them is considered critical and which can wait until morning.
- Target recovery indicators: how quickly the environment must come up and what volume of data is acceptable to lose.
- Infrastructure and dependency diagram — even a sketch or outdated version; we will find and report discrepancies anyway.
- Owners responsible for services and the procedure for coordinating work in the drill window.
- Constraints: periods when stopping is not allowed, regulatory requirements, and internal procedures.
- Access for deploying an isolated stand — or confirmation that we provide the stand.
- Existing recovery procedures and checklists, if they already exist in any form.
Next, we fix the scenarios, run them on the stand, measure actual recovery time and discrepancies with target indicators, and deliver a report with timings for each step and a list of bottlenecks. Describe the task — we will return with an estimate and work plan.