Runbook development: why runbooks break during an incident
An incident response runbook is usually written on a calm day — when the service is responding, metrics are normal, and no one is counting the minutes.
The result is a document “just in case”: it passes approval, lands in the knowledge base, and quietly goes stale there, because infrastructure changes faster than anyone remembers to make edits.
The real test comes at the moment of failure, and then it turns out that the service recovery instruction describes a scheme that no longer exists.
The first thing that breaks is escalation. It says “notify the responsible person,” but does not say who that is on the night shift, from what degradation threshold the second level engages, and what to do if the first one does not respond within fifteen minutes.
The on-call engineer spends the first minutes not on diagnostics, but on figuring out whom to call at all.
Second — steps that have lost touch with reality. An outdated command or node address in the instruction is worse than no instruction: the engineer follows it, gets an unexpected result, and begins to doubt the rest of the text. Documentation for the on-call shift is devalued entirely as soon as one step stops working.
Third — concentration of knowledge. When the sequence of actions is kept in the heads of two people, any vacation or move to another team turns a routine failure into a long outage.
Formally, the document exists; in practice, recovery depends on whether a specific engineer is available.
This is exactly where Runbook development begins: the runbook is built from real failure scenarios, tied to code, and tested in drills — we break down how this is covered.
What changes in the on-call shift's work after implementation
The first measurable shift is time to fault localization. Before scenarios are implemented, the on-call engineer spends from half an hour to an hour on the question “what exactly broke”: gathering symptoms from logs, metrics, and messages from adjacent teams.
After — the first checks are documented, and the diagnosis is made in 5-10 minutes: the engineer does not search but confirms or rejects hypotheses in order.
The second effect is reduced service recovery time. A unified procedure for failure removes variability between shifts: steps, branching points, and the criteria “we go no further, we escalate” are fixed, so the night shift acts the same as the day shift. On typical failures, this yields a twofold reduction in time to return the service to normal.
Third — the share of incidents closed according to the described scenario: on mature sets this is 70-85% of requests, the rest is escalation, but already with gathered context, not from scratch.
Runbook development proceeds from practice: first we fix the procedure, then we package repeating steps into scripts and checks.
The reduction in repeat incidents follows from the fact that the scenario covers not only the symptom: next to the step “bring the service up” is the step “eliminate the cause and add a check.” Therefore, the class of failures that returned a week later goes away.
Knowledge transfer to the on-call shift stops depending on who is on the phone: a new engineer starts on-call duty using a checklist, not stories from senior colleagues.
Runbook formats: what suits your team
The format of a runbook determines not the neatness of the documentation, but the time to recovery: the on-call engineer reads exactly what they can get through in the first two minutes, and from that text decides whether to wake the second line.
Therefore, runbook formats are compared by how quickly they lead to the first diagnostic step and what they suggest doing when a step does not work.
Approaches do not compete but layer on top of each other: a consolidated document holds the picture of services and responsibility boundaries, incident-type scenarios cover specific failures, and on-call shift checklists cover the routine of shift handover. The choice is where to start and how much support the team is ready to bear.
Comparison of documentation approaches
| Option | When it fits | Limitations |
|---|---|---|
| Single consolidated document | Few services, need shared context of architecture and failure points | The needed section takes longer to find than to read; diverges from reality with frequent releases |
| Incident-type scenarios | Several services, different failure classes, scheduled on-call duty | Duplication of steps; with a composite failure — switching between two documents |
| On-call shift checklists | Round-the-clock rotation, less experienced first line | Work until the first non-standard step; after that a scenario is needed |
| Automated prompts on failure | Frequent same-type alerts, there is an alerting system, e.g., PagerDuty | Live in code and require maintenance; a false alert sends the on-call engineer in the wrong direction |
A common limitation is divergence from reality: an outdated instruction leads down a false path more confidently than its absence. Therefore, each scenario includes checkpoints and a signal at which the on-call engineer stops and escalates instead of blindly following the steps.
Runbook development: stages of working on a runbook
A runbook makes sense only if it is assembled from actual failures of a specific environment, and not from a template from someone else's repository.
Therefore, the work begins not with the text of the document, but with analyzing what has already broken and how on-call engineers got out of the situation.
Each stage below yields an artifact that the next one relies on — from the list of failures to a document tested in a drill.
-
Incident audit. We analyze the history of failures: what went down, how long it took to notice, how long recovery took, what actions the on-call engineer took. Recurring failures go into the runbook first, rare ones go into the escalation section.
-
Service dependency map. We build a graph of connections: what calls what, where the shared failure point is, in what order to bring the system up. It also sets the boundary — where the on-call engineer's zone ends and the service owner's begins.
-
Scenario separation. We divide failures into those that can be resolved step by step and those that require immediate involvement of a specialized engineer; for each we define recognition signs and an escalation threshold.
-
Writing the runbook. The structure is strict: trigger condition, diagnostics, remediation steps, recovery criterion, rollback procedure. The wording is verifiable — “make sure the processing queue is not growing,” not “check the service state.”
-
Drill testing. We stage a training failure in a test environment, and the on-call engineer follows the scenario strictly according to the document. The places where they stopped or deviated are rewritten: this is a defect in the runbook, not in the on-call engineer.
-
Team handover. We review the document shift by shift, tie its launch to the on-call system and work chats, appoint an owner and a review period. After that, the runbook lives in the repository next to the configuration, not in chat.
Deliverable package: runbooks, diagrams, checklists
The documentation package lives in the same repository as the infrastructure code and goes through review like any ordinary change.
Then the runbook does not diverge from reality: a change to the configuration and a change to the recovery scenario are reviewed in a single merge request. Below is what exactly remains with the team after handover.
- Runbook scenarios — step-by-step procedures for a specific class of failure: dependency degradation, disk overflow, node failure. The decision point, rollback command, and the sign by which the failure is considered closed are specified.
- Service and dependency diagrams — what depends on what, where the failure boundary lies, which external systems participate in request processing. Updated with every topology change.
- Escalation matrix — on-call engineer level, service owner level, management level: response time at each and the criterion for handing over higher.
- On-call checklists — shift acceptance and handover, monitoring walkthrough, checks before and after deployment. Wording in the form “checked — marked,” without ambiguous items.
- Document review procedure — owner and review frequency for each document; outdated versions are marked, not deleted, so decisions are not lost.
- Review log — what broke, which scenario was applied, which item in it should be added based on the outcome.
Case study: runbooks for a service support team
A service platform of several dozen components, three teams on call, a round-the-clock schedule. Knowledge about failures lived in chat: each engineer had their own check order, and some notes were in personal documents.
Incident analysis began with the question “has anyone seen this before?” — context was gathered anew every shift.
Before: diagnostics from memory
Median time to fault localization was 47 minutes, and half of it went not to diagnostics but to finding out who had already checked what. Approximately one third of incidents were closed according to a pre-written scenario; the rest was improvisation by the on-call engineer. False escalations to development consumed the time of two teams at once.
After: a connected set of scenarios
First, we went through six months of incident logs and grouped failures by external symptoms, not by services. For each group — a scenario: check steps in order, signs that make escalation mandatory, and a rollback point to the previous state.
The runbooks were placed in the same repository as the service configurations, changes go through review; monitoring dashboards in Grafana are tied directly to the steps.
The median localization time dropped to 14 minutes, and the share of incidents closed according to the scenario grew to 82%. False escalations fell by almost half, and a new on-call engineer reaches independent shifts in days, not weeks.
How does an incident response runbook differ from an instruction?
An instruction describes the service: component composition, dependencies, behavior of nominal modes, metric collection points. It is read to understand how everything is arranged and where to look during analysis.
A runbook answers a different question — what the on-call engineer should do when the signal has already fired. It is the order of steps, branching points based on the check result, and the criterion by which the incident is considered closed.
The difference between a scenario and a description is in the point of reference. A scenario is built from a trigger and leads through “if — then” branches; a description captures system behavior regardless of what happened.
Mixing them harms both: the instruction becomes cluttered with steps for a rare failure, and the runbook turns into a reference that is not read to the end during an outage.
A runbook is needed where a failure recurs, affects several services, and recovery time is critical. Then the steps can be tested in drills, an owner can be assigned, and the document can be kept alive — it is updated together with infrastructure changes.
Where a failure is isolated and dealt with on the spot, a step-by-step scenario does not pay off: it goes stale faster than it can be applied. A service description and a post-factum review are enough.
The boundaries of application are fixed in the document header: which signals it covers, what to do when they are exceeded, and which instruction to reference instead of duplicating.
The separation of genres is inherited from ITIL practices, where a service description and a recovery procedure are different artifacts with different owners and different lifecycles.
Frequently asked questions about runbooks and their support
Before approving a runbook, teams ask the same questions: who is responsible for the document after handover, what to do with already written drafts, and how the runbook lives alongside the on-call schedule. Below are short answers; the rest we discuss at the task estimation stage.
Who becomes the owner of the runbook, and what does that change?
The owner is appointed as a role, not a specific person: usually the lead engineer of the area. The document header records the role, review period, and update triggers — a change in service topology, a new class of failures, a change in access policy. This way, keeping it current becomes a responsibility, not goodwill.
We have draft runbooks — will they have to be rewritten?
No. First we analyze the existing drafts: we compare the steps with logs of real incidents, mark places where an action is not reproducible, and supplement them with result checks. Most of the text is preserved, but control points and an unambiguous order of actions appear.
How is the runbook connected to on-call duty and alerts?
Each step is tied to an on-call role and an alert channel: who sees the signal first, what they do before escalation, to whom they hand over control. The connection is verified by a training run using the checklist, not by reading the document.
After a major incident, the runbook is reviewed, and versions are stored in the repository with edit history and change review.
We address other concerns — about access, rollback, and the division of responsibility zones between teams — specifically. Describe the task: we will return with an estimate, a work plan, and the procedure for supporting runbooks after implementation.Let's discuss the task and return with an estimate and plan
To start, access to the entire infrastructure is not needed.
To discuss the task specifically, four things are enough: a repository with configuration and scripts, a list of environments with a note on how they differ, a list of incidents over recent months with clarification on which were analyzed from memory, and one service that is not a problem to take first.
Based on this set, we estimate the scope and see where the gap is between “the team knows what to do” and “it is written so that an on-call engineer at three in the morning will execute it.”
The start of the project is structured as a pilot on one service — this way the Runbook is tested in real on-call duty, not during approval of wording:
- Limit the pilot service: one team, one on-call engineer, real recovery scenarios.
- Collect a timeline of recent incidents and pull out the steps that were performed anew each time.
- Write procedures step by step: what to check before the action, what the success sign is, at what point to escalate.
- Run the scenarios in a maintenance window and compare actual steps with the recorded ones.
- Give the document to an engineer who did not participate in writing it: if they complete recovery without hints, the steps are considered verified.
- Record the pilot metrics — recovery time, number of escalations, share of scenarios closed according to the document.
- Agree on replication to the remaining services and the order in which we take them.
The work plan comes as a sequence of stages with dependencies, the composition of the team on the customer side, and acceptance criteria for each step. Describe the task and send what you have — we will return with a scope estimate and a plan.