Why runbook automation breaks on live incidents
A live incident almost never looks like the one described in the knowledge base.
The on-call engineer opens the procedure page, sees a dozen steps with “if necessary” caveats, and starts improvising — because an incomplete description of steps does not answer the main question: in what order to test hypotheses when the service is already unavailable.
While the engineer finishes reading the discussion thread and clarifies with the procedure author what was meant, service recovery time grows with the number of steps, not their complexity. Each transition between systems is a context switch and decisions based on outdated data.
The bottleneck of the on-call shift turns out not to be qualification: two engineers following the same instruction arrive at different actions and different outcomes.
Manual response procedures rely on specific people’s memory. While they’re on shift, the failure is closed quickly; on vacation, at night, or during a new engineer’s first on-call shift, the same outage takes twice as long.
Some steps are simply not performed — they’re missing from the text or hidden at the end, and the omission surfaces only during the postmortem.
Runbook automation breaks precisely at this transition: the procedure was moved into the knowledge base but was not made executable. We examine how this is addressed — from the structure of step descriptions and explicit checkpoints to automatic markers for completed stages.
Measurable effect of managed response procedures
The effect of managed procedures is visible in four numbers: time to the on-call engineer’s first action, share of incidents closed according to a scenario, service recovery time, and variation in actions between shifts.
Until scenarios exist, all these values depend on a particular engineer’s experience and change from shift to shift.
Reaction and recovery time
The signal reaches the on-call engineer together with context: what exactly failed, which dependencies are affected, which scenario is suitable. The first minutes go not to diagnosis but to confirming the scenario — typical steps are performed automatically.
Time to first action drops from tens of minutes to one or two, and service recovery fits into approximately the same interval.
Share of incidents closed according to scenario
Scenarios do not cover everything: some failures are unique and require investigation. A working target is 70–85% of requests closed without manual improvisation; the rest goes into a queue for procedure refinement.
Each postmortem becomes a new scenario, so the share grows quarter over quarter rather than staying flat.
Repeatability of on-call engineer actions
The scenario removes the main source of variation — different people understand “restart the service” differently.
Platforms like Kubernetes provide standard automation points, so the scenario invokes the same operations the engineer performed manually, but consistently and without skipped steps. A new on-call engineer reaches the target reaction time in days, not months.
The cost of downtime stops depending on who is on shift: unavailability is limited by the scenario execution time, not by diagnosis speed. These metrics form the basis for acceptance of the work.
Procedure formats: from regulation to self-healing
The procedure format determines how much time passes between signal and recovery.
A text regulation describes steps but does not run them; a checklist fixes the order; a scenario executes commands for the on-call engineer; event-based triggering removes the person from the chain; self-healing restores the service without on-call involvement.
Each subsequent format requires a clearer failure indicator and a safe rollback — this is not an upgrade for automation’s sake.
| Format | When it fits | Limitations |
|---|---|---|
| Text regulation | Rare incidents, new service, few repeatable actions | Steps are interpreted differently; recovery time depends on the on-call engineer’s experience |
| Checklist with manual launch | Operations that require control at every step | Manual launch is a bottleneck; steps are skipped under time pressure |
| Button-triggered scenario | Typical recovery, decision made by a person | Part of the time goes to the on-call engineer’s reaction, not the work |
| Event-based triggering | Failure with an unambiguous indicator and a known scenario | False triggers: thresholds, dry run, and run limits are needed |
| Service self-healing | Failure with a safe and reversible scenario | Automation masks the root cause; rollback and state recording are mandatory |
The limitations of automation come down to two questions: how to distinguish a real failure from a load spike and how to roll back if the scenario triggered incorrectly.
That is why formats are arranged as a ladder — a button-triggered scenario on the same checklist, then event-based triggering with reduced permissions, and only then self-healing with a log of every action.
Stages of runbook automation implementation
Response automation begins not with scripts but with analyzing what on-call engineers do manually. Skip this step and you automate the current chaos: the scenario triggers but does not address the right failure cause.
- Audit of response procedures — we analyze existing runbooks and record which checks and commands on-call engineers repeat, and what they decide from experience. The output is a list of procedures with trigger frequency and average duration.
- Failure response scheme — we build a map: which monitoring signal corresponds to which failure class and which chain it launches. We separately mark points where human confirmation is required.
- Setup of automation scenarios — we move repeatable steps into executable scenarios: diagnostics collection, service restart, traffic switching, calling the responsible person. Each leaves a trace in the log.
- Scope limitation — we assign scenarios minimal permissions and manual confirmation for irreversible operations so automation cannot bring down the production environment by individual calculationVerification on a training incident — we run scenarios on a controlled failure in a test environment, first one at a time, then as a chain. We adjust exit conditions where the scenario stalls.
- Handover to the on-call shift — we prepare documentation, embed scenarios into the on-call engineer’s instructions, and review with the team: what automation does on its own and where the decision remains with the person.
Next comes a scheduled review: scenarios that never triggered or were rolled back manually are refined or returned to manual mode.
What we hand over to the team: diagrams, policies, and regulations
Handover does not end with a demonstration of a working scenario. The team receives a set of artifacts by which an engineer who did not participate in the implementation can reproduce the response logic and safely modify it.
Each document is tied to a live object in the system — configuration, service catalog, access roles — so it does not diverge from the actual state.
- Scenario configurations — descriptions of triggers, trigger conditions, and steps in a version control system, with clearly marked points where on-call confirmation is required.
- Service dependency diagram — a graph of calls and queues: you can see which component’s failure raises the scenario and what a restart or rollback will affect.
- Rights and roles matrix — who launches the scenario manually, who changes thresholds, who works with secrets; dangerous actions are separated from routine ones.
- On-call regulations — schedule, areas of responsibility, escalation procedure, and shift handover rules with transfer of unresolved incidents.
- Instructions for the on-call engineer — step-by-step guides: where to start the investigation, which metrics and logs to check, when to stop automation and involve development.
- Refinement procedure — how to add a new scenario, which checks are mandatory before rollout, and how to roll back a change if it behaves differently.
- Change and postmortem log — what we record after triggering: whether the scenario helped, where manual intervention was required, which thresholds should be adjusted.
Such a set removes dependence on the implementation authors: the on-call engineer acts according to instructions, not memory, and changes to automation are made predictably.
Case study: payment environment failure — metrics before and after
We examine a payment service project where the same failure — desynchronization of the payment queue and transaction database — was handled in two different ways.
The signal came to the on-call shift via PagerDuty, and then manual work according to the knowledge base instruction began.
Before scenarios were implemented, recovery took an average of 48 minutes and 14 manual steps: collect logs from three nodes, compare queue counters, stop consumers, bring up a replica, switch traffic, check checksums.
The order relied on the on-call engineer’s memory, and if a step was skipped, the service returned to failure in one third of cases — already in front of customers.
| Metric | Before scenarios | After scenarios |
|---|---|---|
| Service recovery time | 41–52 min | 6–8 min |
| Number of manual steps for the on-call engineer | 14 | 2 |
| Repeated failure within 24 hours | every third case | not recorded |
| Diagnostics for postmortem | collected manually | attached to the incident |
Now the scenario itself collects a state snapshot, saves it as an incident artifact, and performs idempotent steps in a strictly defined order. The engineer gets involved only for the traffic switching decision, when an assessment of business impact is needed.
What was eliminated is not a single failure but an entire class of failures — the error of action order during manual recovery.
The customer’s operations manager summarized the result as follows: incident review fits into one call rather than stretching into a day of searching for the cause.
How does a paper regulation differ from a scenario in the system?
A text regulation answers “what to do” but does not verify that it was done. The on-call engineer reads the instruction, manually translates it into commands, and each time reproduces the order in their own way — especially at three in the morning. Hence the variation: one restarts the service, another first removes traffic from the node.
An executable procedure removes the interpretation layer. Steps are launched from the system where the incident is already recorded, and parameters — node name, environment, release number — are substituted from context rather than typed manually.
The scenario lives alongside infrastructure code, for example as an Ansible configuration: one file that both people and machines read.
A separate part is manual confirmation points. Traffic switching, data schema rollback, volume deletion remain a human decision: the scenario stops and waits for explicit confirmation.
Everything up to that point has already been done consistently — this is also the control over the on-call engineer’s actions: they manage the course rather than remembering the sequence.
Step logging has a secondary effect: each operation remains with a timestamp, executor, and result, and the postmortem proceeds from the log rather than memories. You can also see where the scenario systematically stumbles — this is the first candidate for rework.
The boundary runs along frequency and predictability. One-off and diagnostic actions are cheaper to leave as text: the instruction explains the logic and helps you think. Repeatable steps with a clear result benefit from becoming a scenario with stops and a log.
Common technical doubts before implementation
Engineers and tech leads ask the same questions before starting: what to do with incomplete descriptions, how to divide responsibility between shifts, and whether the scenario will become outdated in a month. Below are direct answers, without promises that “we’ll do everything ourselves.”
Procedures are described partially — where will you start?
With an inventory of actual actions: we look at conversations in on-call channels, the history of manual commands in the terminal, and incident postmortem records. We build a map of steps as it is and mark places where a person makes the decision. We automate only unambiguous areas; disputed ones remain with the operator with a clear prompt.
There are several on-call teams — whose steps do we take first?
Those that most often trigger escalation between shifts. Usually this is diagnostics and initial isolation: one team can restart a component, the second knows where to look in logs.
We divide the scenario into common steps and team-specific ones, and grant permissions by on-call roles.
Infrastructure changes often — won’t the scenario become outdated?
Everything that describes state manually becomes outdated. That is why we build steps on dynamic discovery: we get the list of nodes, groups, and services from the inventory system, not from a hardcoded list.
A new node enters the scenario on its own, and the description changes in one place — in the data source, not in the procedure text.
How do you verify a scenario without waiting for a real failure?
By running it in a training environment: we simulate a failure, see which steps worked, where the operator intervened, and at which step the delay occurred. Plus regular drills — once a quarter we run the scenario with the on-call shift. Result: reaction time is known in advance rather than discovered during an outage.
What if automation triggers incorrectly?
Every step has a reverse action and a stop point: the scenario can be interrupted at any stage and handed over to the operator.
Dangerous operations — restart, traffic switching, migration — proceed only with on-call confirmation, and all actions are recorded in the log with the initiator indicated.
Let’s discuss your response procedure and estimate the scope of work
The first conversation is not a presentation but an analysis of how an incident actually travels from monitoring signal to closure.
It is important for us to understand where a decision is made manually, where the procedure diverges from practice, and which steps are repeated time after time.
The more source data on incidents you provide at the input, the more accurate the scope estimate: guesses at this stage are the most expensive.
What is useful to prepare in advance:
- an export of incident history for several months: detection time, time to localization, composition of the on-call shift;
- current instructions and step-by-step action lists — even if they are outdated or live in chat;
- a diagram of services and dependencies: what is connected to what and what fails first;
- a list of monitoring and alerting systems where signals arrive;
- constraints: regulations, security requirements, prohibition of automatic actions in the production environment.
Based on the results, we come back with three things: a stage-by-stage scope estimate, an implementation plan with sequencing — from the most frequent scenarios to rare ones — and a team composition for the task.
Describe the task in a convenient form: a short call or an email with a description of the procedure. Then we analyze what exactly changes in your response.