Why on-call procedures break on the third incident
The on-call engineer closes the first night call using the runbook, the second from memory, and the third exposes defects in the procedure itself.
Ownership is blurred: it is unclear which service belongs to whom, who acknowledges the incident, and along which route escalation proceeds. Formally, engineers are on call around the clock; in practice, it rests on the personal responsibility of two or three people.
Alerts pour into a shared channel, where everyone sees them and no one responds. Thresholds trigger on normal load fluctuations, some signals are duplicated by different systems, and the on-call engineer spends the shift filtering noise instead of diagnosing. Incident escalation goes into direct messages — to the person who “usually helps.”
The consequences do not show up immediately. Night calls concentrate on the same engineers, and time to recovery grows from incident to incident because the route is assembled anew each time.
Next comes on-call fatigue, missed signals, refusal to take an extra shift, and attempts to suppress a problem before anyone hears about it.
We break down how this is addressed: a service ownership matrix, escalation with explicit levels and timeouts, cleanup of noisy alerts, and a rotation that people—not heroes—can sustain.
What an on-call policy provides: response time and eliminated failures
An on-call policy moves failure response from a “who noticed first” mode into a process with measurable boundaries.
As long as ownership areas and the escalation order live only in people’s heads, time to first response varies from a minute to hours, and incident review turns into a search for someone to blame.
The policy fixes who holds the shift, what counts as a signal, and after how many minutes the second line joins.
Time to first response becomes a manageable value: the shift schedule covers the full day, each level has its own area, and escalation triggers by timer, not by a phone call.
The on-call engineer receives only the signals that require intervention—routing cuts off noise and does not wake them at night for a deviation that can wait until morning.
Together, this compresses response from tens of minutes to single minutes and eliminates gaps during shift handover.
Missed alerts are rarely due to inattention—more often an overloaded channel is to blame: identical notifications, lack of acknowledgement, and solo on-call without backup.
The policy introduces acknowledgement, duplication to the second line, and a limit on consecutive shifts so that by the third night the on-call engineer does not lose vigilance; this logic underlies PagerDuty and open-source alternatives. The share of unanswered signals goes to zero—visible in the log, not just in words.
Incident review is described in a separate block: which signal fired first, why so late, and which changes we lock in. A timeline without personal conclusions—details are not hidden, and the cause is found, not assigned.
The load on on-call engineers becomes measurable: night call-ups, the share of repeat failures, and the services that are the sources of most signals.
Options for organizing on-call: comparison of approaches
The on-call mode is chosen not by team habit but by the load profile: service criticality, spread of peak activity hours, and external commitments on response time.
The same shift schedule will not suit both a payment gateway and an internal reporting tool—in the second case, around-the-clock support requires more resources than rare downtime.
| Option | When it fits | Limitations |
|---|---|---|
| 24/7 shift | Services with a continuous flow of operations and promised availability | Requires rotation of 4-6 engineers; otherwise, by the third night shift, the share of review errors grows |
| On-call duty | Load with pronounced peaks, rare failures at night | Response time depends on how quickly the engineer picks up the alert; without an escalation policy, the call hangs |
| Hybrid scheme | The service core is critical, peripheral modules can tolerate a pause | Harder to administer: two sets of escalation rules and different target response times |
| Business-hours on-call only | Internal services where nighttime downtime does not block the business | Night incidents wait until morning, so automatic transition of the system to safe mode is needed |
Each row is a trade-off between response time and team capacity, and the procedure fixes this explicitly: who takes the call, after how many minutes the second level joins, and when an alert becomes an incident.
In practice, the hybrid scheme is assembled most often—the critical contour is covered by shift-based on-call, and the rest is moved to calls with an extended response window. Next, we describe how this choice is fixed in the document.
Stages of developing turnkey on-call procedures
Developing the policy starts not with a template but with an inventory of what already happens: who receives signals, through which channels, and where they get lost. Without this, the document describes the desired state and diverges from reality at the very first night outage.
What the work sequence looks like
-
On-call audit. We look at actual schedules, how context is handed over between shifts, how many signals arrive per week, and what share requires intervention. The output is a list of areas where ownership is blurred.
-
Escalation route. We fix the levels: who receives the signal first, to whom it is passed if there is silence, and at what point the manager joins. A separate branch is a cause outside the on-call engineer’s ownership area.
-
Alert setup. We bind signals to people and shifts, remove shared addresses and duplicates, and separate priorities: what wakes someone at night and what waits until morning. False triggers go into a separate debrief according to an individual calculation. Training call.
We run the policy on a controlled incident: does the signal reach the right person, does escalation trigger, and does the team meet the target response time.
-
Policy handover. We deliver the document, route diagram, and on-call checklists, and review them with the team in a live session, not by email.
-
Support. For the first weeks, we review deviations from the policy and amend the wording until it becomes a working tool.
The result is a policy under which on-call can operate without verbal agreements: it is clear who responds, within what time, and what counts as a violation.
Documentation set: escalation matrix and policies
On-call documentation is valuable not for its volume but because it can be acted on at three in the morning—without the author nearby and without chatting in a messenger.
Therefore, we assemble the set as a working tool: every action of the on-call engineer, escalation owner, and manager is described before the incident begins. Below is what exactly is handed over and in what form.
- On-call policy — rotation calendar, shift duration, rules for substitutions and overtime compensation, requirements for on-call availability and response time.
- Escalation matrix — first, second, and third lines with timeouts: how long the on-call engineer waits before raising the next level, and who has the authority to stop a release or move the service into a gentler mode.
- On-call instructions — step-by-step scenarios for typical failures: what to check, which monitoring dashboards to open, where the boundary of independent actions lies, and what is forbidden.
- Shift handover procedure — a template for entering unresolved incidents, planned work, and monitoring “tails”; a shift is considered accepted only after explicit acknowledgement.
- Incident log — a single record form: detection, response, and recovery times, user impact, working hypothesis of the cause, and actual cause.
- Incident review procedure — a blameless review, timelines for conducting it, what becomes a separate task, and who verifies its completion.
- Contacts and access matrix — whom from adjacent teams, vendors, and external services to raise for each failure class and in what order.
- Update policy — who reviews the documents and how often, and how infrastructure changes get into them so that the paperwork does not diverge from reality.
Case study: how the on-call shift response time was reduced
Before: requests were lost between channels
An on-call shift existed, but there was no single entry point: some requests came in direct messages, some into a shared chat, and some surfaced only at the morning review.
According to our estimate over several months, about 40% of requests never entered the tracking system, so on-call metrics were not calculated: neither time to first response nor the share of incidents closed by the first support line.
After: signal, rotation, escalation
On-call signals were moved to PagerDuty: schedules, acknowledgement, and timeout-based escalation—if the on-call engineer does not respond, the signal goes to the second.
For the first support line, we assembled checklists and short instructions for frequent failure classes so that routine issues were closed on the spot instead of going to development.
Two months later, the picture is this: time to first response—median from 26 to 4 minutes, 95th percentile from 1 h 40 min to 11 minutes, share of requests left in direct channels—from 40% to 3%.
The share of incidents closed on the first line grew from 38% to 72%, and the number of problems first reported by users rather than monitoring was cut in half.
The effect came not from the tool itself but from the fixed rules: who acknowledges a signal and within what time, when they escalate, and what they must leave in the ticket after closing.
These same rules became the basis for a monthly on-call review—metrics show which failure class goes to the second line and what should be closed with an instruction.
How does scheduled on-call differ from around-the-clock support?
The difference starts with the ownership area, not the schedule.
Scheduled on-call: during the day, the engineer is busy with development and maintenance, and during their window is obligated to be the first to respond to a monitoring trigger and drive the incident to localization.
Around-the-clock support means shifts covering the entire day, with a first line, request intake, and escalation to subject-matter specialists.
The scheduled on-call engineer is responsible only for what is covered by monitoring rules and availability checks. Everything that does not produce a signal lives in the shared task queue and is handled during business hours.
The around-the-clock shift takes on the entire incoming flow, including requests that are not yet automated or described in the policy.
The failure class determines the choice.
Mass and well-detected failures—node unavailability, growth in error rate, response degradation, replica divergence—are handled by scheduled on-call: the signal arrives on its own, and the on-call engineer responds according to the instruction.
Silent failures, isolated complaints, and data questions require a live support channel; otherwise, they accumulate into a major incident.
Response speed is set not by the mode but by time to detection and escalation depth. In a schedule, the target time is fixed for the trigger and for taking the incident into review; in the around-the-clock scheme, response time to a request is added.
The load on the team with a rotation of several engineers with half-day windows remains sustainable, while full daily shifts require either expanding the team or distributed shifts across different time zones.
In practice, a hybrid more often works: scheduled on-call for infrastructure failures plus a support window for product requests, with a unified escalation policy.
We analyze which failure classes actually occur and build the schedule, instructions, and shift handover procedure around them.
On-call limitations and common technical doubts
On-call limitations almost always run into the same things: a small team, an already working schedule, services shared with adjacent departments, and vacations in the middle of a quarter.
Below are analyses of doubts that come up before the start, with engineering logic rather than promises.
We have three engineers—are on-call duties even possible?
They are possible, but the mode is built differently than in a team of ten: one primary on-call engineer, one backup on offset hours, and weekly rotation.
We limit night calls with automation—the on-call engineer is raised only by a confirmed signal, not by the first metric deviation. After a night review, the engineer leaves the rotation for a day; otherwise, the schedule collapses within a month.
On-call already exists, but everything rests on verbal agreements
We capture the actual practice: using the alert log and incident reviews, we reconstruct who raises whom and how. Then we compare it with what was stated—discrepancies are usually in escalation levels and acknowledgement timeouts. The result: the policy describes what the team actually does, not what is convenient to draw.
Some services belong to adjacent teams
For each service, we fix the owner, ownership area, and handover point: with what timeout the on-call engineer escalates externally and who is responsible for the response.
The interface is described by a protocol; otherwise, the on-call engineer hangs on someone else’s incident for hours and does not resolve their own. We agree on the boundaries with the leaders of adjacent teams before publishing the document.
What to do about vacations and reluctance to take night shifts?
The schedule is built with absences in mind in advance, and backup is put in place before a person leaves, not on the day of the shift. The night format is not mandatory: for some services, daytime response windows are enough if acceptable recovery times are agreed. Where they are strict, we keep one night level and move the rest to business hours.
Let’s discuss the task: on-call plan and timeline estimate
To make the on-call plan working rather than declarative, in the first conversation we clarify several things: which services on-call covers, whether a 24/7 mode is needed or support hours are enough, how incident signals currently arrive, and who has the authority to pause a release.
Separately, we analyze roles—primary on-call, escalation, business liaison—and the access level each needs.
Based on the discussion, we send not general wording but a concrete set:
- on-call scheme with rotations, days off, and substitution rules—the schedule does not fall apart during vacations and sick leave;
- escalation matrix: level, response time, to whom the incident is passed, and at what moment;
- stages of procedure implementation with timelines and approval points at each step;
- documentation handover procedure: which artifacts, in what form, and who accepts them;
- instruction templates for typical incidents—the on-call engineer acts step by step, not from memory;
- post-incident review policy: what we record after a failure and how edits get into the documentation.
Describe the task: which services are in scope, how on-call is organized now, and what is getting in the way. We will return with a timeline estimate, on-call scheme, and implementation stage plan.