Why on-call rotation breaks in the third week
An on-call rotation doesn't break in the first week — it breaks in the third.
By then the schedule has already been through several night shifts, some of which turned out empty: in the practice of Google SRE, a shift where the on-call engineer gets no more than a couple of real firings is considered healthy — not twenty checks "just in case."
The mechanics of failure are simple. An alert arrives without context: it's unclear what exactly fired, which path degraded, and whether this happened yesterday.
False monitoring firings flow in the same stream as real incidents, so every call starts with manual triage — pull up the graphs, open the logs, ask in chat.
Shift handover is most often verbal: two messages, "everything's calm here." What was fixed, what was postponed, which threshold was temporarily raised — none of it survives anywhere except someone's memory.
The next on-call engineer follows the same route, and night shifts turn into interrupting sleep for the sake of diagnostics.
After two or three weeks of such a rotation, the schedule rests on a single volunteer: everyone else finds reasons to swap. This is on-call burnout, not a matter of discipline — and it's treated not with persuasion but with the design of the on-call process. Let's break down how this gets closed.
What a managed on-call scheme gives engineers
A managed rotation differs from a list of on-call names in a calendar in that every call has a measurable route.
An alert passes through a significance filter and gets an owner, and if it isn't acknowledged it automatically goes to the next line — without manual phone calls and searching for "who's on duty today."
Time to response and share of closed calls
After routing is configured, time to response stops depending on who noticed the message in the shared channel first.
A single escalation point removes diffuse responsibility: a priority call goes to a specific engineer, not to "the team," and if unacknowledged it rises to the backup.
The share of closed calls grows precisely through coverage — night, weekend, and holiday intervals are distributed in advance, and no gaps remain between shifts.
How many shifts an engineer can withstand without losing quality
The load on an on-call engineer is measured by the number of wake-ups per night, and this is the main parameter we keep under control.
By cutting off alerts that don't require immediate action, we manage to halve them: the rest goes into a morning digest that gets reviewed during working hours.
We pick the length of the rotation itself so that a series of shifts doesn't wear the engineer out, and transitions between shifts are explicit and confirmed by both sides.
On-call formats: daily, weekly, distributed
The on-call mode is chosen to fit the firing profile: their frequency, criticality, and whether the engineer has the authority to stop a rollout.
The wrong choice doesn't show up immediately — a month later the person stops reacting to night notifications, and accumulated fatigue turns into a missed incident. Below are the options people actually choose between, and their limits of applicability.
Solo and paired on-call
| Option | When it fits | Limitations |
|---|---|---|
| Daily shifts | Rare firings, triage fits into minutes, a team of five or more engineers | The night comes after a working day; with several calls per day, recovery takes longer than the triage itself |
| Weekly rotation | A stable service, night calls are isolated, deep context immersion is needed | Seven nights in a row for one person; unclosed context is hard to hand over mid-shift |
| On-call with reinforcement | A young service, night changes, no authority for a sole decision | More expensive in terms of the number of people involved; explicit rules are needed for when reinforcement joins |
| Distributed scheme | A team across several regions, the service runs around the clock | Handover across time zones requires a regulation and a shared incident log |
Shift handover between time zones
Handover across time zones removes the night wake-ups, but it breaks down where there's no written contour: what's already done, what's left, which firings are silent.
Shift handover takes a fixed fifteen minutes at the end of the shift, decisions are recorded in the shared log, and the on-call schedule is built a quarter ahead and published in advance — then a swap can be planned without a scramble.
How we build an on-call rotation: implementation stages
Before putting people on the schedule, we analyze what's already making noise in the system: an on-call engineer receiving dozens of notifications per shift will miss a real failure among the false ones.
That's why implementation proceeds step by step — from an inventory of alerts to drills on live systems.
Audit of alerts and thresholds
- We compile an inventory of alerts from monitoring and availability checks: who receives them, at what time, what they do next. Some turn out to be duplicates of a single event.
- We split the stream into "react immediately" and "review in the morning" — the latter doesn't wake the on-call engineer at night.
- We rewrite thresholds: the confirmation window, tying to load seasonality, grouping by service — one failure produces one notification.
On-call schedule and escalation scheme
- We build the escalation scheme: first on-call engineer, after a set interval — the second, then the person responsible for the service.
- We configure the schedule by weeks, accounting for time zones, vacations, and context handover between shifts.
- We fix the regulation: where to look at the current state, how to take over a shift, whom to hand an unclosed incident to.
Testing live firings
- We run drills: simulate a failure, measure time to response and to localization, look for gaps in the chain, and adjust thresholds.
We hand over to operations with a service map, a regulation, and a review of past firings — the client's team then runs the on-call rotation themselves.
Escalation policies and on-call regulations
On-call works when it has a written frame: who responds first, after how many minutes the second level joins, and what counts as grounds for escalation. Below is the list of documents and settings the client receives along with the rotation.
The set is stored next to the infrastructure description, not in a chat thread, and is updated when the composition of services changes.
Responsibility matrix and escalation levels
The matrix links each service to its owner, the on-call engineer, and the person who decides on stopping or rolling back a release.
Then the timers take over: if the first level doesn't acknowledge the signal within the allotted time, the call goes to the second, then to the head of the department. The cascade is stopped by an explicit action, not on its own, otherwise false noise will wake the whole chain.
- Responsibility matrix: service, owner, on-call engineer, the circle of people with decision-making authority on an incident.
- Escalation policy: levels, acknowledgement timers, conditions for moving further, and the procedure for clearing the cascade.
- On-call regulation: shift length, swap and replacement, limits on consecutive night shifts, rest after triage.
- Shift handover template: open incidents and their status, unclosed checks, planned work for the day.
- On-call instructions: first steps for a typical failure, where to look at the logs, how to isolate the service, whom to notify.
- Alert routes: what goes to the phone, what goes to the work chat, which thresholds cut off the noise.
- Checklist for onboarding a new on-call engineer: access, a test signal, a review of past incidents before the first shift.
We test every document on a training signal: we raise a test incident and see whether the response fits within the policy timers.
Duplicate calls, "hanging" acknowledgements, and services outside the matrix become visible already in this run — before a real night failure.
Case: night wake-ups cut by three times
Before the rotation was implemented, on-call rested on a phone list and personal agreements: alerts went to a shared chat, and at night they were seen by whoever happened not to be asleep.
Average time to response reached 40 minutes, and about 60 night calls accumulated over a month — more than two-thirds were closed with the words "it went away on its own."
What we changed in thresholds and call routing
First we dealt with the noise. Every alert got an owner, a firing threshold, and a "holds for longer than N minutes" condition; metrics and alert rules were gathered in one place — Prometheus.
Call routing was split by levels: the first-line on-call engineer, after 10 minutes — the second, and only then the general development chat.
Planned work was closed off with silence windows, and an alert without an action described in the instructions was decided not to be raised at all.
Two months later, night calls went from 60 to 18, the share of false firings fell from 70% to 12%, time to response — from 40 to 6 minutes, service recovery — from three hours to forty minutes.
The schedule became even: two weeks per person, swaps known in advance, no one trading places at the last moment. The rules live in the repository next to the service configuration — a threshold changes through review and change history, not in chat.
What to do if on-call is already deployed and not working?
Existing on-call rarely breaks entirely — it degrades gradually. Alerts arrive, but some of them no one acknowledges, thresholds fire on noise, and the engineer sleeps with the phone on and still misses the signal.
You can't stop the scheme and rewrite it from scratch: response must work every minute while the rebuild happens without stopping.
We start not with a new schedule but with measurement. We look at how many firings came in over a week, what share was closed without action, what's duplicated between services, how much time passes from the first signal to acknowledgement. From this data it's clear which thresholds are noisy and which alerts have no owner at all.
Next comes the separation of alert streams: only what requires immediate action and indicates service degradation goes to the on-call engineer's page, the rest goes into a separate triage queue.
This removes alert fatigue and restores constant readiness: the on-call engineer responds not because they have to, but because every signal means a concrete action.
We introduce the second escalation level before changing the schedule. If the first on-call engineer doesn't acknowledge the signal within the allotted window, the second is raised — with the same context, history, and a link to the triage. We test the chain on a training firing so that it reaches the end.
We change the schedule and rotation last, when the stream is already filtered and escalation is verified. That way the order of shifts and the rules are refined without the risk of losing a critical signal at the moment of handover.
Common doubts: automating schedules and alerts
Doubts about an on-call rotation are usually the same: will there be enough people, will the schedule hold up to vacations and sick leave, who's responsible at night, and won't on-call turn into round-the-clock reading of alerts.
We address them in order — from the side where the decision is made by engineering, not by organization.
What is the minimum team size needed for a rotation?
Formally, even one person can be on call, but a sustainable rotation starts with four to five engineers: only then are night shifts distributed without regular overtime and a reserve for vacation is preserved.
If there are fewer people, the scheme is built differently — the primary contour (receives and qualifies the signal) and the secondary (joins by escalation) are separated, and incident triage is moved to working hours.
Manual schedule or automation?
Scheduling automation isn't needed for the beauty of the calendar. Rules in the on-call system account for time zones, shifts, swaps, and overlaps — during illness or vacation the schedule rebuilds itself instead of by phone calls.
A manual schedule works while the team is small and sits in one time zone; beyond that it starts to fall apart at the first unplanned absence.
How do you get rid of alert fatigue?
Alert fatigue is treated not by "switching off the excess" but by routing rules: grouping related signals from one service into a single incident, suppressing repeat firings, duration thresholds, and separate channels for night and day shifts. The engineer gets one actionable signal instead of a stream, and on-call becomes meaningful again.
Can you do without night on-call?
Completely — rarely. The night window is narrowed where it's justified: critical services remain on round-the-clock readiness, the rest goes into a working-hours response mode with automatic recovery.
It's important to fix the rule in advance: what exactly counts as grounds to wake a person, and what waits until morning.
What to do about shift handover between time zones?
Shift handover across zones is the most effective way to remove night hours: the rotation follows the Sun around the globe instead of staying in one city.
It requires external monitoring independent of the on-call team: if the observation node itself is unavailable, the shift is taken over by another contour, not by the on-call engineer manually. Plus a short handover — what's open, what's under observation, what changes were rolled out.
Let's discuss your on-call scheme and firing threshold
The first conversation is not a presentation of services but a breakdown of your current scheme: who's in the rotation, how the windows are distributed, which signals wake you at night and what happens to them next.
To discuss the task substantively, we need that context — then the assessment of the escalation scheme rests on facts, not on general recommendations.
For the on-call implementation plan we clarify:
- A review of current monitoring: which systems send signals, where they arrive, and who triages them — this is the starting point.
- How many engineers are in the rotation and which windows they can realistically withstand: this determines whether the shift will be weekly, daily, or split into day and night.
- Firing thresholds: which events must wake a person, and which are enough to record and review in the morning.
- Assessment of the escalation scheme: who the signal goes to if the first on-call engineer doesn't respond, after how many minutes, and through which channel.
- Access and permissions: who can mute an alert, change a rule, add a participant to the rotation.
- How night firings are triaged and who watches for recurring false calls.
Describe the task in whatever form is convenient — the current rotation, a list of systems, a couple of examples of night firings.
We'll come back with an assessment of the escalation scheme, an on-call implementation plan, and project start timelines: where we'll begin, what we'll close first, and by what signs we'll see that the noise in alerts has dropped and incident triage no longer stretches over days.