On-call rotation: where support duty breaks
On-call rotation breaks not at the scheduling stage, but at the handoffs. One on-call engineer for all night incidents is a typical setup for a website support team, and it holds up as long as one request comes in at night.
As soon as there are three or four, incident response time grows with each next one: the first triage takes twenty minutes, the fourth takes hours, because the on-call engineer can no longer switch between tasks and loses context.
The second failure point is lost alerts. A notification goes to a shared channel where no one reads at night, there is no acknowledgment, no escalation.
The on-call engineer sleeps, the incident waits until morning; formally, there is a rotation, but in practice night support duty does not work.
Third is shift handover. If it comes down to "it was quiet at night," an unresolved incident surfaces a day later, and the new on-call engineer starts triage from scratch: the same hypotheses, lost context, the same diagnostic path all over again.
All three failures have one cause: the scheme lacks delivery acknowledgment, a response timeout, and an explicit list of what is handed over.
We break down how this is addressed: an alert route with acknowledgment, a second line in case the first is silent, and a shift procedure with a checklist of open incidents.
What managed on-call duty provides: four measurable effects
Response time is measured in minutes, not hours
Previously, a failure signal waited until someone freed up from their current task: the request was noticed via an email or a call.
After setting up on-call duty, a signal has a specific recipient in a specific interval — response shrinks from hours to minutes, and every failure gets an owner.
First-line support resolves most requests
The on-call engineer has clear diagnostic scenarios, permissions, and ready-made actions for frequent failures, so they resolve typical issues themselves.
The share of incidents closed at first line reaches 70–80% — developers return to product tasks instead of round-the-clock response.
The "noticed only by the customer" class of failures goes away
Checks of key scenarios and alert thresholds catch degradation before a visitor feels it. The team learns about a failure from a signal, not from a support message — the share of such cases drops almost to zero.
On-call coverage and workload distribution
A long-term schedule, backups, and a cap on night shifts eliminate the situation where the same person is always on call.
The workload is distributed evenly, predictable rest reduces team burnout, and documented incident escalation guarantees that the on-call engineer always has someone to hand the problem to.
Options for organizing on-call rotation: comparison of approaches
The on-call scheme is chosen not by team size, but by workload profile: how many requests come in outside working hours, how fast a response is needed, and whether there is a failure that cannot wait until morning.
Below are four working schemes that teams actually choose between, with their limits on workload and staffing.
Working schemes and their limits
| Scheme | When it fits | Limitations |
|---|---|---|
| Fixed weekly on-call schedule within the team | Predictable workload, rare night failures | With a team of fewer than five engineers, each person is on call too often — fatigue and response time increase |
| Escalation chain: first line → subject-matter engineer → manager | Round-the-clock support is needed without dedicated shifts | Response depends on who is available; without defined handoff deadlines, escalation gets stuck at the first line |
| Dedicated contractor on-call shift | Critical service, downtime at night and on weekends is unacceptable | The in-house team loses incident context — a handover procedure and shared access to the log are needed |
| Hybrid: in-house team during the day, on-call shift at night and on weekends | There is a core of engineers and a need for round-the-clock coverage | Requires a unified on-call scheme and a single point of request registration, otherwise incidents are duplicated |
The key limitation of rotation is not technology, but staffing depth: a schedule in which one engineer is on call every other night formally works, but in practice produces delays and missed alerts.
That is why the scheme is usually assembled from two levels: a permanent line for round-the-clock support and an escalation chain within the team for tasks that require code knowledge.
How we build on-call duty: implementation stages
We start implementing on-call duty not with the schedule, but with an analysis of what has already happened in support: how many incidents, at what time, who responded, and at which step time was lost.
This data determines both the number of on-call engineers and the depth of rotation — without it, the schedule is guesswork.
- Incident and workload audit. We break down the failure history by time of day and type, separating what requires immediate response from what can safely wait until morning. This shows how many people are actually needed in the rotation.
- Responsibility levels. The first line receives the signal and classifies it, the second fixes the application, the third is the domain owner, who joins to decide on a release rollback or data changes.
- Alert routes. For each failure class, we define the channel and sequence: first a ticket and a messenger notification, if the signal is not acknowledged within a few minutes — a call, then handoff to the next level.
- Queue setup. We assemble the on-call calendar taking into account time zones and weekends, enable automatic backups, and require shift handover with open incidents.
- Escalation procedure. We fix what counts as an incident, how quickly a signal must be acknowledged, and when to raise the level — a written document instead of chat agreements.
- Trial on-call duty. We run the rotation in shadow mode on real signals, but without responsibility for the result: we check whether alerts arrive at night and whether the on-call engineer is called for things that can be resolved without them.
- Handover to the team. Checklists for typical failures, a service map, a review of the first on-call shifts — after that, the rotation lives on without our involvement.
What your team retains after on-call duty implementation
On-call duty rests not on specific people, but on documents and settings.
The partner leaves — the artifacts remain in your environment: the calendar, alert rules, and triage procedure are in your repository and monitoring system, not in chats and the memory of individual engineers. Below is what exactly you receive and can change yourself.
- On-call calendar — a shift schedule taking into account time zones, weekends, and vacations, with windows for shift swaps and a clear backup procedure. It is filled out months in advance.
- Escalation scheme — who joins and after how many minutes if the on-call engineer does not acknowledge the signal: second and third lines responsible for related services.
- Alert rules — which events wake you at night and which wait until morning: trigger thresholds, quiet windows, grouping of repeating signals.
- On-call instructions — what to check in the first minutes, which actions are safe on a running system, where to view event logs, and when to call the second line.
- Support procedure — request levels, response and resolution deadlines for each, and the procedure for interacting with external contractors and vendors.
- Incident review procedure — a description template, timeline, root cause, remediation tasks, and verification that the fix actually works.
The documents remain with you and are updated through regular review. When the team composition changes, the rotation continues to work, and onboarding a new on-call engineer takes days, not weeks.
Case study: online store on-call duty without night overload
The online store «Гранат» runs on night traffic: a third of orders are placed between midnight and eight in the morning, and a failure in this window is more noticeable than during the day.
Support was carried by two engineers — six nights in a row, without a formal on-call rotation or backup on-call engineer. The support case began with measurements: how many signals actually require intervention and how quickly the team responds to them.
What the measurements showed
Average response time to a critical alert was 24 minutes, recovery was from one to two hours. During the night, the on-call engineer received 12–14 signals, more than half false: thresholds triggered on scheduled tasks and short spikes in response latency. Night incidents accounted for about 40% of all support requests.
What was changed
The night window was split into two five-hour shifts with weekly rotation, a secondary on-call engineer was added for backup and timer-based escalation: five minutes without response — the secondary, fifteen — the lead engineer.
Thresholds were rewritten for business events: payment unavailability, checkout errors, growth of the background task queue, and typical failures were covered by auto-diagnostic scenarios.
The on-call workload stopped being a lottery — the first-fifteen-minutes checklist describes what to check, what to restart, and when to call a developer.
Two months later, response time was 4 minutes, recovery was 32 minutes, and false positives were less than a tenth of signals.
The before-and-after metrics differ not because of "heroism," but because of shift distribution and predictable routes: night incidents are handled by the same rules as daytime ones.
How scheduled on-call duty differs from chain escalation
Scheduled on-call duty answers the question "who is available right now." Each shift has a primary on-call engineer with a fixed responsibility window and their own set of signals that come specifically to them.
Backup on-call is planned in advance: a person changes their slot in the schedule before the shift starts, and the entry point is not lost.
The escalation chain solves a different task — what to do if the first one does not respond. The signal is not tied to a specific person: it goes to the first level, after a defined silence interval — to the second, then to the shift lead.
PagerDuty and similar platforms describe this with rules: how long to wait, to whom to pass it, and what counts as a response. The scheme survives no-shows, illness, and vacation without manual schedule edits.
The difference shows during a failure. A schedule provides predictability: it is known who is responsible and during which hours. Escalation provides resilience: the chain does not break if the on-call engineer is unavailable.
Separately, both schemes fall short — a schedule without escalation accumulates missed signals, and escalation without a schedule blurs responsibility.
Therefore, the practical choice of scheme is a hybrid: an on-call schedule as the first level, with an escalation chain on top with waiting intervals and a backup on-call engineer.
We fix the time to escalation, the number of levels, and the signal return rule so that support continuity does not depend on one person.
Technical concerns: alerts, shifts, escalation
We configure alerts from the symptom, not from every metric: what wakes the on-call engineer is what already affects the user. We take thresholds from the service history, collapse repeats into one message, and review false positives — noise decreases over time rather than grows.
We cover planned maintenance with a quiet window: a release does not wake the shift, but if metrics are not normal after the window, an alert arrives. With multiple sites, each has its own message stream and first recipient, while the on-call schedule is shared.
Will alerts wake me at night for no reason?
We tie the threshold to user errors, not server load. For the first weeks, the rotation runs in observation mode: we collect statistics and adjust conditions before going live.
What about on-call duty during planned maintenance?
A quiet window is set in advance and only for affected services; other alerts keep working. A real outage during this time will not be lost, and after the window the on-call engineer will receive a signal if metrics have not returned to normal.
We have multiple sites — how is the rotation divided?
Each site has its own alert stream and first recipient; escalation follows a shared ladder. The shift immediately sees what went down without reading the entire stream; we fix the routing scheme so that new sites do not require rewriting the rules.
Where are the boundaries of our responsibility?
We are responsible for response, diagnostics, and service recovery within agreed timeframes. Access, infrastructure, and external services are on the customer's side: we agree on the list of what is needed from you before the rotation launch.
Let's discuss your team's on-call duty: where to start
To come back with an on-call assessment and implementation plan, we need baseline data — not a full technical specification, but the facts about your support.
We will gather it in a conversation: shift composition, critical services, current procedures, and how failure signals reach the engineer today.
- Coverage boundaries: which hours and days the on-call engineer must cover, whether night shifts and weekend slots are needed.
- Shift composition: how many engineers participate, their qualifications, and the acceptable frequency of on-call duty without burnout.
- List of services and acceptable response time for each — the depth of rotation depends on this.
- Signal entry points: monitoring systems, request queue, support chats, and current alert channels.
- Escalation matrix: who joins on the second and third lines, and where the on-call engineer's responsibility ends.
- Access and permissions: what the on-call engineer resolves independently and what requires approval from the service owner.
- Incident history: recurring failures show which checks to include in shift handover checklists.
After discussing the task, we will return with an on-call plan: a rotation calendar with shift overlap, escalation rules, shift handover templates, and alert settings divided by criticality.
The on-call assessment will be based on the actual labor costs of your team, not on averaged figures.
Describe the task — we will return with an assessment and implementation plan.
