Setting up incident management: where the chain breaks
An alert goes to the team's general chat — and drowns among release discussions and everyday messages. Formally, there is a response; in practice, no one measures time to response: the signal fired at night, someone noticed it in the morning, and reproduced it by lunch.
Lost notifications are almost never related to transport failure — the signal simply has no owner who is obligated to respond to it.
The second bottleneck in escalation is the absence of formal on-call schedules.
While an incident is not critical, everything rests on informal agreements, and at the moment of failure escalation turns into manually calling on-call engineers: first they call the lead engineer, he doesn't answer, then the department head.
Tools like PagerDuty or OpsGenie don't help here if the escalation order isn't described in them — they only record that someone didn't answer.
Noise in notifications masks real failures. When a rule is written for every threshold, without deduplication and grouping, a single database degradation generates hundreds of identical signals, and the operator learns to ignore them.
A useful signal stops standing out from the background, and time to response grows with the number of alerts instead of falling.
We explain how this chain is closed: how to separate routing by service and criticality so that a signal reaches a specific person, escalation triggers without phone calls, and the notification flow remains readable.
The effect of on-call schedules and escalations in measurable metrics
We take the baseline before changes: how much time passes from a monitoring signal to the first meaningful human action. On an environment without routing, this is 20–40 minutes — the alert lands in the general chat and waits until someone opens a laptop.
After binding rules to the on-call schedule, time to first response drops to 2–5 minutes — the metric is stable because it depends on configuration, not on how quick a specific engineer is.
The false positive rate is the second metric. Noise is created by conditions without thresholds: a single degradation arrives as ten alerts from different nodes.
Grouping by service, deduplication, and suppression during planned maintenance cut the false-positive share from 60–70% to 10–15% — the on-call engineer stops ignoring notifications, and a real failure is not lost in the noise.
Timer-based escalation closes a separate class of failures: a signal arrived but went unnoticed.
If the first level does not acknowledge the incident within a given window, the second is engaged, then the service owner — there are fewer night outages without response, and the outage itself is counted from the first signal to recovery.
The schedule covers all calendar hours, including holidays and vacations, with explicit substitutions: we keep on-call coverage at 100%, not "by agreement in chat."
Platforms of the PagerDuty class add reporting: response time, share of acknowledged incidents, and on-call workload are visible for each service. The conversation about reliability stops being about feelings — what remains are numbers that show where the process is failing.
Options for organizing on-call schedules: comparison of approaches
The on-call model is chosen not by team size, but by acceptable response time and the cost of a night outage. Below are four schemes that teams realistically choose between, with their applicability conditions and limitations.
| Model | When it fits | Limitations |
|---|---|---|
| Permanent on-call | One dedicated engineer, a small alert flow | Burnout; vacation and illness break coverage |
| Shift schedule | A team of four or more engineers, round-the-clock coverage needed | Schedule, shift handover, incidents at shift boundaries |
| Hybrid on-call model | In-house engineers during the day, on-call for critical services at night | More complex escalation rules, a single incident registration channel is needed |
| External response service | No round-the-clock team, strict response requirements | External initial diagnostics, access regulations, context handover |
What is configured in any scheme
Automatic escalation is configured on top of any scheme: a notification goes to the on-call engineer, and if he does not acknowledge it within a given interval, the signal is raised to the backup engineer, then to the department head.
This closes the main failure mode of a manual schedule — the alert went nowhere because the person was in a tunnel or asleep.
The limitation of an external service is not speed, but the depth of the first line: it receives the signal and records the incident, but investigation in the infrastructure remains with your team.
That is why the hybrid model often turns out to be a compromise: in-house engineers cover daytime and critical scenarios, while the external service provides night coverage.
Stages of implementing the incident response process
Enabling notifications is easy — the harder part is making sure the on-call engineer receives signals only for their service and understands the next step.
That is why the work follows a fixed sequence: first we describe the system, then design the routes, and only then enable escalation in production.
- Alert audit. We consolidate all sources — monitoring, logs, builds, signals from databases and load balancers — and remove noise: duplicates, thresholds without action, "just in case" notifications.
- Service and owner map. We assign an owner and on-call team to each service — otherwise routing turns into a manual search for "who supports this."
- Escalation rules. We define levels, acknowledgement wait time, and the order of handoff upward, including a backup group for nights and weekends.
- On-call schedules. We set up rotations and substitutions taking vacations and time zones into account, so every signal has a specific responsible person.
- Configuration and integrations. An incident automatically pulls context from tasks, chats, and the change log — correspondence is not lost in direct messages.
- Test alert. We run a real failure scenario: measure time to acknowledgement and to second-line engagement, and look for where the route broke.
- Handover and support. We hand over documentation and access, review the test alert results, and adjust rules as the number of services grows.
After a test alert, some rules are usually rewritten — that is a normal outcome. We consider the process implemented when the on-call engineer resolves a typical incident without prompts.
What we hand over: configurations, diagrams, procedures
Implementation ends not with access to someone else's dashboard, but with a set of configurations and procedures that lives within your team.
This set is enough for the on-call engineer to act step by step, and for a new engineer to join the shift without verbal explanations from the contractor.
- Escalation diagrams by line: who receives the notification first, after how many minutes of silence it goes higher, at which step the department head is engaged.
- On-call procedures: shift composition, handover rules, time zone overlap, replacement procedure for illness and vacation, limit of night on-call shifts per person.
- Responsibility matrix: service → owner → on-call team → the person who decides to stop a release or roll back.
- On-call engineer instructions: scenarios by failure class — what to check first, what to restart, when to escalate, and when to declare a major incident.
- Configured routing and suppression rules: separation of flows by service, suppression of repeated notifications, planned maintenance windows, so the on-call engineer is not woken by known noise.
- Incident journal: a single record template (symptom, impact, detection and closure time, actions taken) and a regular summary of recurring causes.
- Review procedure: how the incident meeting is held, what tasks are created based on the results, who tracks their closure, and after how many weeks the result is checked.
- Readiness check: test alerts for each scheme, measurement of time from trigger to on-call acknowledgement, identification of bottlenecks, and their elimination before handover.
We prepare all artifacts in your documentation storage and version control system — edits go through review, change history is preserved, and access is granted to your roles, not our engineers.
Case study: how we reduced time to response and removed noise
The customer is a product service with six teams and round-the-clock on-call. Signals converged into a general chat and a single phone: different systems duplicated each other, and some critical events did not reach the on-call engineer at all.
A review of incidents from the previous period showed that up to a third of significant failures were confirmed by user complaints, not by monitoring.
What we changed
We split the flow into two: pages — what requires immediate action, and tickets — what is handled during working hours. Each source got an owner from a specific team, a duplicate suppression rule, and an unambiguous criticality level.
On-call was moved to a schedule with shift handover via checklist, and escalation routes were described declaratively and kept in a repository — changes go through review. The notification transport is PagerDuty.
What the result was
Time to response dropped from tens of minutes to 2-5: a page goes to the on-call engineer's phone and is either acknowledged or escalated to the backup by timer.
The share of signals acknowledged on time grew from about half to 90+ %, and the number of night wake-ups for insignificant events fell by about three times — noise stopped masking real failures.
Shift handover became verifiable: the next on-call engineer sees open incidents, their state, and owner, instead of reconstructing the picture from chat.
Recurring alerts of the same nature go into tasks to eliminate the root cause, and such sources are gradually removed from on-call.
How does incident management differ from an on-call chat?
An on-call chat records discussion but does not answer the main question: who exactly owns the signal right now and by what deadline they must respond.
At night, a message in the channel is easy to miss, and manual on-call relies on the engineer noticing the alert and handing over the shift themselves. The response process begins where there is a schedule, an owner, and a measurable deadline.
| Approach | What it covers | Where it breaks |
|---|---|---|
| General chat | Team notification and discussion history | No owner or deadline — the signal drowns in chat |
| Manual scheduled on-call | Formally assigned responsible person | Manual shift handover, no escalation |
| Response platform | Routing, acknowledgement, escalation | Requires described rules and configured schedules |
The key differences lie in three mechanisms. Routing rules match an incoming signal to a service, criticality, and the required expertise, so the alert goes not to a general channel but to a specific on-call engineer.
Acknowledging signal receipt turns "saw it" into an explicit action: until the engineer presses the button, the incident is considered unaccepted.
Automatic escalation follows this through — if there is no acknowledgement within the allotted time, the signal goes to the second on-call engineer, then to the department head.
This is how PagerDuty and platforms close to it are designed: the value is not in a beautiful interface, but in the fact that the signal path is described and verifiable.
After an incident, it is visible how much time went into acknowledgement and response, where escalation worked, and where the schedule led to a person who cannot help.
The chat does not disappear — it remains a place for review and discussion, but stops being the only response channel. Below we discuss how to build this process for your infrastructure and what routing rules are needed for it.
Common technical concerns before implementation
Concerns before implementation usually relate not to the on-call system itself, but to how it will fit into an already operating response environment. Below are the questions engineers ask most often, and how each is resolved at the configuration level.
Will this turn into an avalanche of notifications?
An avalanche is a consequence not of the channel itself, but of missing grouping rules. Identical alerts from different nodes are collapsed into one incident by service label, and the suppression window silences repeated signals until it is closed. An unresolved and unacknowledged incident is escalated to the next on-call engineer.
Will we have to rebuild the on-call schedules?
No. Existing schedules, windows, and substitution rules are transferred into the on-call system as is — only routing changes: which signal goes to whom and in what order.
First we describe the current schedules, then configure the binding and test it with test alerts, so that a shift does not end up without a responsible person.
What should we do if the communication channel is unavailable?
The key element is acknowledgement of receipt: until the on-call engineer marks “accepted,” the incident is not considered taken into work.
If the mobile notification is not acknowledged within the allotted time, the system calls the phone, and then uses a backup communication channel — a second messenger or a call to the shift lead. The signal is not lost when the primary application is unavailable.
How does this relate to the monitoring system?
The monitoring system remains the source of truth for metrics: it generates an event, and the on-call system turns it into an incident with priority and owner. Matching rules and labels are configured once, after which the route is determined automatically.
Let's discuss your on-call scheme and estimate the scope of work
To estimate the scope of work, we need not general wishes but the actual picture: a list of services with priorities, the current on-call scheme, and how notifications are currently distributed.
The more accurate the inputs, the fewer assumptions in the estimate — and the less often it turns out during implementation that some services were never added to monitoring.
- List of services and owners: what can fail, who is responsible for recovery, and during which hours.
- Criticality and severity classes: which failures wake people at night, which wait until morning, and what criterion distinguishes them.
- On-call scheme: number of escalation lines, rotation, time zones, substitution and weekend rules.
- Signal entry points: monitoring, application errors, support service requests, manual triggers.
- Notification channels: call, mobile notification, messenger, email — and what is considered primary versus backup.
- Incident history for the previous period: how many there were, how many were duplicated, and where they were lost.
- Constraints: access, chat retention requirements, corporate policies, and integrations that must not be touched.
In response, we will return an implementation plan by stages: routing, schedules, escalations, integrations, and validation with test failures. The scope estimate comes as a separate document broken down by stages and external dependencies. Describe the task in free form — we will clarify the details and come back with a plan.