Slack alert setup: where routing breaks
A typical Slack alert setup starts with a single incoming webhook to a shared channel where all monitoring systems post. While there are only a few services, the scheme works.
With dozens of services, the channel turns into a stream of identical messages, and the on-call engineer stops reading them — alert noise does exactly what the system was supposed to protect against.
The next breakdown is alert routing. Metrics, logs, availability, and queues each post to their own channel, thresholds are set locally, and priority is not assigned anywhere. One failure arrives as four messages, and a truly critical one drowns among warnings.
Escalation is a separate problem. If no one acknowledges a message, nothing happens: there is no rule “in 10 minutes we call the second on-call engineer.” A missed incident surfaces in the morning through user complaints rather than an alert, and diagnostics start from scratch.
Fragmented communication channels complete the picture: some notifications settle in email, some in a messenger, some are visible only in the monitoring dashboard.
The status of resolution is not recorded anywhere, so finding out who has already taken the incident and what has been checked is possible only by asking in chat.
We'll break down how to bring everything into one scheme: priorities, deduplication, time-based escalation, and a single point where the state of every alert is visible.
What changes after a unified alert and on-call bus
Before a unified alert bus, the picture is usually this: some events go to email, some to a messenger, critical ones to an on-call service, and every morning someone manually reconciles three or four channels.
What ends up missed is not what didn't fire, but what fired in a channel no one was watching at that moment.
The bus removes this reconciliation: an event has one entry point, and then routing is defined by rules — by criticality, service, time of day, and shift composition.
Event grouping merges an avalanche of notifications from one cause into a single incident, so the on-call engineer reads one coherent context instead of dozens of identical lines. This is noise reduction: only alerts requiring manual action make it into notifications.
On-call duty and escalation no longer rely on human memory. If the on-call engineer does not acknowledge an event within the allotted window, it automatically goes to the next person in the chain — that's how escalation works in PagerDuty.
Time to the on-call engineer's response is calculated not as “when someone saw it,” but from firing to acknowledgment, and this number immediately shows at which step the chain breaks.
Measurable result: the share of events brought to acknowledgment, time to first action, the number of channels that must be checked manually — and the share of false positives removed by grouping rules.
Comparison of alert routing methods
Alert delivery methods differ not by channel, but by where the routing logic lives and who is responsible for escalation.
Direct messenger integration is a webhook to a channel: it can be set up in half an hour, but if the on-call engineer stays silent, the event simply remains in the chat history.
An on-call system adds scheduling, acknowledgment, and re-paging the next person in the chain on top of delivery. A custom-built intermediary gives full control over event criticality thresholds, but requires maintenance: a silent router means silence instead of alerts.
| Option | When it fits | Limitations |
|---|---|---|
| Direct messenger integration | One channel, a small team, non-critical services | No acknowledgment or escalation; critical alerts mix with background noise; access to the channel equals access to alerts |
| On-call system with scheduling | Round-the-clock services where the fact of response matters | Dependence on an external service and its limits; non-standard routing still has to be moved outside |
| Custom-built intermediary | Dozens of sources, custom suppression and grouping rules | Maintenance on our side: updates, fault tolerance, checking routes before changes |
In practice, a hybrid is assembled: critical events go to an on-call system with escalation, informational ones go directly to the team channel. We deploy this option when Slack alert setup is needed together with nighttime calls.
How we implement routing: work stages
Alert routing breaks not when configuring an individual channel, but at the joints: an event arrives with inflated criticality, the on-call engineer receives it in a chat they don't read, and escalation goes into the void.
That's why we start not with Slack alert setup, but with an inventory of what, to whom, and with what urgency should arrive.
Work sequence
- Audit of event sources. We inventory the systems that send notifications: formats, current integrations, duplicates, and channels without an owner.
- Stream classification. We mark events by criticality and service owner: what goes to the work chat, what pages the on-call engineer, what is written only to the log.
- Routing scheme. Rules by source labels: grouping of similar events, suppression of repeats, quiet windows, escalation order by levels.
- Channel setup. We connect chats and on-call services, define a unified notification template with links to graphs and logs.
- Delivery verification. Test firings for each rule, measuring time from event to notification and to acknowledgment, checking escalation when a channel fails.
- Regulations. We fix who edits the rules, how a new service is connected, how a false positive is cleared.
- Handover to operation. We provide documentation and the scheme, train on-call engineers, review the first weeks, and adjust thresholds.
What remains in your environment: configurations and regulations
Everything described above works only when configuration lives in a repository next to the code, not in the on-call engineer's personal dashboard.
We hand over an environment with a set of artifacts: an owner is assigned to each item, a change goes through review and enters the history. Below is what exactly remains with you after implementation.
- Description of delivery channels: which types of firings go to the messenger, which go to the on-call service with escalation, where duplicates are disabled so that one event does not arrive in three places.
- Alert configurations as files in a repository: rules, thresholds, grouping of related events, and quiet windows during maintenance. Any edit goes through change review, not manually in the web interface.
- Team separation rules: mapping a service and environment to the owning team and criticality level, so a firing goes to the responsible person, not to a shared chat for the whole department.
- Escalation scheme: on-call levels, time until transfer to the next level, who receives the event if there is no acknowledgment, and what counts as acknowledgment.
- On-call engineer instructions: sequence of actions on a firing, where to look at metrics and logs, at what point to engage the second line, and how to document the review.
- On-call regulations: schedule, substitution procedure, contacts of service owners, a separate list of those not to wake at night without reason.
- Monitoring documentation: a list of checks with owners, what each firing means and what class of failures it catches, plus a list of noisy rules and the procedure for reviewing them.
- Route health check: a test firing for each channel, a note that the event reached the on-call engineer, and an entry in the check log.
Case study: how we reduced alert noise and response time
Before: one stream for all events
All monitoring firings went to the shared chat of the on-call shift, on top of that — calls and mobile notifications for any failure, even a short-lived one.
During a night shift, the on-call engineer received about 380 messages, three quarters of them repeats of the same degradation from different nodes. Critical alerts drowned in this stream and were noticed only by the next reminder.
After: routing by criticality
The causes were in the configuration: most rules had no suppression window, and each node reported itself separately.
Events of the same service and symptom are now merged into one card by a deduplication key, repeats within the window are suppressed, and recovery closes the card automatically.
Critical alerts requiring a person immediately go through PagerDuty with a call, degradations with a clear scenario go to the on-call Telegram channel, and service events for daytime review go to Slack.
The stream to the on-call engineer dropped from 380 messages to 40–60 per shift, the share resolved as false positives — from 78% to 12%. Median time from firing to being taken into work decreased from 25–40 minutes to 3–5.
Rules and thresholds live in a repository, a change goes through testing in a test environment, and rollback is one edit, so we maintain the metric rather than “setting it up once.”
How messenger on-call differs from an escalation system
A messenger delivers a message and that's where its participation in on-call ends: it doesn't know whether the engineer woke up, took the alert into work, or what to do if there's silence in response.
An escalation system manages not a message but an obligation — the alert lives until someone explicitly confirms they have accepted it.
| Criterion | Messenger on-call | Escalation system |
|---|---|---|
| Delivery | depends on the bot and channel limits | several independent delivery channels |
| Read acknowledgment | “read” mark, without responsibility | alert is taken into work until resolution |
| Overdue | notification just hangs in the chat | transition to the next person, then to the manager |
| Shift handover | manually, by agreement in chat | by schedule, with an assigned responsible person |
The key difference is read acknowledgment. A notification is considered delivered not when the engineer opened the chat, but when they confirmed taking it into work.
If there is no acknowledgment within a given interval, escalation goes to the second on-call engineer, then to the department manager. The failure doesn't wait for a person to get distracted or turn off the sound.
Shift handover is arranged just as strictly: a schedule, an assigned responsible person for each interval, a list of unresolved incidents, and acknowledgment history transfer together with the on-call duty.
In chat, this usually remains on a promise, and at shift handovers the most unpleasant alerts get lost.
In practice, a hybrid works: PagerDuty and similar systems hold escalation and accounting, while the messenger serves as a convenient delivery channel. The point of responsibility remains in the system where delivery reliability can be verified.
Common technical doubts before implementation
Doubts at the evaluation stage almost always boil down to one thing: will we have to break working monitoring, and will on-call duty turn into a stream of duplicates. Below are answers to what is most often asked before starting.
We already have monitoring set up — will it have to be redone?
No. Rules, thresholds, and checks remain where they are: the monitoring system continues to generate events, and we connect to its notifications and add a routing layer.
Roles are separated explicitly — the source is responsible for the alert's accuracy, the router for where and in what form it goes.
How do we avoid getting the same alert in three channels?
Each event gets a deduplication key by source and check name, and repeated firings are merged into one message with a counter.
Then grouping by service and a quiet window work: while the incident is open, repeats do not go out again, and after closing, one short recovery notification arrives.
How do we separate night and day streams?
The route is built on event importance and time of day. A critical firing at night goes to the on-call channel with escalation, daytime warnings are collected into the work stream, and review of non-critical items arrives in a morning digest. The same alert can have different routes for weekdays and weekends.
Sources emit events in different formats — is that a problem?
No, that's a standard part of the work. We normalize fields to a unified form and preserve the source data so that the message retains the host, service, metric, and link to the graph. The on-call engineer sees the same structure regardless of which system the alert came from.
Can we implement in stages without switching everything at once?
Yes, and that's the preferred scenario. First one channel on a non-critical environment, then migration of the remaining routes, and only after that — disabling old notifications. Rollback to the previous scheme remains available at every step.
Let's discuss your alerting and on-call scheme
To make the conversation substantive, it is enough to describe the current alerting scheme: where signals come from, how they are distributed across channels, and who resolves them.
It is useful to recall recent incidents — which notifications arrived late, which were duplicated in three chats at once, and which did not wake anyone at night.
Based on the discussion of the task, we will return with a work estimate and implementation plan: routing by criticality, escalation rules, and on-call shift schedules. To avoid guesswork, prepare:
- List of signal sources: availability monitoring, build and delivery, logs, business logic checks.
- Current alerting scheme: channels, recipients, and places where rules are set manually and diverge from one another.
- Criticality policy: what must wake the on-call engineer at night, what waits until morning, what remains in the work chat.
- Composition of on-call shifts and escalation procedure, including weekends, holidays, and substitutions.
- Constraints: access, closed environments, requirements for the log of delivered notifications.
- Control points — by which metrics you will understand that signals have stopped getting lost.
In response, you will receive a routing scheme with separation by criticality, a phased implementation plan, and a work estimate by timeline and team composition. Describe the task in a convenient format — we will return with a plan and clarifying questions.