Manual incident triage vs automatic RCA
The on-call shift receives a stream of alerts, most of which stem from a single cause that generated dozens of triggers.
An engineer manually filters alert noise, cross-checks metrics with logs, and only then gets to the essence — time to root cause grows with the number of services, while degradation spreads across dependencies.
Another obstacle is fragmented data: metrics, event logs, traces, deployment history, and tickets live in separate systems with their own timelines.
Correlating them for a single incident can only be done manually, and each new system adds another tab to the triage chain — while some events have already been evicted from storage by the time of triage.
The visible relationship between services is also often missing: the dependency map is either outdated or held only in senior engineers' heads.
Grafana dashboards show graphs, but a human reconstructs causal links between adjacent services — from past shift experience, not from data.
Automatic RCA closes exactly this gap: the alert stream collapses into one incident, events from different sources are aligned on a common timeline, and the dependency graph is built from actual traffic between services. Below we explain how it works and what changes for the on-call shift.
What automatic RCA delivers: the effect in metrics
Automatic RCA reduces not the number of failures but the time during which the cause remains unclear.
The on-call engineer receives an alert and manually stitches together metrics, logs, traces, and deployment events — and until the cause is found, they cannot decide what to roll back or where to shift traffic.
The median time for such localization is tens of minutes, and throughout that time the outage continues.
The foundation is a unified observability loop: signals converge into a single model where OpenTelemetry links the symptom to a specific call via an end-to-end request identifier.
The system compares the current service profile with its own behavior in previous periods and points to the component that first deviated from normal — the cause, not the effect. The engineer joins triage with a ready hypothesis.
There are four measurable effects:
- Time to localization: tens of minutes become minutes — by the time the incident opens, the "release — service — dependency" chain is already built.
- Share of automatic diagnostics: 60–80% of incidents are closed with a ready explanation, the rest go to an expert with collected context.
- Recurring incidents: each triage leaves behind a check that catches the same failure class before it manifests in prod.
- Load on on-call engineers: zero-based triage is replaced by hypothesis confirmation, shift handover takes minutes.
Downtime cost here is measured in minutes, not hours, and shrinks with each iteration. The triage base accumulates: the share of incidents the system explains from known patterns grows.
Diagnostic modes: rules, historical learning, hybrid
No single mode replaces the others: they cover different data maturity and different cost of error.
Correlation rules work from day one — engineers describe dependency topology and service degradation order, and the system assembles a root-cause hypothesis from a template without statistics.
| Mode | When it fits | Limitations |
|---|---|---|
| Correlation rules | There is a dependency map, few closed incidents | Breaks on unknown links, requires manual maintenance with every architecture change |
| Historical learning | Hundreds of triaged incidents with confirmed root cause have accumulated | Accuracy drops on rare failure classes new to the dataset |
| Hybrid mode | Both topology and history are available | Harder to triage: it is not always clear what produced the hypothesis — a rule or a model |
| Operator hints | Cost of error is high, decision remains with the human | Triage speed is limited by on-call engineer actions |
Accuracy limitations are more honestly measured retrospectively: run automatic RCA over closed incidents and calculate the share of cases where root cause was in the first three hypotheses.
Hybrid is usually taken as the target — rules cut off the known, the model fills in the non-obvious, and disputed cases go as operator hints with a confidence flag so the on-call engineer can see where the system doubts.
Switching mode does not require rebuilding telemetry collection — only the hypothesis output layer changes. We start with rules, history for learning accumulates on them, and transition to hybrid is one configuration step.
Implementation stages: from data audit to handover to operations
Automatic RCA starts not with a model but with signal inventory: root-cause analysis is possible only where logs, metrics, traces, and deployment history align in time and object.
Therefore, the work order is structured so each next stage relies on an artifact from the previous one — the output is not a demo but a working loop and a shift that maintains it.
- Data source audit — what is already being written and with what granularity: application and storage logs, metrics, traces, deployment events. We record retention periods, time zones, and gaps that break the "cause — effect" chain.
- Service relationship map — dependency map: who calls whom, through which queues and databases, where shared failure points and external providers are. The map sets hypothesis checking order and is updated with architecture.
- Rule configuration — event correlation, thresholds, priorities, suppression of recurring noise, service owner directories. The rule set is versioned: any change can be explained and rolled back.
- Validation on historical incidents — run against past triages: share of cases where root cause is correctly indicated, false positive rate, time to hint. Additionally check on training failures absent from history.
- Embedding into response loop — hypothesis reaches the on-call engineer when the incident opens, together with rationale: which events and links confirm it.
- Pilot operation — system works in hint mode, on-call engineers mark correct and incorrect conclusions, and rules and thresholds are adjusted based on these marks.
- Handover to operations — triage regulations, shift instructions, roles and access scheme, rollback procedure, and dependency map update order.
Artifacts of each step — source report, relationship map, rule set, validation protocol — remain with the customer.
Implementation artifacts: model, rules, documentation, regulations
Work on an automatic diagnostic system ends not with model launch but with a set of artifacts that remain in your environment.
At acceptance we check it as a whole: without a dependency map the system confuses primary failure with cascade; without regulations the on-call engineer doesn't understand which signal to trust and what to do with a false positive.
- Model and correlation rule configurations — sensitivity thresholds, event aggregation windows, exception lists; with notes on what exactly changes when each parameter is edited.
- Service dependency map — who affects whom and which nodes are shared; the system uses it to distinguish root cause from failure cascade.
- Access policies — roles for viewing incidents, running checks, and changing rules, linked to your account directory.
- Documentation — data flow diagram, connection points to monitoring and ticketing system, component update order.
- Response regulations — which signals wake the on-call engineer at night, which go to morning triage, who approves rule changes.
- Shift instructions — step-by-step scenarios for typical incidents and actions when the system is uncertain about cause.
- Reference incident set — run after each rule change to see whether accuracy on past cases has dropped.
We hand over artifacts in your format: in the repository next to the infrastructure description, so changes go through normal review. After handover you change thresholds, expand the service list, and update instructions yourself — no involvement from us required.
Case study: incident triage in a high-density event stream
High-density event stream — hundreds of thousands of records per day from applications, load balancer, and network.
The on-call shift could not keep up with correlating them, so triage began with manual search: what failed first, what changes preceded it, which dependencies were affected. Median triage time stayed at 50 minutes, and some incidents were closed by restart without a named cause.
Baseline
Over three months of observation, the share of incidents with a ready cause explanation was practically zero — the conclusion rested entirely on who was on shift.
Recurring failures of one class reached 17 per month: the cause was suppressed but not eliminated, and after a few days it returned in a different formulation.
What was changed and what changed
The automatic diagnostic layer on top of OpenSearch builds a chain from symptom to source: correlates error spike with latest changes, checks dependency state, and cuts off coincidences in time. The engineer receives ranked cause versions with confirming events, not a log dump.
Median triage time — 6 minutes, fully automatically explained result for 70-80% of incidents, the rest arrive with an already narrowed search area. Recurring failures of the same class — 4-5 per month.
The most expensive class — cascades during degradation of a shared dependency — is recognized automatically and goes to triage first.
How does event correlation differ from root cause search?
Event correlation works at symptom level: the system collects alerts in a time window, collapses them by common labels — host, service, availability zone — and gives the on-call engineer one incident instead of hundreds of messages.
This removes noise but does not answer "why": cause and its consequences equally fall into one group.
Root cause search works with the failure chain. It goes from symptom upward through the dependency graph and looks for the node whose state change preceded the others: the root explains all observed symptoms but is not explained by any of them. This requires service topology, change history, and metrics in a unified time.
Cascading failures are the main source of correlation errors. Storage degradation pulls the queue, the queue pulls timeouts on the upper service, and all three events fall into one window.
The loudest alert is usually at the upper level, where the failure is most visible, while the root is two steps lower. Time-based correlation does not distinguish such links and confidently leads the on-call engineer in the wrong direction.
The difference shows in metrics: correlation reduces the number of alerts per on-call engineer, root cause search reduces time to localization and the share of incidents with correctly indicated cause.
One does not replace the other: if the wrong node falls into the group, savings on noise turn into extra hours of triage.
The boundary in practice is simple. A single signal is closed by an alert, symptoms spread across services by grouping, and a causal link through a cascade requires chain reconstruction. Otherwise recovery follows the symptom, not the source.
Technical concerns: data, accuracy, false positives
The engineering team's concerns are usually the same: is there enough history, what to do with dirty telemetry, will we drown in false positives. Let's look at how each is addressed in the system.
How many incidents are needed for the system to work?
For the first scenarios, a few months of labeled incidents are enough — automatic RCA looks for recurring relationships between events rather than memorizing everything.
Then the model is further trained on the current stream, and quality improves without rebuilding history from scratch.
Telemetry is incomplete, timestamps diverge — is that a dealbreaker?
We start with source audit: logs, metrics, traces, unified time, deduplication and duplicate removal. We do not take a scenario for which data is insufficient as first priority — instead of guessing, the system honestly shows it could not narrow the scope.
How do you limit false hints?
A hint goes to the on-call engineer only above the confidence threshold and always with the event chain it was built on. We calculate the share of confirmed root causes: if it drops, we investigate which signals are noisy and adjust rules and features.
How phased will implementation be and who is responsible for quality?
First an observation mode: the system proposes a version but changes nothing in the loop. Then one incident class, then expansion. We agree on the share of correct root causes and time to hint before start — we are responsible for these metrics.
Let's discuss your task: we'll design a diagnostic scheme for your stream
We suggest discussing the task concretely: we need not a general story about pain but a picture of your incident stream. From it begins the scope estimate and a realistic implementation plan — by stages, with a clear result at each.
What we request from your side:
- access to monitoring data for read — metrics, logs, and traces for several months, including periods of major failures;
- list of services and links between them, to correlate symptoms with the actual cause, not with the nearest noisy signal;
- export of incident history with marks about what turned out to be the cause — on it we check where rules produce false positives;
- description of the flow scheme: how an event travels from source to on-call engineer and at what point context is lost;
- your accepted escalation rules and criteria by which an incident is considered closed;
- an engineer on your side for validation — an hour or two per week for data clarifications.
In response we send a scope estimate, stage-by-stage implementation plan, and preliminary diagnostic flow scheme: where and what data is needed, what we validate in the pilot area, what rollback looks like if rules are wrong. If some data is missing — that's not a blocker, we'll discuss what to replace it with.
Describe the task in free form — we'll come back with an estimate and plan.
