AIOps development: why a failure is noticed only after an outage
Everyone has logs. Collection is usually configured: the application writes to a file, the platform stores records in a storage, alerts for individual metrics arrive in chat.
The problem is different — events from different services are not linked to each other, and at the moment of an error spike no one sees that this is one incident, not five independent ones.
Fragmented monitoring works like this: each service has its own threshold, its own team, its own notification channel.
An error in the queue and an increase in response time at the entry point look like two different signals, although the latter is almost always a consequence of the former. While they are compared manually, time passes, and confirmation arrives only after the outage.
Event correlation is not “collect more logs” but a model of relationships: which service calls which, which metric depends on which, which error code repeats with what frequency.
Without this model, incidents in logs remain lines that no one reads — at the current volume, it is physically impossible to parse this manually.
Hence the typical scenario: a spike is confirmed after the fact, analysis takes hours, the next similar case is caught after a crash.
AIOps development starts from the opposite — with a description of normal service behavior and rules by which a deviation becomes a signal rather than a record in an archive. Next, we will look at how this is addressed in practice.
What changes after implementing log analysis
Before implementing automatic log analysis, the on-call engineer learned about an outage either from users or from a graph that had to be manually correlated with logs from the application, load balancer, and database.
Incident detection time was measured in tens of minutes, and the share of false positives reached two-thirds — for each real event there were several empty on-call wake-ups. Service downtime was recorded after the fact rather than prevented.
After implementation, log analysis runs as a stream: events are normalized, linked by service, node, and specific request, and an anomaly turns into a single incident with already collected context. Detection time is reduced to a few minutes — an error spike is visible before users notice it.
Service observability stops being a set of fragmented panels in Grafana: metrics, logs, and traces converge into a single picture suitable for decision-making, not for forwarding between teams.
The share of false positives drops severalfold: rules rely on accumulated history, not on thresholds set by eye, so noise spikes go into suppression and do not wake the shift.
At the same time, an entire class of failures is covered — external dependency degradation, connection pool exhaustion, silent accumulation of timeout errors — which previously was diagnosed only after a crash.
The load on the on-call shift changes qualitatively: instead of manually correlating extracts, the engineer receives an incident with a log slice, chronology, and affected services.
Night investigations stop stretching for hours, and after an incident a rule remains that will trigger automatically next time.
Log analysis modes: streaming, batch, hybrid
The log processing mode is chosen based on the volume of the event stream and the acceptable signal delay.
Streaming analysis reads events as they arrive and keeps an aggregation window in memory, batch processing parses the accumulated array as a whole, and a hybrid scheme separates the hot and cold paths.
The difference shows when system behavior deviates from the norm.
The stream answers what is happening now, the batch — what actually happened: the stream catches a deviation on the rise, but some events are lost to noise, while the batch reconstructs the picture, but with a delay.
Comparison of log processing modes
| Mode | When it fits | Limitations |
|---|---|---|
| Streaming analysis | Steady constant stream, reaction needed at the moment of deviation | More resource-intensive, some events are lost to noise, the picture cannot be reconstructed retroactively |
| Batch processing | Volume accumulates, analysis can be delayed | Signal is late, re-parsing a large archive takes hours |
| Hybrid scheme | Need both a fast signal and a full archive for incident analysis | More complex maintenance: two paths and coordination of the storage scheme |
The choice rests on infrastructure. The streaming path requires headroom for concurrent connections and predictable storage response, otherwise the queue grows and the signal is late exactly when it is needed.
The batch path is limited by storage volume: without a cleanup policy, the archive is heavier than the system itself.
The hybrid scheme removes this limitation: the hot window provides the signal, the archive provides context for analysis. Both paths work on the same events, so the incident picture does not diverge from what the system saw at the moment of failure.
Implementation stages: audit, scheme, development, validation
We start not with a model but with data: as long as it is unknown what and how sources write, any automation produces noise instead of a signal. The stages proceed strictly in order — each relies on the artifacts of the previous one.
- Log audit. We collect a list of sources, formats, and write volumes, noting duplicates, time gaps, and different time zones. The output is a source map with a connection queue.
- Event normalization. We bring records to a unified schema — time, node, service, severity, text — and stitch related messages from different systems into a single incident. The fragmented stream is compressed severalfold, and losses during analysis disappear.
- Rule configuration. We describe correlation, deduplication, silence windows, and routing by owners. We take thresholds from deviation statistics, not “by eye”: otherwise the on-call shift drowns in false positives and stops reacting.
- Validation on history. We run the rules over accumulated incidents and compare with manual investigations: what was confirmed, what turned out to be noise, what was missed. This removes most false positives before launch.
- Trial run on a narrow scope. One or two services in production mode: we look at signal delay, grouping completeness, and alert readability. We expand coverage only after a stable result.
- Solution handover. We provide configurations, flow schemes, rule documentation, and the procedure for editing them, and train your team. Adding sources and parsing new rules is then handled by your engineers themselves.
What we hand over: configurations, rules, procedures
Handover is not an archive with settings but a working system that your team can develop without us. Therefore, each artifact is designed so that it can be read, verified, and changed, not just launched.
- Collection and analysis configurations — where logs are collected from, which formats are parsed, which fields are extracted, and which are discarded as noise.
- Data flow schemes — the event path from source to storage, buffering points, and system behavior when the source is unavailable.
- Correlation rules — how fragmented events combine into a single incident: grouping windows, thresholds, duplicate suppression, priorities.
- Access policies — role separation, masking of sensitive fields, retention periods, audit log of data access.
- Response procedures — what is closed automatically, what goes to the on-call engineer, who and when is escalated, where the analysis is recorded.
- Operational instructions — adding a new source, editing thresholds, analyzing false positives, typical causes of their occurrence.
- Checks and rollback plan — running rules on historical incidents, tests for missing an event, return to manual analysis without losing accumulated data.
Separately, we hand over a decision log: why the threshold is set exactly this way and what trade-offs are behind it. In six months, this saves more time than any installation documentation.
Case study: incident detection time before and after implementation
A log stream from several dozen services in Kubernetes was collected into a common storage without a unified schema, and alerts were configured manually for each service.
The on-call engineer received around two hundred notifications per day, of which only a few were confirmed: the rest were repeats and duplicates from neighboring nodes.
An incident was more often found from a user complaint rather than from a system signal: the median detection time held around twenty-five minutes, and all that time the failure propagated along the dependency chain.
What was changed
We built a unified pipeline: logs were brought to a structured form with mandatory fields — service, environment, release version, request identifier.
On top, we added correlation in a sliding window: similar errors collapse into a single incident, and a spike on a service is linked to changes in dependencies.
Thresholds are calculated from the normal background of a specific service, so isolated nighttime errors no longer wake the on-call engineer.
The bottleneck turned out to be not the model but the labeling: some services wrote unstructured text, and before normalizing these streams, analytics produced false positives.
After normalization, the share of alert noise fell from 80% to 12%, median detection time was reduced to three to four minutes, and time to localize the failing node was halved.
The failure class “external dependency degradation” began to be detected before user impact: an increase in the share of slow responses is caught as an anomaly, not as a fact of a crash.
On-call duty by alerts turned into on-call duty by incidents, and the failure history became the basis for rules that cut off new classes of failures.
How AIOps differs from classic log monitoring
A threshold rule is a predicate over a single time series: the value crossed a boundary, a notification went to the on-call engineer. It does not know that latency on a dependent call has grown nearby and that the noise source is one node out of forty. Correlating metrics, logs, and incident history further has to be done manually.
AIOps does not cancel the rule but adds context to the event: related services, recent configuration changes, behavior of the same node in past failures.
The event stops being a separate signal and becomes a link in a chain — with attributes by which it can be grouped and compared.
| Approach | What it relies on | When sufficient | Limitations |
|---|---|---|---|
| Threshold rules | metric and boundary | stable load | does not see correlations |
| Rules and tags | thresholds and labeling | dozens of services | rules maintained manually |
| Contextual analysis | context and history | hundreds of services, frequent releases | metadata and history required |
The difference is visible during analysis. The threshold approach gives the on-call engineer a scattering of notifications about one failure; contextual analysis brings them into a single incident, shows the probable cause and a similar past case.
This works when telemetry arrives with the same attributes — service, owner, release; a unified collection schema like OpenTelemetry removes manual normalization.
The boundary between approaches runs not by log volume but by the presence of a model of normal behavior and relationships between services.
While there is no history, thresholds work; when metadata is filled in and incidents are accumulated, context work begins — and that is the subject of AIOps.
Technical concerns before starting a project
Before starting a project, engineers and tech leads ask roughly the same questions — and this is more useful than any presentation. Below we have collected those that most often slow down a decision.
Almost none of them require that the infrastructure be put in order in advance: log collection, access, and incident labeling are part of the work, not an input condition.
Logs are written carelessly — will analytics have nothing to latch onto?
Records in different formats are a normal entry point: we identify mandatory fields (time, service, level, request identifier) and bring formats to a common schema. Whatever is missing, we configure on the collector side without rewriting applications.
How much data will have to be stored?
Enough for incidents to have time to repeat. We keep hot logs in a searchable storage, the archive — in compressed form; we set the retention rule by volume and value of records, so the registry does not grow linearly.
What access to our systems is needed?
Read-only access to logs and incident metadata, a separate service account with a limited key lifetime, masking of personal data at the input. No rights to modify the production environment are required.
How will this integrate into our systems?
At the input — a log collector and an incident management system, at the output — events to the on-call system and tickets to the tracker. Everything through open interfaces; the tools familiar to the team remain in place.
How long does implementation take?
The first working path on one or two services takes weeks; we connect the rest of the landscape in iterations. The timeline is driven not by models but by access approval and labeling of historical incidents.
Questions about rollback, isolation, and handover of the solution to the team are covered below.Let's discuss your task: what is needed for assessment
We start task analysis with initial data: the more accurately you describe the current path and the nature of failures, the fewer assumptions we build into the project estimate. You do not need to send full extracts — samples of records and answers to a few questions are enough.
- Log sources: applications, load balancers, orchestrator, DBMS, monitoring systems — what is already being written and where it goes.
- Formats and volumes: structured records or text files, daily volume, retention period, whether rotation is configured.
- Infrastructure description: production and test paths, where components are located, whether there are restrictions on external services.
- Incident history: for what period data is available, how often failures repeat, how much time localization takes.
- Current process: who is on call, by what rules alerts trigger, and where notifications arrive.
- Integrations: ticket management system, on-call shift channels, automatic recovery tools.
- Access and requirements: in which environment work is allowed, whether a closed perimeter is needed, how rights are separated.
This set is enough to come back with an estimate and work plan: a breakdown by stages, team composition, and result checkpoints. If some data is closed, we will propose an option with step-by-step clarification — describe the task, and we will come back with a calculation.
