Why manual log analysis misses anomalies
Manual log analysis runs into human throughput: an on-call engineer scans the event stream with their eyes and holds a few recent lines in their head, rather than the connections between services over the past hour.
Some missed incidents are not carelessness but physics: there is no way to compare a service's behavior across two different hours without relying on metrics.
An anomaly almost never looks like a single red line. It is a rise in the error rate on a specific route relative to the baseline, a spike in timeouts for one consumer, or a series of request retries.
In the overall stream, such records are few, while routine ones are orders of magnitude more numerous, so the rare signal drowns in noise.
Manual analysis of events for one service does not reach the cause: it lives in a neighboring one — in a queue, in a connection pool, in the response of an external system.
Time to detection in this scheme is determined not by monitoring, but by the first user complaint. While an incident has not reached support, it accumulates.
Industry data collected by DORA shows: teams that find failures from system signals rather than from reports recover many times faster and less often return to the same failure again.
We'll break down how this is addressed: which event stream should be considered normal, where the boundary of noise lies, and how alerting starts firing before the user does, without turning into a stream of false positives.
What automatic log analysis provides instead of manual checks
Manual analysis scales linearly: the more services and sources, the more spot checks. An engineer looks at what they have time for, and the anomaly lives between shifts — hours pass from the first failure to detection. Continuous observation instead of sampling moves detection speed to minutes.
The model builds a behavior baseline separately for each source and compares the current stream against it, accounting for daily and weekly peaks.
Thresholds are calibrated using already analyzed incidents, so the false positive rate drops to single-digit percentages — the on-call engineer stops treating alerts as noise.
Elasticsearch remains the log storage: the analytics layer reads the same stream, and no duplicate collection appears.
Source coverage grows without rewriting rules. A new service is connected as a separate stream, its behavior profile is built from accumulated history — systems that manual checks never reached also fall under observation.
The load on on-call engineers decreases due to two things: routine work goes to automation, and signals arrive grouped by incident rather than as separate lines.
The on-call engineer receives context — what changed and which nodes were affected — and investigates the cause instead of manually assembling the picture.
Anomaly detection modes: comparison of approaches
The detection mode sets the main trade-off: how many real failures the system will miss and how many false positives it will send to the on-call engineer.
There is no universal option — the choice depends on the stability of the log format, the availability of labels from past incidents, and the cost of a miss. Below are the approaches to choose between at the start.
In practice, modes are combined: threshold rules cover critical metrics, baselines cover seasonal spikes, message clustering covers text noise. A hybrid detection scheme brings signals from all layers into a single stream and noticeably reduces the false positive rate.
| Option | When it fits | Limitations |
|---|---|---|
| Threshold rules | A metric with a clear norm: queues, errors, latency | Manual tuning for each service, breaks on seasonal peaks |
| Statistical baselines | A steady stream and a repeating load profile | Requires history for training, reacts slowly to a sharp change in behavior |
| Message clustering | Semi-structured text, many noisy templates | Requires normalization, clusters lose meaning when the log format changes |
| Unsupervised models | No labels from past incidents | Show an anomaly without a cause; analyst investigation is needed |
| Hybrid detection scheme | A heterogeneous fleet of services and different log types | Longer implementation, requires configuring connections between layers |
The order is as follows: first we assess the predictability of the stream and the quality of labels, then assemble a minimal set of layers — and only after validation on historical data do we expand the scheme.
This way the system goes into operation with controlled sensitivity, rather than with a stream of alerts that the on-call engineer starts to ignore.
How we implement the system: stages of engineering work
The work proceeds in iterations: first we assemble a complete picture of sources and streams, then we build the processing pipeline, and only after that do we enable alerts.
This order is needed so that rules are born from real records, not from assumptions about how the system is structured.
- Source audit. We go through applications, services, databases, queues, and network equipment: recording log formats, occurrence frequency, daily volume, fields, and access owners.
- Stream diagram. We describe the path of an event from the collection agent through the buffer and transport to storage, marking areas where losses and delays are possible.
- Event normalization. We bring heterogeneous records to a unified schema — time, source, level, request ID, text. Duplicates and service noise are filtered out here so that both search and rules work on the same fields.
- Configuring rules and models. We set thresholds, connections between events from different services, and baseline profiles of normal behavior for each node.
- Validation on historical data. We run the rules against accumulated logs and compare the result with what on-call engineers remember as real failures: this is how sensitivity is tuned and false positives are removed.
- On-call engineer training. We go over alert labeling, priorities, the escalation procedure, and actions to take if the analyzer itself is unavailable.
- Handover and support. We provide documentation, access, and a procedure for changing rules so the team can develop the system without us.
What we hand over to the customer based on project results
A log analysis project ends not with a monitoring dashboard demo, but with a set of artifacts your team can work with further without us.
Each item is a file, diagram, or document, not a description of "how it was intended." We hand it over to your environment and walk through it on a call: where a rule lives, what happens if a source fails, how to expand for a new service.
- Collector and parser configurations for each source format, including multiline records and non-standard lines. A new source is connected by copying a profile and editing fields, without code changes.
- A stream diagram from source to alert, with buffering points and behavior when the receiver is unavailable. It shows where an event can be lost and how that is tracked.
- Rules and thresholds with justification: for each one, it indicates which historical data it was tested against and what false positive rate it produces. The threshold is changed in configuration, not in code.
- Event access policies: role-based separation, masking of sensitive fields at storage ingress, access log. Access auditing does not require a separate export.
- On-call schedule procedure: criticality levels, escalation procedure, response time targets, actions during a spike of similar alerts.
- Instructions for connecting new sources — a step-by-step checklist that events arrived and were parsed correctly.
Case study: how we reduced incident response time
The client is a product service with a distributed system: dozens of applications, a load balancer, a queue broker, a database, and orchestrator events.
Logs from all environments were collected into a single storage, but they were analyzed manually and only after user reports — that is, retrospectively.
Before: incidents were learned about from support
Alerting was based on threshold rules for individual metrics: either nothing fired, or dozens of notifications arrived, of which only a few were useful. On-call engineers disabled some rules by raising thresholds — the noise went away along with the sensitivity. The average time from the first anomaly in the logs to an engineer's reaction was about an hour.
The most expensive issue was not loss of availability, but slow degradation: growth in database response time and connection leaks accumulated over hours, and users noticed it before monitoring did.
What we changed and what we achieved
We connected nine sources: application logs, load balancer, broker, database, service journals, and cluster events. Parsing and storage were built on Grafana Loki and a model that builds a profile of normal behavior from message templates and frequencies.
The signal becomes a deviation from the profile, not exceeding a static threshold, so the alert arrives with ready context: service, template, window, and how many times the frequency increased.
Time to detection dropped from 55 minutes to 4 — roughly twelvefold. The false positive rate fell from 78% to 15%: the on-call engineer stopped sorting through noise and restored rule sensitivity.
The failure class "gradual degradation of database responses and connection leak" stopped reaching users — the anomaly is caught from latency growth, before the first mass errors.
What to do if logs are already collected but not analyzed?
Log collection and log analysis are different tasks, and the first is often completed long before the second. Agents write to storage, rotation is configured, volume is growing, but inside is flat text that can neither be grouped into records nor used to build a metric.
Below is what can be built on top of existing storage and what runs into limitations at the sources.
On top of storage, a parsing layer is added: template-based rules, field extraction — severity level, component, request ID, response code, operation duration — and bringing events to a common schema.
Collection is not reworked, and history is run through the same parsing retrospectively: you immediately see where an anomaly already occurred and how long it has been recurring.
At the source level, rework is needed where the message carries no context: "connection error" without a node, request, and end-to-end identifier produces formally populated but useless fields.
This is fixed by changing logging in the services themselves — a unified log format, a mandatory set of fields, a trace ID, and separation of severity level and text. The amount of change is small, but it is exactly what determines alerting accuracy.
Message order is also important: if an event is glued together from several lines or written asynchronously, the sequence is lost during parsing, and a spike in errors after a version update does not come together as a single incident.
Grouping by request ID and time window partially addresses this; writing the event as a single record is more reliable.
The work proceeds as follows: inventory of formats by source, parsing and structuring events on top of storage, targeted logging changes, then detection rules and alerting.
The result metric is the share of events parsed into fields: when it is above 95%, alerts are built on structure rather than substring search, and false positives become many times fewer.
Technical concerns before starting work
Concerns at this stage are usually engineering-related: how to connect another source without rewriting processing, and what happens if the event stream grows severalfold. Below are direct answers to what is most often asked before work starts.
How is a new source connected?
With an adapter at the input: we parse the format, bring fields to a unified schema, filter out duplicates, and bind records to a service and owner. After that it is configuration, not model rework.
Before enabling it in the production stream, we run accumulated history through it and look at what alerts the system would have produced — comparing with the current picture shows the cost of an error in advance.
How do you deal with false positives?
We configure thresholds on history, not by eye: rules filter out known noise, and the model catches deviations from a service's usual profile.
Repeats of the same event are grouped into one alert, and the suppression window prevents the on-call engineer from receiving an avalanche. We separately calculate the share of false positives among all alerts — it shows whether the feed can be trusted.
What about sensitive data?
Masking happens on the source side, before sending: personal and payment fields do not enter parsing. Data remains in your environment, access is role-based, and requests are logged. Analyzing an incident does not require exporting raw events outside.
Will the system withstand a sharp increase in volume?
Ingestion scales horizontally, and spikes are absorbed by the buffer. Critical services have priority: secondary sources are sampled down, and the system explicitly reports degradation rather than silently losing events. We verify this with a load test before launch, not during an incident.
Let's discuss the task and come back with an estimate and plan
To make the timeline estimate accurate rather than "by eye," we need context on your current logging landscape.
Tell us where records are collected from — applications, load balancers, databases, queues — what formats are used, how many sources are involved, and what daily volume passes through. It is also useful to understand how notifications are currently set up and where they go.
Describe the nature of the failures that need to be caught: rare error spikes, gradual latency growth, anomalies for an individual user or node. The more specific the examples of past incidents, the more accurately we will select detection methods and alerting thresholds.
If there are historical logs and "normal / anomaly" labels for at least part of the period, this noticeably shortens model tuning and reduces the false positive rate.
In response, we send not a one-line commercial proposal, but a stage plan with checkpoints:
- a diagram of log collection, storage, and rotation tied to your infrastructure;
- a list of sources and formats that will be included in analysis in the first step;
- the composition of the detection model and alerting rules: thresholds, windows, priorities;
- a work calendar: pilot on one service, expansion to the rest, and production rollout;
- the order of integration with notification systems and on-call schedules;
- acceptance criteria — false positive rate and time from event to alert;
- a list of access, environments, and data needed from your side.
Describe the task in free form — we will come back with a timeline estimate, stage plan, and areas of responsibility.
