Why log-based metrics diverge from reality
Logs are being written, dashboards are green, and the user is the one reporting the failure — this is not a rare case, but a gap between event recording, their aggregation, and alerting rules.
Metrics built on logs (log-based metrics) inherit all the defects of the input stream: if an event is not described structurally or is filtered out at the collection stage, it will not make it into the counter, the graph, or the alert.
The first gap is at the recording stage. Some errors are caught in handlers and suppressed, some go to the container's standard stream without a level or code, and multi-line stack traces break apart into separate entries.
As long as an event has no mandatory fields — service, level, error code, request identifier — the failure counter is computed by substring, and missed errors in production are baked into the design itself.
The second gap is at aggregation. Sampling, short retention, and high label cardinality produce the opposite effect: rare but severe failures drop out of the sample, and single values do not form statistics.
The aggregation window smooths out the spike, the average latency stays flat, while the percentiles have already drifted upward.
The third is at alerting. A rule built on the average over a long interval fires after everything has already recovered, or does not fire at all.
Without grouping by error signature, the same exception turns into an avalanche of notifications, and the team stops responding to them.
Here is how this gets closed: a single event schema, from which both the record and the metric are computed from one field; failure counters by code and service with thresholds on spikes rather than averages; latency histograms broken down by route.
Plus regular reconciliation — the number of errors in the metric is compared with the number of events in the log over the same interval, and the divergence between logs and metrics becomes visible earlier than the user.
Real-time error monitoring relies on the stream, not on a snapshot, which is why it keeps an incident in view from the very first second.
Log-based metrics: measurable effect for operations
Manual log reading holds up until the first large service: an engineer goes into the stream based on a complaint, filters matches by substring, and reconstructs the timeline after the fact.
Error counters from logs change the order of work — first the capture rule, then the number, and only then the analysis of specific records.
The rule describes a message pattern or logging level, grouping by response code, endpoint, and release version, an aggregation window of 15–60 seconds.
The output is a series showing the share of requests with errors and the latency distribution, with no changes to the service code: the fields with the code and processing time are already in the record.
We collect and store the series on Grafana Loki, so service observability does not require separate instrumentation of every handler.
Time to detection and recovery
Time to detection is set not by reading speed, but by the moment when the condition has held long enough.
An alert on exceeding a threshold over a window arrives earlier than a user complaint, and the latency quantile per endpoint shows degradation even before the failure count grows.
Recovery is sped up by linking metrics to records: going from a graph to the logs of a specific window and release version is one step. The cause — a dependency slowdown, a configuration error, a disk failure — is visible by grouping rather than by scanning rows.
Alert noise and the cost of false positives
A threshold on the absolute number of errors is noisy: during the day a spike comes from load, at night from a single recurring client.
We compute the share of total request volume and hold the condition for 5–10 minutes, merging repeats by message pattern: the share of noisy firings drops severalfold, and the on-call engineer stops ignoring alerts.
Separately, we close the class of failures that previously reached the user — errors in background jobs, deferred operations, and integrations without an incoming request.
They are almost invisible in tracing, but in logs they produce a counter, and the regression is caught before the client notices it.
Comparing approaches: streaming counters, batch, and hybrid
The collection mode determines not only the latency of metrics, but also which failures we see at all. Streaming log processing computes counters and percentiles on the fly, batch aggregation rolls up journals on a schedule, and a hybrid collection scheme combines both loops.
The choice between them is made not by taste, but by the required reaction time and storage resources.
| Option | When it fits | Limitations |
|---|---|---|
| Streaming counters | Error and latency alerts, real-time incident analysis, windows in minutes | Higher load on the collector; unique labels (request ID, user) bloat the index |
| Batch aggregation | Reports, long windows, rare events, historical slices | Metric latency of several minutes; short spikes get averaged out and disappear |
| Hybrid | Production loop: thresholds in the stream, heavy analytics in batch | The two loops must be reconciled by formulas, otherwise the numbers diverge |
When streaming mode is excessive
If a decision is made once a day — nightly summaries, capacity planning, breaking down error shares by service — the stream only consumes resources. The collector keeps state for every event stream, and no one uses the accuracy on a short window.
Such tasks are better given to batch, while the stream is left for thresholds where reaction within minutes matters.
What is lost with batch aggregation
Batch averages: an error spike within the interval collapses into a single number, and from it you cannot tell whether the failure lasted two minutes or stretched over an hour.
Extremes — p95 and p99 latency — are not recoverable from averages; they require histograms that preserve bucket boundaries. That is why in a hybrid, the latency tail and failure counters are computed in the stream, while heavy slices are done in batch.
How we implement error and latency monitoring
We build log-based metrics from the sources: without a full inventory, the collection scheme leaves blind spots, and the numbers diverge from reality.
Then we assemble the event stream, define aggregation rules, and validate everything on live load — only after that do we hand the scheme over to operations.
- Audit of log sources. We go through applications, services, load balancers, and queues: we record entry formats, time zones, and daily event volume.
- Event collection scheme. We determine where the collector sits, what is buffered on the application side, and how the stream survives a connection drop or node restart.
- Aggregation rules and labeling. We extract errors and latencies from the stream and tie them to the service, version, and environment — otherwise graphs mix different loops.
- Thresholds and alerting. We fix what counts as degradation and what counts as a normal spike: the rule fires on a deviation, not on a single failure.
- Handover to operations. We deliver the stream scheme, rules, dashboards, and instructions for adding a new source.
Validation under peak load
We test not on a staging environment but on production traffic: we run a peak profile and check whether the collector keeps up, whether events are lost during a spike, and whether the metrics match a reference count from raw records. If the divergence exceeds the tolerance, we rebuild the aggregation rules until the numbers match.
Limitations and failure points of the scheme
The scheme has a limit: under a tenfold spike the collector falls behind, and a wide aggregation window smooths out short but important failures.
That is why we spell out which data we store in detail, which in rolled-up form, and which failures the scheme fundamentally will not catch.
Implementation artifacts: configurations, schemes, regulations
Implementation ends not with launching a dashboard, but with handing over a kit that your team can use to run monitoring on its own.
The artifacts live in the repository next to service configurations, so changes to metric extraction rules go through review and remain in the change history.
We hand over:
- Metric extraction rules from logs: parsing templates, field list, aggregation windows.
- Data flow schemes: from the application to the metric store and the dashboard, with failure points.
- Dashboard sets: error share by route, latency at the tails of the distribution, stream volume.
- Metric dictionary: what each value means and which log line it comes from.
- Instructions for adding a new metric — from a log entry to a graph, with a worked example.
- Reference logs with the expected result: the configuration can be run locally and compared.
Access policies and roles
Access is granted by groups, not individually: a developer sees the logs of their service, the on-call engineer sees all services in the loop, and exporting raw records is a separate role. Policies are described declaratively, so a new service inherits permissions automatically. Personal data masking is configured at the parsing stage — it never reaches the metric.
On-call and escalation regulations
The regulations specify who responds to a firing, within what time they acknowledge it, and at what point they bring in the second line. Alert thresholds are described in text next to the configuration, and a change goes through the same review as code.
After every postmortem, a check is added to the set that catches this class of failure at an early stage.
Case study: how we removed noise and sped up failure detection
Starting point and constraints
A service of several dozen containers, with alerts built on regexes over log text. Per shift, the on-call engineer received several dozen firings, and most of them were normal situations: request retries, client-side connection drops, isolated timeouts.
The useful signal drowned, and reaction time to real degradation was measured in tens of minutes — until the first user complaint.
The second problem was that raw logs were written in full, without parsing into fields, so they could not be used to group errors by code and method or to build duration distributions.
We labeled the entry as a structured event and learned to extract counters and histograms from it: on the Prometheus side they became ordinary time series with labels by error code, method, and service, without a request ID in the label — otherwise cardinality blows up the storage.
What we achieved in metrics
| What we measured | Before | After |
|---|---|---|
| Alert noise | 7 out of 10 firings false | 1 out of 10 |
| Degradation detection | from a complaint, tens of minutes | from a graph, under a minute |
| Incident analysis | manual text search | slice by error code and method |
| Storage volume | all logs in full | aggregates plus a sample of slow requests |
The rules were rewritten from the symptom, not from a line in the log: a firing triggers on growth in the share of server errors over a window and on a shift in the duration distribution, not on every message.
The number of pages per shift fell severalfold with the same coverage, and rolling a rule back to its previous formulation takes a single configuration change.
How does streaming log processing differ from batch?
Streaming processing computes a metric as log lines arrive: the aggregator keeps a sliding window and emits a result every few seconds.
Batch aggregation reads the already accumulated array for an interval and recomputes windows on a schedule — once every five minutes or once a day.
Hence the main divergence — latency versus counting accuracy. A streaming window shows an error spike and a rise in response time within tens of seconds, but events that arrive late fall into the next window: the total for a closed interval drifts slightly.
Batch recomputation sees the full set of events and gives an exact number, but you learn about service degradation after users have already noticed it.
Resilience to event loss also differs.
A streaming collector records the read offset and, after a failure, continues from the last checkpoint — at most an unclosed window is lost; when the receiver is overloaded, backpressure or partial stream dropping kicks in, and this shows up as a gap in the graph rather than a silent error in a report.
Batch reprocessing is idempotent: the interval is recomputed from the primary source, the divergence is reduced to zero, but recovery takes longer.
Storage volume differs by an order of magnitude. A streaming scheme stores only aggregates by labels — for example, Prometheus with counter series — and growth is limited by label cardinality.
Batch aggregation needs raw logs for the entire recomputation period: volume grows linearly with traffic and retention period.
In practice, the modes are combined: streaming metrics handle fresh alerts, and a nightly batch reconciliation serves as the source of truth for reporting.
Cardinality, noise, and event loss: common concerns
Concerns about log-based metrics almost always come down to four things: whether labeling will blow up the metric storage, whether the on-call engineer will drown in alerts, what happens to events when the connection drops, and how long all of it must be stored. We answer each point the way we handle it in real production loops.
Is a label with a request ID or user ID the end?
Not the end, but it must not be done: the number of time series grows linearly with traffic, and the metrics collector is the first to fall over. We move such values into separate log fields and turn only bounded sets into labels — response code, service name, method. High-cardinality samples are computed by querying logs on the fly rather than stored as series.
How do we remove alert noise without missing real degradation?
A threshold on the absolute number of errors is noisy against the daily load profile. We set rules on the error share and on a sliding window, add a requirement that the deviation hold for several minutes in a row, and group notifications by service and failure type. The on-call engineer gets one alert with context instead of a stream of identical ones.
What happens to events when the connection to the collector drops?
The agent buffers the stream to disk and delivers it after recovery, so short network outages do not lose data. Metrics from logs are resilient to redelivery: a duplicate event does not double the counter.
For critical paths we add a check on the divergence between the number of requests in the application and the number of parsed events.
Will logs have to be stored forever?
No. We separate the hot and cold tiers: full text for a short period, while metrics and aggregates over them live for years, because they take up orders of magnitude less space.
This makes it possible to analyze an incident from six months ago, when the logs themselves have already been moved to archive.
We already have monitoring deployed — will everything have to be redone?
Usually not. We connect to the existing log collector and add parsing rules on top of the current format, while migrating dashboards and alerts gradually. The old metrics keep working until the new ones confirm a match on historical data.
Let's discuss your task: what to send for an estimate and plan
To come back with a work estimate and implementation plan rather than generalities, we need a minimum of inputs about your infrastructure. The more concrete the input, the fewer assumptions we have to remove during setup and acceptance.
- List of sources: which services write logs, in what format, how many records per day, and how this volume grows under peak load.
- Current collection: what collects logs today — agents on nodes, container output, or files on disks — and where they are stored.
- Target metrics: which errors and latencies are needed on the dashboard and by what features to extract them from message text; what is already structured and what will have to be parsed with templates.
- Access: test and production loops where the collector and parsing rules can be deployed, and who on the customer side approves changes.
- Storage and privacy: lifetime of raw logs, whether a return to source records is needed for incident analysis, and personal data masking requirements.
- Alerts and on-call: which thresholds count as an incident, where notifications go, and who handles firings at night.
- Acceptance criteria: which indicators confirm that the metrics match real service behavior rather than noise.
In response, we send a stage-by-stage work estimate, an implementation plan with the order of connecting sources and control points, as well as the composition of metric extraction and alerting rules.
Documentation handover is a separate item: dashboard descriptions, message parsing logic, instructions for the on-call shift, and the procedure for changing rules without us. Describe your task in free form — we will come back with an estimate and a plan.