Text logs are unreadable: what structured logging solves
An incident looks like this: some requests to a service respond with an error, and you need to understand what differs between successful and unsuccessful calls.
Logs exist, but each error is written as free text, and the same cause looks different — “failed to process order for user 42, timeout” and “order 42 failed: timeout.” You have to sort through this by eye: an engineer pulls a daily archive, runs it through regular expressions, gets thousands of matches, and manually filters out neighboring events.
The effort of such work grows with volume. The archive contains gigabytes of lines, the search pattern does not cover all formulations, and the service response time and error code have to be extracted from the text with separate expressions.
Incident analysis from logs takes hours, recovery drags on, and some conclusions remain guesses: grouping records by error code or calculating the share of slow requests from such a file is technically possible, but in practice no one does it.
Structured logging changes the event recording format: instead of a string, it is a set of fields with fixed names and types.
The timestamp, level, service name, error code, request ID, and trace ID are stored as separate values, while arbitrary text remains in a single message field.
Searching application logs turns into a query with a field filter: “all records for this trace ID,” “all responses with code 500 for the interval,” “share of requests longer than the threshold per route.” The collector reads fields directly, builds graphs and alerts, and does not parse strings.
Unreadable logs stop being one engineer’s problem — they become data by which the cause is calculated. Below we discuss how this is handled in the application and in the collection pipeline.
Effect in operations: incident analysis, noise, and log storage
Structured logging moves service observability from “read by eye” mode into field-query mode. Below is what we check at acceptance so that the transition produces a measurable effect rather than simply changing the file format.
- Time to localize a failure: using the request ID, we collect the entire chain of calls between services in one query — the on-call engineer finds the cause in minutes instead of sifting through files on several hosts.
- Noise in logs: levels and the required set of fields are defined by the schema, debug messages do not go into the production stream, and the number of records per request stops growing with load.
- Log storage volume: level, service, and record lifetime are described by policy, so volume growth is calculated in advance rather than discovered when the disk fills up.
- On-call analysis: timestamp, service name, request ID, and response code are already inside the record — the incident context is read without access to the server.
- Eliminated failure class: log loss during container restart and “silent” services are closed by mandatory shipping to the shared collector Grafana Loki with receiver availability checks.
We confirm each point on a training incident: we take down one service and count how many steps and how much time the path from signal to cause takes. If the cause is found by fields, without reading raw text, logging works.
Structured logging vs text logging: comparison of modes
The event recording mode determines who will be the consumer of the log — a human or a collector.
Free text is read by eye, but only until the first wording change: after that, parsing on the collection side stops matching the message, and some events silently fall out of the sample.
Structured recording moves parsing into the application — the consumer receives ready-made fields and filters by them without guessing the format.
Predictability comes from an event schema that is unified for all services: timestamp, logging level, request ID, service name, event code, and nested context.
Levels then work as policy rather than as a note in a string: debug records go to a separate pipeline, warnings and errors go to the shared stream and do not drown in it.
| Option | When it fits | Limitations |
|---|---|---|
| Free text | One-off utilities where only the author reads the log | No fields: the sample is built by substring |
| Text with a “key=value” template | An existing service that cannot be changed entirely | Agreement on keys is maintained manually; nested structures lose their shape |
| Fixed field schema (JSON object per event) | Events are parsed by the collector: search, alerts, incident analysis | A field registry and a rule for changing them across teams are needed |
| Streaming to the collector | Many service instances, a shared picture is needed immediately | A channel break requires a disk buffer, otherwise events are lost |
In practice, a hybrid is chosen: a human-readable string to the console during local runs and a structured record to the collector for production pipelines.
The transition is carried out service by service — starting with those where incident analysis takes the most time, and with a field registry so that a new event does not require changes on the receiving side.
Implementation step by step: from event audit to handover of regulations
The sequence is arranged so that the event schema appears before the first log goes to storage: changing the format on running services is more expensive than agreeing on it at the start. Below is the order that every project goes through.
- Audit of log sources. We collect an inventory: which services write themselves, which send the stream to stdout, which have events spread across several streams and files. The output is a table with event frequency and daily volume for each source.
- Event schema design. We fix the required set of fields: time, level, service name, request ID, error code, message text. Separately — the level value dictionary and naming rules so that
erroranderrdo not live in the same index. - Application setup. The record goes out as one line per event, without line breaks inside the message — otherwise the collector will split it apart. The request context is passed through a shared wrapper and gets into every record automatically, including background jobs.
- Log collector setup. Field parsing is not needed: routing goes by values, disk buffering and rate limiting are added. Masking of tokens and personal data is performed at the input, before writing.
- Testing on a real failure. We reproduce a controlled error and go through the chain end to end: writing in the service, collector, storage, search by request ID. Separately, we check whether calls between services are assembled into one chain.
- Quality control and regulations. We set up regular checks: share of records without required fields, share of unrecognized levels, volume of noise from availability checks. We hand over a document with the field schema, rules for adding new events, and retention periods.
Artifacts: field schemas, collector configurations, regulations
After implementation, the team is left not with a set of settings on a server, but with a set of files and documents by which structured logging can be extended without us.
All artifacts are versioned in the same repository as the service code: a change goes through review and does not live only in someone’s head.
- Event dictionary — event names with required and optional fields, types, and allowed values. It also fixes what does not get into the log: tokens, full request bodies, contents of user files.
- Field contract and rules for changing it — how to add and rename fields without breaking parsing in storage and already-built reports.
- Log depersonalization rules — what is masked, what is hashed, where a value is replaced by an ID. They are set in shared configuration, not in each service’s code, so they are applied uniformly.
- Log collection configuration — stream parsing, buffering on the application side, behavior when storage is unavailable, file rotation, and thresholds at which a record is dropped instead of blocking the service.
- Log retention policy — periods by event type, separation of operational and archival storage, deletion procedure. It separately states which events live longer because of incident analysis.
- On-call instructions — how to collect all events of one request, which fields to filter first, ready-made queries for typical failures, and the escalation procedure.
- Schema change regulation — who approves a dictionary change, how compatibility is checked, and what to do with old records after fields change.
Panels for error frequency and latency are built on top of the same fields — no separate data source is needed for them.
Case: incident analysis reduced from hours to minutes
Before: six formats, search by guesswork
The customer’s platform — six applications: orders, billing, notifications, integrations, admin panel, and background jobs.
Each wrote logs as free text to its own file on its own node: time and level went inside the message string, and the request ID was not passed between services.
Incident analysis began with walking nodes and searching by fragments of phrases — it took 3-4 hours to reach the first version of the cause, and six out of ten alerts turned out to be false: they triggered on text matches.
What was changed and what was gained
A unified event schema was introduced: level, timestamp, service, request ID, operation code, duration, result.
The request ID is generated at the input and passed through the entire chain of internal calls — the path “request → service → storage → queue → response” is visible with one field search instead of trying out expressions across strings. Collection and search go to OpenSearch.
Retention rules were configured separately: availability checks and routine polls went to the debug level and are not written for every call, the hot pipeline keeps two weeks, then data moves to cold storage.
Log volume grew about threefold, but without these rules the growth would have been many times greater, and storage consumption stopped growing with traffic.
After three months: median time to find the cause — 8-10 minutes instead of 3-4 hours, false positives — one case out of ten instead of six, and the on-call shift closes twice as many incidents without developers.
At the junction of services, the effect is most noticeable: linking logs by a single ID immediately shows which call returned an error and what preceded it.
How does a log differ from a metric and a trace?
Three signals answer different questions, and confusion begins when one is used instead of another. A metric says “how much and how often”: it is an aggregate over a numeric series — failure rate, response time by tail percentile.
A log says “what happened”: it is an event with context — request ID, user, call parameters. A trace says “where time was lost”: the request path through services with the duration of each step.
The difference between a log and a metric is visible during analysis: the graph shows that failures increased after rollout, but not on which input. A metric deliberately loses context for the sake of cheap storage and fast aggregation.
Therefore, signals on which alerts are built are moved into metrics, while what is analyzed for a specific case goes into structured logging.
A request trace covers the third layer: when processing goes through several services and a queue, one log record will not show where almost all delays accumulated. A trace ID in each log record provides a transition from the graph to a specific call.
The main mistake is duplicating telemetry: one fact is written at once to a metric, log, and trace.
This bloats storage, accelerates divergence between systems, and confuses the choice of diagnostic tool: the on-call engineer opens three panels and does not understand which one is correct. The working rule is that each fact has one home, and the other signals reference it by ID.
The unified telemetry transmission standard — OpenTelemetry — removes vendor lock-in: the application’s observability backend can be changed without rewriting instrumentation code.
Limitations and frequently asked questions about implementation
Structured logging has a trade-off, and it is more honest to name it before implementation. A “key — value” record takes two to three times more space than a text string, and serialization on the hot path takes time away from processing the request itself.
We address this with levels — we do not write debug logs in production — by buffering the record and moving it to a separate thread so that logging does not slow down the main scenario.
The second limitation is a heterogeneous pipeline: some services write in the old text format, and a common schema cannot be assembled for them until the collector parses them with a rule.
The third is sensitive data: request fields regularly contain tokens, addresses, IDs, and personal data. We mask them at ingestion, before writing to storage, otherwise the shared index exposes them to everyone who has the right to read the logs.
Will we lose records during a load spike?
Delivery is the quietest point of failure: the collector buffer overflows, and some records are lost without a single error in the application.
Therefore, we count not only volume but also ingestion throughput, enable a disk buffer, and keep the share of dropped records as a separate metric with an alert.
Will old services have to be rewritten?
No. Bringing them to the common schema is done on the collector side with parsing rules; we do not touch the application. We rewrite only those services that will change in business logic within the next year anyway — there we introduce the new format along the way.
How do we separate personal data from other fields?
We perform parsing by schema and field list: what is considered an identifier and what is an operation body. Then masking at ingestion, separate storage streams with a shortened lifetime and access separation, so that analytics runs on depersonalized slices.
Let's discuss the task and come back with an estimate and plan
The conversation starts with source data — without it, an estimate turns into an assumption.
We only need the current application logging configuration, examples of real log records, the deployment diagram, and where events currently go: files on disk, container stdout, external collector.
What we will clarify on the call:
- which services and environments fall within the scope of work, and which remain outside the task;
- which events are critical for failure analysis, and which create noise and crowd out useful records;
- whether an end-to-end request ID is needed to link records from different services into one chain;
- how the company handles personal data in logs and who is responsible for the masking policy;
- retention periods, event stream volume, and requirements for archive search speed;
- who on the customer side accepts the result and in what form they expect documentation.
Next, we audit the current logging: we look at which fields are written now, where context is lost, how importance levels and rotation are arranged.
Based on the results, we come back with an implementation work plan — the order of migrating services, the required field schema, collector and storage settings, broken into stages with a clear result at each one.
Handover of knowledge to the team is a separate item: description of the record format, rules for adding new fields, ready-made query templates for logs, and analysis using live examples. Describe your logging task — we will come back with an estimate and plan.