Centralised Logging & Distributed Tracing Setup Services

Logging and tracing configuration: ELK, Loki, Fluentd, Jaeger, Tempo

We’ve seen it happen: a service fails at night, an on-call engineer receives an alert, opens Kibana — and sees 40,000 lines without context. A request to /api/checkout returned 500, but where exactly did it fail: in the Rails application, in the downstream payment provider service, or in Redis during cache invalidation? Without structured logs and distributed tracing, answering this question takes hours. Our team has 8+ years of DevOps experience and 50+ observability implementations.

Why does ELK throttle with naive configuration?

Standard Elasticsearch + Logstash + Kibana setup in “everything on one node” mode works perfectly until log volume exceeds several gigabytes per day. After that comes the classic problem: Logstash consumes CPU on grok parsing, Elasticsearch enters GC pauses, indices grow uncontrollably, and retention policy is not configured.

Concrete situation: production cluster on three Elasticsearch 8.x nodes, Logstash parses nginx access logs via grok with pattern %{COMBINEDAPACHELOG}. At traffic spike of 50k rps, Logstash accumulates backlog in persistent queue and starts dropping events. Solution — transition to Filebeat → Elasticsearch ingest pipeline instead of Logstash for simple patterns, keeping Logstash only for complex transformations.

Ingest pipeline for nginx in Elasticsearch:

PUT _ingest/pipeline/nginx-access { "processors": [ { "grok": { "field": "message", "patterns": ["%{IPORHOST:client_ip} - %{DATA:user} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:method} %{DATA:path} HTTP/%{NUMBER:http_version}\" %{NUMBER:status:int} %{NUMBER:bytes:long}"] }}, { "date": { "field": "timestamp", "formats": ["dd/MMM/yyyy:HH:mm:ss Z"] }}, { "remove": { "field": "timestamp" }} ] } 

This removes Logstash from the critical path and reduces ingestion latency from 2-3 seconds to 200-300ms. Elasticsearch ingest node documentation

ELK vs Loki: not a silver bullet, but different tradeoffs

Loki from Grafana Labs is built on fundamentally different architecture: it does not index log contents, only labels. This makes it cheaper on storage (70% cost reduction compared to ELK for the same log volume) and faster on writes — but slower on arbitrary text search.

When Loki wins:

• Kubernetes environment where logs are already structured by namespace/pod/container
• Team already uses Grafana and Prometheus — single UI without Kibana
• Large log volume, but search mainly by known labels ({namespace="production", app="api"})

When ELK wins:

• Full-text search over log body needed: audit, compliance, PII search
• Complex aggregations: terms aggregation by user_id with HTTP status filter
• Existing investments in Kibana Dashboards and Watcher alerts

Typical mistake when migrating to Loki — put everything into labels that was fields in Elasticsearch. {status_code="404", method="GET", path="/api/v1/users"} — this is an antipattern. Loki creates separate stream for each unique label combination. At high cardinality (path with UUID), number of streams explodes and Loki starts working slower than Elasticsearch.

Correct Promtail configuration for K8s with filtering:

scrape_configs: - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod pipeline_stages: - cri: {} - match: selector: '{namespace="kube-system"}' action: drop - labels: app: namespace: pod: 

Only three labels. Everything else — in line_format for search via |= or regexp.

Distributed tracing: Jaeger vs Tempo

Logs tell you what failed. Tracing shows why and where exactly in the microservices chain.

Jaeger — battle-tested, emerged at Uber in 2015, CNCF graduated. Supports Cassandra and Elasticsearch as backend. Tempo — from Grafana Labs, stores traces in object storage (S3, GCS), no indexing, only by trace ID. Cheaper at scales above 1M+ spans/minute (up to 3x lower storage cost).

Principal difference: in Jaeger you can find trace by tags (http.status_code=500, db.statement LIKE '%timeout%'). In Tempo — only if you know trace ID or use TraceQL with metrics from Prometheus (via Grafana Tempo + metrics from spans via spanmetrics connector in OpenTelemetry Collector).

OpenTelemetry Collector as universal receiver:

receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 jaeger: protocols: grpc: thrift_http: processors: batch: timeout: 1s send_batch_size: 1024 tail_sampling: decision_wait: 10s policies: - name: errors-policy type: status_code status_code: { status_codes: [ERROR] } - name: slow-traces type: latency latency: { threshold_ms: 500 } exporters: otlp/tempo: endpoint: tempo:4317 tls: insecure: true jaeger: endpoint: jaeger-collector:14250 

Tail sampling here is critical — without it at 10k rps you save 10k traces per second, most of which are not interesting. Rule: save all errors + all slow requests (>500ms) + 1-5% random for baseline.

Fluentd and Fluent Bit: difference matters

Fluentd — Ruby, many plugins, flexible routing. Fluent Bit — C, minimal memory consumption (< 1MB RSS), for agents on nodes. Fluent Bit consumes 10x less memory than Fluentd. On K8s standard scheme: Fluent Bit as DaemonSet collects logs from nodes → Fluentd as aggregator performs transformation and routing to multiple sinks (Elasticsearch + S3 for archival).

Fluent Bit configuration for parsing JSON logs from containers:

[FILTER] Name parser Match kube.* Key_Name log Parser json Reserve_Data On Preserve_Key Off [FILTER] Name record_modifier Match kube.* Remove_key stream Remove_key _p 

Reserve_Data On — preserves original pod fields (namespace, pod_name) during parsing. Without this, Kubernetes metadata is lost.

How we evaluate and build

At the start — audit current state: log volume per day, retention requirements (compliance often dictates 1-3 years), current stack, whether tracing exists at all. Without these numbers, designing architecture is pointless — difference between 10GB/day and 1TB/day changes everything.

What’s included in the work

We deliver a full observability package под ключ:

• Architecture document with stack choice and justification
• Instrumentation scripts for your services (OpenTelemetry SDK integration)
• Helm charts and Terraform modules for deployment
• Runbook for on-call engineers (how to find traces, read span trees, escalate)
• Training session for your team (1-2 hours)
• Post-deployment support for 2 weeks

Work stages:

1. Audit — volume, structure of logs, collection points, current pain points (drops, latency, blind zones)
2. Design — stack choice (ELK/Loki/hybrid), index sharding scheme, retention policy, ILM in Elasticsearch or compactor in Loki
3. Instrumentation — adding OpenTelemetry SDK to services, structuring logs (JSON instead of plain text), correlation ID through all services
4. Deploy — Helm charts for Loki Stack or ECK (Elastic Cloud on Kubernetes), setup PVC for hot data + S3 for archive
5. Alerts and dashboards — Grafana for Loki/Tempo, Kibana for ELK, SLO-alerts on error rate and latency percentiles
6. Documentation — runbook for on-call: how to find trace by request ID, how to read span tree, typical error patterns

Timeline

Cost calculated individually after infrastructure audit. We guarantee SLA for your observability pipeline — contact us to evaluate your project.

What to check before implementation

• Label cardinality in Loki — check uniqueness of each label via label_values() before deploying to production
• Index lifecycle in Elasticsearch — without ILM, hot indices will consume SSD within a week at volume > 50GB/day
• Sampling rate in Jaeger — probabilistic 100% will kill Cassandra at any significant traffic
• Correlation ID — if missing in logs of all services, tracing is useless: impossible to link span with log line
• Resource limits for Elasticsearch — ES_JAVA_OPTS="-Xms4g -Xmx4g" and heap not more than 50% of node RAM, otherwise OS page cache starves

Common pitfalls and solutions

We’ve delivered this setup for clients logging 2TB/day. If you want to see numbers for your case, оценим проект — just write to us.