Correlating Logs, Metrics, and Traces with OpenTelemetry

Correlating Logs, Metrics, and Traces

You're investigating an incident: metrics show a sharp latency spike at 03:47, but Grafana has no annotation, Loki spits out thousands of log lines for that minute, and Jaeger has traces but you can't tell which span was the first to fail. Three systems, three interfaces, zero linkage — and your team spends an hour manually cross-referencing timestamps. We've been through this dozens of times. Our approach: set up end-to-end correlation in 5–10 working days, cut MTTR by 60%, and free your team from manual toil.

Correlation of logs, metrics, and traces gives you the ability to click from a metric anomaly straight to a trace, and from a trace to the exact log line with the error context. Without it, diagnosing distributed systems feels like detective work with a magnifying glass. Contact us for a consultation on setting up correlation for your project.

How OpenTelemetry Provides a Unified Layer

The modern stack for correlation is the OpenTelemetry Collector, which receives data from all three signals and forwards them to respective backends. A typical configuration:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: 'app'
          static_configs:
            - targets: ['app:8080']
processors:
  batch:
    timeout: 10s
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert
exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    labels:
      resource:
        service.name: "service_name"
        deployment.environment: "env"
  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: otel
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [loki]

The key point: all three pipelines run through a single Collector that maintains a unified resource. According to the OpenTelemetry specification, trace_id must be a 128-bit hex string.

Exemplars: Linking Metrics to Traces

Prometheus since version 2.29 supports exemplars — data points on histograms that carry a trace_id. Two conditions are needed:

# prometheus.yml global: scrape_interval: 15s storage: exemplars: max_exemplars: 100000 

And in the application itself (example in Go with prometheus/client_golang):

histogram.With(prometheus.Labels{"handler": "/api/users"}).ObserveWithExemplar(
	duration,
	prometheus.Labels{"traceID": span.SpanContext().TraceID().String()},
)

In Grafana, with exemplars enabled, small diamonds appear on the graph — clicking one takes you directly to the corresponding trace in Jaeger/Tempo. Our experience shows exemplars reduce root-cause search time from 40 minutes to 5.

Propagating trace_id into Logs

Each request should extract the trace_id from the active span and add it to the log entry. In Node.js with winston:

const { trace } = require('@opentelemetry/api');
const addTraceInfo = winston.format((info) => {
  const span = trace.getActiveSpan();
  if (span) {
    const ctx = span.spanContext();
    info.trace_id = ctx.traceId;
    info.span_id = ctx.spanId;
  }
  return info;
});

Once Loki receives logs with a trace_id field, Grafana can build a link from a log line straight to Tempo/Jaeger via derivedFields in the datasource.

Why correlation_id Is Not Enough

Many teams stop at X-Request-ID headers and correlation_id in JSON logs. That's necessary but insufficient. Problems arise when a Go service logs unstructured stdout, Logstash parses with grok patterns that miss trace_id, and Prometheus collects metrics without a trace_id label — it can't, it's a fundamentally different data type. Jaeger traces store span_id and trace_id, but the service logs don't contain them because the developer didn't propagate context through context.Context.

True correlation rests on three pillars: a unified time format (UTC, milliseconds), trace context propagation into logs, and exemplars in metrics to link to traces. Without exemplars, correlation is incomplete.

How Grafana Unifies All Three Sources

Grafana 10+ can "glue" all three sources together via Explore. One query in Loki shows a "View in Tempo" button; from Tempo, a "Logs for this span" button. This works only if correlations between datasources are configured. Correlations set through the Grafana UI tend to be fragile — it's better to define them in provisioning/datasources/.

Compare three correlation methods:

Typical Problems During Implementation

• Different trace_id formats. Jaeger uses 128-bit hex, Zipkin uses 64-bit. If a legacy service uses Zipkin, trace_id in logs will have different formats and correlation will fail. Solution: OpenTelemetry Collector with zipkinreceiver and conversion.
• Asynchronous tasks losing context. A Celery worker processing a task from a Redis queue doesn't know the HTTP request trace_id. You must explicitly pass the trace context in the payload and restore it via propagate.extract().
• Unstructured logs. If an application writes ERROR: connection refused to stdout, you can't extract trace_id without grok parsing. Sometimes it's simpler to add a JSON formatter.

What's Included in the Work

• Audit of the current stack: inventory of services, languages, log formats, and existing tracing instrumentation.
• Implementation of OpenTelemetry SDK or auto-instrumentation via otel-operator for Kubernetes.
• Deployment of OpenTelemetry Collector with three pipelines and configured exporters.
• Setup of exemplars in Prometheus and correlations in Grafana via provisioning.
• Propagation of trace_id into logs across all services.
• Testing of end-to-end correlation: generate traffic, verify trace_id from HTTP to database.
• Documentation: architecture description, developer instructions.
• Team training (2-hour webinar).
• 2 weeks of post-deployment support.

How We Do It: Step by Step

1. Audit current stack (1–2 days).
2. Implement OpenTelemetry SDK in all services (3–10 days).
3. Configure Collector and Grafana (1–2 days).
4. Test and document (1–2 days).

Our engineers have 5+ years of observability experience, are certified in OpenTelemetry and Kubernetes. Over 5 years we've completed 50+ observability projects. The project pays for itself in 2–3 months by reducing incident diagnosis time. Contact us for a project assessment — order a turnkey implementation.