Setting Up Grafana Tempo for Distributed Tracing

Metrics show a latency anomaly, logs record an error, but connecting them into a single picture fails. We often face this situation: RPS increased, the 99th percentile dropped to 800 ms, and the cause is a single request that touches 10 microservices. Without tracing, finding that point is a needle in a haystack. Grafana Tempo is a distributed tracing backend that integrates seamlessly with Prometheus and Loki in a unified Grafana interface. We offer a turnkey solution: from instrumentation audit to correlation setup and team training. Request an audit of your system—get an implementation plan in one day. Contact us to discuss the details.

How to Configure Grafana Tempo for Distributed Tracing?

Tempo does not build indexes on trace content. It stores data directly in object storage (S3, GCS), indexing only the Trace ID. This yields radical storage cost savings—up to 70% compared to Jaeger, which requires Elasticsearch or Cassandra. In practice, Tempo costs significantly less: storing 1 TB of traces in S3 costs several times less. For a project with 20 microservices and a load of 5000 spans per second, the storage savings exceed $2000 per month. The downside: attribute search without a dedicated query language is impossible. However, the workflow "found an anomaly in metrics → navigated to a specific trace" covers 90% of diagnostic scenarios. For ad-hoc search, use TraceQL, available since Tempo 2.0.

Tempo vs Jaeger: Storage Cost

Savings come from omitting the expensive index. We use tail sampling in the OpenTelemetry Collector to filter noise: all errors and slow requests are saved, others with a 10% probability. This halves stored data volume without losing diagnostic value.

Comparison of Sampling Methods

For production environments, we recommend tail sampling—it provides maximum completeness of critical traces.

Instrumenting Applications with OpenTelemetry

Tempo accepts traces via OTLP (recommended), Jaeger, or Zipkin. Example of Python instrumentation:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
exporter = OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("process_order") as span:
    span.set_attribute("order.id", order_id)
    result = process(order_id)

Traces are sent to the OpenTelemetry Collector, which performs sampling and forwards to Tempo. We recommend using OpenTelemetry as the standard for instrumentation.

OpenTelemetry Collector: Central Hub

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors-policy
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-policy
        type: latency
        latency: {threshold_ms: 1000}
      - name: probabilistic
        type: probabilistic
        probabilistic: {sampling_percentage: 10}
exporters:
  otlp/tempo:
    endpoint: http://tempo:4317
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp, jaeger]
      processors: [batch, tail_sampling]
      exporters: [otlp/tempo]

Deploying Tempo

---
services:
  tempo:
    image: grafana/tempo:2.4.0
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/tmp/tempo
    ports:
      - "3200:3200"
      - "4317:4317"

---
server:
  http_listen_port: 3200
distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:
storage:
  trace:
    backend: local # for development; in production: s3
    local:
      path: /tmp/tempo/blocks
compactor:
  compaction:
    block_retention: 336h
metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
  storage:
    path: /tmp/tempo/generator/wal
  remote_write:
    - url: http://prometheus:9090/api/v1/write
      send_exemplars: true