Application Instrumentation with OpenTelemetry SDK

Application Instrumentation with OpenTelemetry SDK

Without a unified telemetry standard, you lose up to 40% of time finding the root cause: traces in one backend, metrics in another, logs in a third. OpenTelemetry combines everything into a single interface, saving up to 30% on operational costs. We have worked with this stack for over 5 years and implemented it in 50+ projects—typical savings from implementation amount to $5,000–$10,000 per month due to reduced downtime.

Instrumentation means adding observation points to code: where an operation starts, where it ends, what attributes it carries, and whether it completed successfully. The OpenTelemetry SDK standardizes this process: one API for traces, metrics, and logs, regardless of where the data eventually goes.

Two levels of instrumentation exist for each language. Automatic—an agent/patch instruments popular libraries without code changes. Manual—the developer explicitly creates spans for business logic: "order placement," "discount calculation," "external partner API call." Automatic does not replace manual—it only covers the infrastructure layer.

Why Choose OpenTelemetry SDK for Instrumentation?

Unlike proprietary solutions (Datadog, New Relic), OpenTelemetry does not lock you into a vendor: you can change backends (Prometheus, Jaeger, Grafana) without rewriting code. It is also an open standard supported by all major clouds and tools. Our engineers are certified in Kubernetes and OpenTelemetry—we guarantee correct configuration from the start. OpenTelemetry is 3–5 times more cost-effective than proprietary APM tools for large-scale deployments.

Manual Instrumentation: Right Patterns

On Python using a context manager (the correct approach):

from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer("order-service", "1.4.2")

def process_order(order_id: str, user_id: str) -> Order:
    with tracer.start_as_current_span(
        "order.process",
        kind=SpanKind.INTERNAL,
        attributes={
            "order.id": order_id,
            "user.id": user_id,
        }
    ) as span:
        try:
            order = fetch_order(order_id)
            span.set_attribute("order.total_amount", order.total)
            span.set_attribute("order.items_count", len(order.items))
            result = apply_discounts(order)
            span.add_event("discounts_applied", {"discount_percent": result.discount_percent})
            return result
        except OrderNotFoundException as e:
            span.record_exception(e)
            span.set_status(trace.StatusCode.ERROR, str(e))
            raise

record_exception() automatically adds a stack trace to span events. set_status(ERROR) signals backends and sampling policies. Without an explicit ERROR status, a tail-based sampler may not preserve the exception span.

Metrics: Counter, Histogram, Gauge

from opentelemetry import metrics

meter = metrics.get_meter("order-service", "1.4.2")

# Request counter
order_counter = meter.create_counter(
    "orders.created.total",
    unit="1",
    description="Total number of orders created"
)

# Processing time histogram
processing_duration = meter.create_histogram(
    "orders.processing.duration",
    unit="ms",
    description="Order processing duration"
)

def create_order(order_data):
    start = time.monotonic()
    order = _create_order_internal(order_data)
    duration_ms = (time.monotonic() - start) * 1000
    order_counter.add(1, {"order.type": order.type, "region": order.region})
    processing_duration.record(duration_ms, {"order.type": order.type})

Key point about metric attributes: each unique combination of attributes becomes a separate time series in Prometheus. Using user.id as a metric attribute is a cardinality disaster. order.type with 5-10 values is fine.

How to Avoid High-Cardinality Metrics?

Use only low-cardinality attributes: request type, region, service version. For unique identifiers (user_id, order_id) leave them only in spans, not in metrics. This preserves Prometheus performance and avoids storage overload.

How to Instrument Java Applications?

For Spring Boot, two paths are available: Auto-instrumentation via agent (recommended for starting):

-javaagent:opentelemetry-javaagent-2.x.jar
-Dotel.service.name=order-service
-Dotel.exporter.otlp.endpoint=http://otel-collector:4317
-Dotel.logs.exporter=otlp

Spring Boot Starter for manual instrumentation—add the dependency to pom.xml and auto-wire Tracer via @Autowired. Example:

<dependency>
  <groupId>io.opentelemetry.instrumentation</groupId>
  <artifactId>opentelemetry-spring-boot-starter</artifactId>
  <version>2.3.0-alpha</version>
</dependency>

Afterwards, you can inject OpenTelemetry and create spans for business logic as shown in the documentation.

Log Correlation with Traces

To navigate from Grafana Loki to a specific trace, log lines must contain trace_id and span_id. OTel provides helpers for popular log libraries.

Python + structlog:

import structlog
from opentelemetry import trace

def add_trace_context(logger, method, event_dict):
    span = trace.get_current_span()
    if span.is_recording():
        ctx = span.get_span_context()
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"] = format(ctx.span_id, "016x")
    return event_dict

structlog.configure(
    processors=[add_trace_context, structlog.processors.JSONRenderer()]
)

Comparison of Automatic vs Manual Instrumentation

Manual instrumentation provides 3 times more useful data for debugging but requires discipline. Optimal: automatic for quick start + manual for key operations.

Recommended Libraries for Different Languages

The choice between auto and manual depends on service maturity and required detail.

Commercial Deliverables

• Audit of current code and infrastructure – 1 day
• Configuration of OTel Collector with filtering and sampling – 2 days
• Integration with backends (Prometheus, Loki, Jaeger) – 1 day
• Creation of telemetry contracts (attribute and metric documentation) – 1 day
• Team training (workshop on instrumentation and correlation) – 1 day
• 2 weeks of free support after implementation – included
• Documentation of instrumented services – included

For a 10-microservice deployment, migration from a proprietary APM to OpenTelemetry saves approximately $4,000–$8,000 per month in licensing costs.

Instrumentation Completion Checklist

• All external HTTP calls have spans with http.request.method, server.address, http.response.status_code
• DB operations: db.system, db.name, db.statement (no credentials or PII)
• Errors recorded via record_exception() and set_status(ERROR)
• trace_id flows into logs
• Metrics have no high-cardinality attributes (user_id, order_id)
• Span names are stable strings, not containing dynamic IDs

Head-based sampling (e.g., ProbabilisticSampler) decides whether to keep a trace at the start. Tail-based sampling (IdGenerator) makes the decision after completion based on errors and duration. To save resources, use head-based with 10% probability and tail-based only for storing problematic traces.

Instrumentation timeline depends on the number of services and business logic complexity: from 2-3 days for 3-5 services to 1-2 weeks for a large polyglot stack. Contact us—we will assess your project in 1 day and propose an implementation plan. Order implementation and get 2 weeks of free support.

OpenTelemetry documentation