Centralized Kubernetes Logging with Datadog Logs Setup

You spend hours hunting for errors in logs scattered across hundreds of Kubernetes pods. Centralized logging with Datadog Logs solves this: real-time collection, indexing, and alerting. But costs can spiral if you don't control volume. An experienced engineer knows how to set up exclusion filters and log-based metrics to cut expenses by 40-60% without losing critical data. We configure Datadog Logs end-to-end in 3–7 days — from agent deployment to ready dashboards and alerts.

How to Deploy the Datadog Agent in Kubernetes

# datadog-values.yaml for Helm datadog:
apiKey: ${DD_API_KEY}
site: "datadoghq.eu" # or datadoghq.com
logs:
  enabled: true
  containerCollectAll: true
  containerCollectUsingFiles: true
processAgent:
  enabled: false
apm:
  portEnabled: false
clusterAgent:
  enabled: true
agents:
  tolerations:
    - operator: Exists
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi

helm upgrade --install datadog datadog/datadog \
  --namespace monitoring \
  --create-namespace \
  --values datadog-values.yaml \
  --set datadog.apiKey=$(kubectl get secret datadog-secret -o jsonpath='{.data.api-key}' | base64 -d)

Deployment takes about an hour. After that, the agent automatically collects logs from all containers — but without parsing, they arrive as JSON with deeply nested fields. To turn them into structured data, you need annotations and Log Pipelines. As Datadog documentation states: "Datadog Agent automatically collects logs from your containers" — Datadog Documentation.

How to Configure Annotations for Auto-Parsing

Datadog supports Pod annotations for automatic log collection and parsing setup:

---
# Deployment with Datadog annotations
metadata:
  annotations:
    ad.datadoghq.com/app.logs: |
      [{
        "source": "python",
        "service": "payments-api",
        "log_processing_rules": [
          {
            "type": "multi_line",
            "name": "python_traceback",
            "pattern": "Traceback|^\\s+File"
          }
        ]
      }]

For NGINX:

---
ad.datadoghq.com/nginx.logs: |
  [{ "source": "nginx", "service": "web-frontend", "tags": ["env:production", "team:frontend"] }]
---

Without log_processing_rules, multi-line errors (traces) end up in separate entries — making analysis impossible. Our experience shows that 80% of log aggregation issues stem from missing multi-line rules.

How to Reduce Ingest with Exclusion Filters

Datadog Logs is billed by ingested volume. Exclusion filters can be set in the agent's datadog.yaml or via the UI:

---
# /etc/datadog-agent/datadog.yaml
logs_config:
  processing_rules:
    - type: exclude_at_match
      name: exclude_health_checks
      pattern: "GET /health HTTP"
    - type: exclude_at_match
      name: exclude_debug_staging
      pattern: '"level":"debug"'

Through Datadog UI: Log Pipelines → Filters → add exclusion filter with condition env:staging AND status:debug. This reduces ingest by 40-60% in staging environments. In practice, a combination of filters by env, status, and source significantly lowers monthly Datadog costs for a project with 50 GB/day.

Why Log-Based Metrics Save Budget

Datadog allows generating metrics from logs without storing the logs themselves — useful for trends without increasing storage costs:

In Logs → Generate Metrics: create payment.errors from filter service:payments-api status:error. The metric is available in Datadog Metrics and dashboards alongside system metrics.

Log-based metrics let you get trends for errors, latency, request counts by status — without storing the raw logs. This means you pay only for metric volume (which is much smaller) rather than gigabytes of logs. This feature is built into Datadog: just define a filter and measure. We set up 5-10 key metrics per project — enough for basic monitoring. Such optimization leads to substantial observability budget savings.

Typical Problems and Their Solutions

If the agent doesn't see logs, first check RBAC: the agent's ClusterRole must have access to pods/logs and events. Then tolerations: if the agent isn't scheduled on nodes, logs aren't collected. A common mistake is forgetting to set tolerations: [operator: Exists] in values. In our experience, most cases are resolved within 15 minutes after correct configuration.

End-to-End Setup Process

1. Audit current infrastructure — environments, applications, log formats.
2. Design pipeline — sources, exclusion filters, retention, log-based metrics, dashboards.
3. Implementation — agent deployment, annotations, Pipeline configuration, testing.
4. Deliver documentation — runbook for adding new sources, troubleshooting.
5. Team training — how to use Datadog Logs, search, build alerts.

Timeline: 3 to 7 days depending on the number of services. Cost is calculated individually — tell us about your project and we'll prepare an estimate.

What's Included

• Helm chart for Datadog Agent with optimal settings for your infrastructure.
• Log Pipeline configuration: parsing, exclusion filters, multi-line rules.
• Set of log-based metrics for key services (errors, latency, RPS).
• Datadog dashboards: logging summary, error alerts.
• Runbook for your team: step-by-step instructions for adding new sources and common issues.
• Team training (2-hour workshop).

Comparison of Logging Approaches

Datadog wins on simplicity of operations but is more expensive at high volumes. The choice depends on budget and DevOps availability. Compared to Elasticsearch: Datadog deploys 2x faster (hours vs days) and requires no operational overhead.

Recommended Exclusion Filters

Our engineers have over 5 years of experience with Datadog Logs and 50+ observability projects. We guarantee that logs will be collected correctly, dashboards will reflect key metrics, and exclusion filters will save your budget. Get a consultation for your project. Order an audit of your current logging setup — we will prepare a proposal with a detailed plan.