Centralized Logging with ELK Stack: Production-Ready Setup

Note: when an application is scattered across ten pods in Kubernetes and something fails during off-hours, a sequential scan of kubectl logs is not an option. Centralized logging based on the ELK Stack (Elasticsearch + Logstash + Kibana) has long been the standard for debugging and monitoring. Typical storage costs for unstructured logs on cloud disks can account for up to 30% of the infrastructure budget — proper architecture cuts those costs in half.

For example, when payments-api crashes with a 500 error and logs are unavailable, you have to recreate the pod in debug mode and wait for the error to recur. With ELK, you see the full error stack for the last 5 minutes. We are a team of DevOps engineers with years of experience in high-load environments — we will deploy a production-ready log collection system that finds issues in seconds. Request an audit of your current logging setup — we will identify bottlenecks and propose optimizations.

Typical Mistakes When Setting Up ELK Yourself

The most common scenario: Elasticsearch is deployed without limiting heap size, and after a few weeks an OOM occurs. By default, ES takes half of the RAM, but if the JVM heap exceeds 30–32 GB, long GC pauses begin. The correct configuration: -Xms16g -Xmx16g for a dedicated node with 32 GB RAM — no more. The second problem is index strategy. A single index for all data over six months is a performance disaster. ILM (Index Lifecycle Management) with hot → warm → cold → delete phases and rollover/retention settings is essential. Third, Logstash as a single ingestion point in high-load environments. Logstash is a heavy JVM process. For aggregation from thousands of hosts, place Filebeat on each node → Kafka → Logstash → Elasticsearch. Kafka buffers load spikes and guarantees delivery.

How We Configure the ELK Stack for Kubernetes

The setup proceeds in four steps:

1. Deploy an Elasticsearch cluster (3+ nodes) with ILM policy.
2. Install Filebeat DaemonSet with autodiscover for container log collection. Configure RBAC for access to the Kubernetes API and use the sidecar pattern for application metrics.
3. Set up Logstash pipelines for enrichment and parsing (if additional transformation is needed).
4. Create Kibana dashboards and alerts via ElastAlert2.

In K8s environments, we use the standard scheme: Filebeat DaemonSet reads /var/log/containers/*.log and sends data directly to Elasticsearch or through Logstash. Below is the configuration we use in production.

---
# filebeat-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: logging
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    spec:
      serviceAccountName: filebeat
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:8.12.0
          args: ["-c", "/etc/filebeat.yml", "-e"]
          env:
            - name: ELASTICSEARCH_HOST
              value: "elasticsearch-master.logging.svc.cluster.local"
            - name: ELASTICSEARCH_PORT
              value: "9200"
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              readOnly: true
              subPath: filebeat.yml
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
      volumes:
        - name: varlog
          hostPath:
            path: /var/log

filebeat.yml with autodiscover for Kubernetes:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*-${data.kubernetes.container.id}.log
      templates:
        - condition:
            equals:
              kubernetes.labels.app: "payments-api"
          config:
            - type: container
              paths:
                - /var/log/containers/*-${data.kubernetes.container.id}.log
              processors:
                - decode_json_fields:
                    fields: ["message"]
                    target: ""
                    overwrite_keys: true
                - add_kubernetes_metadata:
                    host: ${NODE_NAME}
                    matchers:
                      - logs_path:
                          logs_path: "/var/log/containers/"
output.elasticsearch:
  hosts: ["${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"]
  index: "k8s-logs-%{[kubernetes.namespace]}-%{+yyyy.MM.dd}"
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/ssl/certs/ca.crt"]

Why ILM Policy Is Critical for Storage Costs

ILM manages the index lifecycle: hot data on fast NVMe, warm on HDD, and cold on HDD or S3. Example policy:

PUT _ilm/policy/k8s-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

In the hot phase, we store data for 1–7 days; in warm up to 30 days; in cold up to 90 days. After 90 days, we delete it. For longer archives, we use Searchable Snapshots in S3. ILM can reduce storage costs by up to 50%.

Kibana: Search and Alerts

In Kibana Discover, we search logs using KQL. Example: kubernetes.namespace: "production" AND level: "error" AND NOT kubernetes.pod.name: "batch-*". For alerts, we use ElastAlert2 — it is more flexible than built-in Kibana Alerting and does not require an X-Pack license:

# elastalert2 rule
name: "Payment service errors"
type: frequency
index: k8s-logs-production-*
num_events: 5
timeframe:
  minutes: 5
filter:
  - term:
      kubernetes.labels.app: "payments-api"
  - term:
      level: "error"
alert:
  - "slack"
slack_webhook_url: "https://hooks.slack.com/services/..."
alert_text: "Payment service: {num_matches} errors in last 5 minutes"

Why Consider OpenSearch Instead of Elasticsearch?

OpenSearch is a fork of ES 7.10, completely free and requires no licenses. It is 2–3 times cheaper than Elasticsearch with similar functionality. If your project does not use X-Pack features, migrating to OpenSearch reduces licensing costs by 30–50%. We have completed over 50 such projects and can help you choose the best option.

Results of Implementation

After a turnkey ELK Stack setup, you will be able to:

• Find errors in seconds by trace ID.
• Build dashboards for HTTP statuses, latency, error rates.
• Receive notifications in Slack/Telegram when thresholds are exceeded.
• Save on data storage through ILM and cold storage. Savings can reach $1.4k–1.9k per month for large projects.

What Is Included in the Work

Timeline

Pricing is determined individually based on architectural complexity. Contact us for a project assessment. Our certified engineers guarantee a reduction in log storage costs.