Filebeat, Promtail, Fluent Bit: How to Choose

Selecting the Right Log Forwarder

The log forwarding agent is the first and most critical link in the logging pipeline. This is where events most often get lost: Filebeat misses a file rotation — you lose 2000 lines. Promtail crashes on a node — it forgets its position. Fluent Bit eats 100% CPU parsing multiline Java stack traces. Based on our data, up to 5% of logs are lost due to incorrect agent configuration. Proper setup is not "set and forget" but regular auditing and adaptation to the infrastructure.

With 10+ years of DevOps experience and 50+ logging projects completed, we've been delivering reliable log pipelines since 2016. One case: configuring Fluent Bit for a 50-node cluster reduced resource consumption by 40% and cut log storage costs in half — saving $3,000 per month. Our typical audit cost ranges from $1,500 to $5,000 depending on cluster size.

Which Agent to Choose for Your Infrastructure?

All three tools do the same thing — read logs and send them to a storage system. The difference lies in resources, performance, and integrations. Summary table:

Filebeat is the choice for Elastic Stack. It has modules (nginx, mysql) with pre-built parsers. Promtail is optimal for Loki — it automatically enriches logs with pod metadata and uses 30% less memory than Filebeat. Fluent Bit is the lightest, 20 times lighter than Filebeat in memory, ideal for edge, IoT, and resource-constrained Kubernetes.

Configuring Filebeat with Edge Cases in Mind

Basic configuration is simple, but the devil is in the details — especially multiline and rotation handling:

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    multiline.type: pattern
    multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    multiline.negate: true
    multiline.match: after
    multiline.max_lines: 500
    multiline.timeout: 5s
    fields:
      app: myapp
      env: production
    fields_under_root: true
    scan_frequency: 10s
    close_inactive: 5m
    ignore_older: 24h
  - type: container
    paths:
      - /var/lib/docker/containers/*/*.log
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/lib/docker/containers/"
    processors:
      - drop_fields:
          fields: ["agent", "ecs", "host.os"]
          ignore_missing: true
output.logstash:
  hosts: ["logstash:5044"]
  loadbalance: true
  ttl: 30s
  compression_level: 3
queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: 5s
logging.level: warning
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 3

multiline is the most common cause of broken logs. A Java stack trace spans dozens of lines, and without a correct pattern each line flies off as a separate event. The pattern ^\d{4}-\d{2}-\d{2} means: a new event starts with a line containing a date, everything else is a continuation of the previous event. close_inactive: 5m closes the file descriptor if the file hasn't been updated for 5 minutes. Without this, Filebeat keeps hundreds of descriptors open on an active server.

Why Proper Multiline Configuration is Critical?

A recent case: a client complained about duplicate logs in Elasticsearch. Filebeat was capturing each line of a stack trace as a separate event, resulting in 10 times more logs than normal. The fix was to configure multiline.type: pattern with the correct expression. After that, the index size dropped from 200 GB to 30 GB per month — an 85% savings in disk space.

Fluent Bit in Kubernetes: Production-Ready Config

Fluent Bit is distributed as a Helm chart and deployed as a DaemonSet. Configuration via ConfigMap:

# values.yaml for helm chart fluent-bit
config:
  inputs: |
    [INPUT]
    Name tail
    Path /var/log/containers/*.log
    multiline.parser docker, cri
    Tag kube.*
    Mem_Buf_Limit 64MB
    Skip_Long_Lines On
    Refresh_Interval 10
  filters: |
    [FILTER]
    Name kubernetes
    Match kube.*
    Kube_URL https://kubernetes.default.svc:443
    Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log On
    Keep_Log Off
    K8S-Logging.Parser On
    K8S-Logging.Exclude On
    [FILTER]
    Name grep
    Match kube.*
    Exclude $kubernetes['labels']['skip-logging'] true
  outputs: |
    [OUTPUT]
    Name loki
    Match kube.*
    Host loki.logging.svc.cluster.local
    Port 3100
    Labels job=fluentbit, namespace=$kubernetes['namespace_name'], pod=$kubernetes['pod_name'], app=$kubernetes['labels']['app']
    Label_Keys $level,$trace_id
    Batch_Size 1048576
    Batch_Wait 1s
    Line_Format json
tolerations:
  - key: node-role.kubernetes.io/master
    operator: Exists
    effect: NoSchedule
  - key: node-role.kubernetes.io/control-plane
    operator: Exists
    effect: NoSchedule

Fluent Bit GitHub allows pods to disable logging via the annotation fluentbit.io/exclude: "true" — useful for debug tools that generate tons of garbage.

Promtail: Positions and Delivery Reliability

Promtail stores read positions in /tmp/positions.yaml. If this file is not persistent (e.g., in an ephemeral container), after a restart the agent will start reading from the beginning and duplicate all logs. For a DaemonSet, be sure to mount a hostPath:

---
# promtail DaemonSet snippet
volumes:
  - name: positions
    hostPath:
      path: /var/log/promtail-positions
      type: DirectoryOrCreate
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
volumeMounts:
  - name: positions
    mountPath: /run/promtail
  - name: varlog
    mountPath: /var/log
    readOnly: true
  - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true

Promtail loses positions in 100% of cases if hostPath is not mounted. Guarantee the safety of positions.yaml.

Common Problems and Their Solutions

What's Included in Log Forwarding Setup

We provide a full cycle of services:

• Audit of the current logging pipeline: volume, losses, delays.
• Agent selection and architecture design.
• Writing configs and Helm charts.
• Deployment via Ansible or GitOps.
• Integration with SIEM and alerting systems.
• Documentation and runbook for operators.
• Post-deployment monitoring and optimization.

Turnkey Log Forwarding Setup Process

Work stages:

1. Log source analysis: application, system, container logs. Determining frequency and volume (up to 10 TB/day).
2. Pipeline design: agent selection, buffering configuration, redundancy (active-active).
3. Implementation: configs for all servers, Helm charts for Kubernetes, SIEM integration.
4. Testing: stress test with a 10k events/s stream, verification of losses (SLO 99.9%).
5. Deployment and documentation: Ansible/Helm deployment, runbook for operators.

Estimated timeline: 2 to 5 business days depending on infrastructure complexity. Pricing is determined individually after an audit.

Ready to evaluate your project? Contact us — we'll set up your log forwarding with a data integrity guarantee. Order an audit of your logging pipeline today.