Centralized Security Event Collection Setup for SIEM

Centralized Security Event Collection Setup for SIEM

Ever faced a situation where after an incident you have to gather logs from 15 servers bit by bit, with timestamps differing by minutes? That wastes hours on investigation. One of our clients — a fintech company with 200 microservices — before implementing centralized collection spent up to 3 days finding the root cause of a leak. After setting up the SIEM pipeline, the time dropped to 2 hours. We solve this problem — we configure end-to-end centralized security event collection, guaranteeing delivery and integrity of data. We have 10+ years of experience in infrastructure security, and we know how to avoid common mistakes.

Centralized event collection (SIEM pipeline) is not just dumping logs into one place. It's an architecture that allows correlating events, building alerts, and quickly finding the root cause of an incident. Below we break down the typical architecture, agent selection, broker role, and common configurations.

Typical Architecture: What Goes Where and Why

Sources (servers, containers, cloud services) → agents (Fluent Bit, Filebeat, rsyslog) → message broker (Kafka — optional) → processor/parser (Logstash, Vector, Fluentd) → storage (Elasticsearch/OpenSearch, Loki, Splunk). A message broker is needed for high volumes (100K+ events/sec) or if storage might be unavailable — Kafka buffers for up to 7 days. For small volumes (up to 10K events/sec), you can go directly agent → storage.

Comparison of Popular Agents

Fluent Bit is 10x lighter in memory consumption than Filebeat, which is critical for high-load nodes.

Fluent Bit: Lightweight Agent for Hosts and Containers

Fluent Bit — 450KB binary, consumes ~1MB RAM per agent. For high-load servers and K8s DaemonSet, this is critical.

# /etc/fluent-bit/fluent-bit.conf
[SERVICE]
    Flush 5
    Daemon Off
    Log_Level info
    Parsers_File parsers.conf

[INPUT]
    Name tail
    Tag audit.*
    Path /var/log/audit/audit.log
    Parser auditd
    DB /var/lib/fluent-bit/audit.db
    Mem_Buf_Limit 50MB
    Skip_Long_Lines Off
    Refresh_Interval 10

[INPUT]
    Name tail
    Tag auth.*
    Path /var/log/auth.log
    Parser syslog-rfc3164
    DB /var/lib/fluent-bit/auth.db
    Mem_Buf_Limit 10MB

[INPUT]
    Name systemd
    Tag systemd.*
    Systemd_Filter _SYSTEMD_UNIT=sshd.service
    Systemd_Filter _SYSTEMD_UNIT=sudo.service

[FILTER]
    Name modify
    Match *
    Add host ${HOSTNAME}
    Add environment production
    Add datacenter eu-west-1

[OUTPUT]
    Name http
    Match *
    Host loki.internal.example.com
    Port 3100
    URI /loki/api/v1/push
    Format json
    Retry_Limit 5

rsyslog for Centralization via syslog Protocol

For infrastructure where rsyslog already exists — forwarding to a central server. We guarantee that rsyslog does not lose events on restart thanks to queue.saveonshutdown. Below are configurations for client and server in one block.

# On each source server /etc/rsyslog.d/99-central-syslog.conf
action(type="omfwd" Target="syslog.internal.example.com" Port="6514" Protocol="tcp" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="syslog.internal.example.com" queue.type="LinkedList" queue.size="100000" queue.filename="fwdq" queue.maxdiskspace="1g" queue.saveonshutdown="on" action.resumeRetryCount="-1")

# On central server /etc/rsyslog.d/00-receive.conf
module(load="imtcp" StreamDriver.AuthMode="x509/certvalid" StreamDriver.Name="gtls" PermittedPeer=["*.internal.example.com"])
input(type="imtcp" port="6514")
template(name="DynFile" type="string" string="/var/log/remote/%HOSTNAME%/%PROGRAMNAME%.log")
*.* action(type="omfile" DynaFile="DynFile")

Learn more about the syslog protocol. As stated in the rsyslog documentation, this approach ensures reliable delivery even during reboots.

Which Agent to Choose: Fluent Bit, Filebeat, or rsyslog?

In modern projects, we most often use Fluent Bit: it's lighter, supports multiplexing, and has a plugin ecosystem. For pure syslog environments (network equipment, Unix) — rsyslog. Filebeat fits well into the Elastic stack. Combining them is acceptable: e.g., Fluent Bit on K8s, rsyslog on network devices.

Kafka as a Buffer for High Volumes

At 100K+ events/sec or multiple storage destinations — Kafka in the middle:

---
# Filebeat → Kafka topic
output.kafka:
  hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
  topic: 'security-events-%{[agent.hostname]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000

Logstash reads from Kafka and writes to Elasticsearch plus S3 for long-term retention:

# logstash.conf
input {
  kafka {
    bootstrap_servers => "kafka1:9092,kafka2:9092"
    topics => ["security-events-.*"]
    group_id => "logstash-security"
    codec => json
  }
}
filter {
  if [event][module] == "auditd" {
    mutate {
      add_tag => ["audit", "compliance"]
    }
  }
  geoip {
    source => "[source][ip]"
  }
}
output {
  elasticsearch {
    hosts => ["https://es1:9200", "https://es2:9200"]
    index => "security-events-%{+YYYY.MM.dd}"
    template_name => "security-events"
  }
  s3 {
    bucket => "security-events-archive"
    region => "eu-west-1"
    prefix => "raw/%{+YYYY/MM/dd}/"
    codec => json_lines
    buffer_interval => 900
  }
}

Why Time Synchronization Matters for Correlation?

Without accurate time on all sources, correlation is useless. A difference of more than 1 second can shift the sequence of attack events. NTP is mandatory — ideally one internal NTP server for the entire infrastructure, synchronized with multiple external ones. We use chrony, which is faster and more accurate than ntpd. On all servers we configure server ntp.internal.example.com iburst prefer and monitor offset via chronyc tracking — this is part of our guarantee.

How We Work on a Project

1. Analysis: audit infrastructure, log volumes, retention requirements.
2. Design: choose stack (Elastic/Loki, Kafka/no), flow diagram.
3. Pilot: configure on 2-3 sources, verify delivery.
4. Production: deploy agents on all nodes, configure rotation and retention.
5. Documentation and team training.
6. 1-month warranty support after go-live.

What's Included in the Project

• Audit of current logging infrastructure.
• Design of collection scheme considering retention and compliance.
• Installation and configuration of agents (Fluent Bit, Filebeat, rsyslog) on all sources.
• Kafka broker setup (if needed).
• Storage connection (Elasticsearch/OpenSearch/Loki).
• Creation of dashboards and alerts for key events.
• Full documentation: diagrams, configs, runbook.
• Training for up to 5 engineers.
• Guarantee of uninterrupted log delivery.

Cost is calculated individually after an infrastructure audit. We assess the project in 1 day. Contact us to get a consultation. Order a centralized log collection project — our engineers will perform an audit and implement the solution, saving you up to 30% on incident response time.