Centralized Log Collection with syslog-ng: TLS, Elasticsearch

After a data leak incident, we couldn't reconstruct the full timeline — logs from five servers were scattered across auth.log, messages, and journald. Rebuilding the chronology took two days instead of a couple of hours, and analysis showed that up to 30% of critical messages had been lost due to the lack of a centralized store. That's when we decided to deploy syslog-ng — a secure, single location for all logs. Today, MTTR for log-related incidents has dropped from 4 hours to 30 minutes. Time saved searching logs after deployment is up to 40%.

We've been using syslog-ng in production for over 7 years. The tool understands syslog RFC3164 and RFC5424, journald, files, and HTTP, and its DSL for message transformation is more flexible than rsyslog's. Official documentation confirms the project's maturity and active community support. During this time we've delivered more than 50 centralized log collection projects — from startups to enterprise environments with 500+ servers.

Basic Architecture: Client → Server

The client (each server in the infrastructure) sends logs to the central syslog-ng server over TLS. The server parses by source, writes to files, or forwards to Elasticsearch.

Client configuration:

# /etc/syslog-ng/conf.d/client.conf
source s_local {
    system();
    internal();
};

destination d_central {
    syslog(
        "syslog.example.com"
        port(6514)
        transport("tls")
        tls(
            ca-file("/etc/syslog-ng/ca.pem")
            cert-file("/etc/syslog-ng/client.pem")
            key-file("/etc/syslog-ng/client.key")
        )
        flags(no-multi-line)
    );
};

log {
    source(s_local);
    destination(d_central);
};

Server configuration:

# /etc/syslog-ng/syslog-ng.conf
@version: 4.5
@include "scl.conf"

options {
  time-reap(30);
  mark-freq(0);
  keep-hostname(yes);
  create-dirs(yes);
  dir-perm(0755);
  perm(0640);
};

source s_network_tls {
  syslog(
    ip(0.0.0.0)
    port(6514)
    transport("tls")
    tls(
      key-file("/etc/syslog-ng/server.key")
      cert-file("/etc/syslog-ng/server.pem")
      ca-file("/etc/syslog-ng/ca.pem")
      peer-verify(required-trusted)
    )
  );
};

template t_json {
  template(
    "$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE @timestamp=${ISODATE})\n"
  );
};

destination d_elasticsearch {
  elasticsearch-http(
    url("http://elasticsearch:9200/_bulk")
    index("syslog-${YEAR}.${MONTH}.${DAY}")
    type("")
    template(t_json)
    flush-lines(100)
    flush-timeout(10000)
  );
};

destination d_files {
  file(
    "/var/log/syslog-ng/${HOST}/${PROGRAM}.log"
    template(t_json)
    create-dirs(yes)
  );
};

filter f_error {
  level(err..emerg);
};

log {
  source(s_network_tls);
  destination(d_files);
  destination(d_elasticsearch);
  flags(flow-control);
};

flags(flow-control) — when the buffer overflows, syslog-ng blocks rather than drops messages. Essential for audit-critical streams.

Transformation and Filtering

syslog-ng can parse structured logs via json-parser and kv-parser:

parser p_json {
    json-parser(prefix(".json."));
};
filter f_app {
    program("myapp");
};
rewrite r_add_env {
    set("production" value(".json.environment"));
};
log {
    source(s_network_tls);
    filter(f_app);
    parser(p_json);
    rewrite(r_add_env);
    destination(d_elasticsearch);
};

Why Choose syslog-ng Over rsyslog?

The choice between syslog-ng and rsyslog depends on your needs. syslog-ng wins when you need complex log transformation and security; rsyslog is better for simple routing with maximum performance. Below is a comparison of key characteristics.

For a real-world case: on a 200-server project with PCI DSS requirements, we chose syslog-ng — needed flexible filtering and guaranteed delivery. Rsyslog couldn't handle the transformation of non-standard logs.

How syslog-ng Solves the Problem of Log Loss

Problem: channel overload or server receive failure leads to message drops. syslog-ng addresses this with two mechanisms:

• Flow-control: blocks the client when the buffer is full.
• Disk buffer: disk-buffer algorithm saves logs to disk if memory runs out. Configure with buf-size and dir.

For critical systems, we always enable both. Delivery guarantee — 99.9% with proper configuration.

Implementation Process: What's Included

• Analysis of current log collection scheme (sources, protocols, rotation policies).
• Architecture design: TLS scheme selection, server clustering configuration.
• Client setup (Ansible playbook for mass deployment).
• Server configuration: parsers, filters, destinations.
• Failure testing: pull the plug — verify log loss during network break.
• Runbook documentation and monitoring scripts (Prometheus metrics via exporter).
• Team training: how to add new sources and modify configs.

Timeline: 3 to 7 days depending on server count (from 5 to 100). Cost is calculated individually — contact us, we'll estimate your project within one business day.

Common Mistakes When Configuring syslog-ng

• Using tls() without peer-verify(required-trusted) — any client with a certificate signed by your CA can send logs. Security risk.
• Ignoring persist-name in logs — index lost on server restart.
• Overly aggressive flush-lines in Elasticsearch destination — causes frequent HTTP requests and reduced throughput. Optimal value: 10-50.

Our experience: over years of working with syslog-ng, we've set up centralized collection for projects from startups to enterprise with 500+ servers. We guarantee post-deployment support — SLA is fixed in the contract.

Order turnkey syslog-ng implementation: we'll configure the server, deploy clients on all servers, and integrate with Elasticsearch. Get full log audit with security compliance. Contact us for a consultation — we'll assess your project within one business day.