One day in production, PostgreSQL crashed: the disk filled up—journald in tmpfs was unlimited. On reboot, logs vanished. Root cause took a day to find. Downtime exceeded 8 hours, and the team spent two days on re-analysis. Sound familiar? Without proper configuration, the systemd journal becomes a black box. Our engineers, with 10+ years of experience, configure journald so logs are never lost, disk never overflows, and data flows into a centralized system. This reduces MTTR by 70% and saves up to 40% on storage budget through rotation and compression. Across 200+ projects, we've refined configurations that work reliably. We use Ansible for mass rollout, cutting fleet setup (50+ machines) to one day. On one project, we encountered a noise service that hit RateLimitBurst too low—critical logs were dropped. After adjusting configuration and setting up forwarding to Loki, MTTR dropped from 4 hours to 30 minutes. Contact us for a consultation—we'll tailor the optimal configuration for your load.
How to Configure Persistent Storage and Size Limits?
By default, journald writes to /run/log/journal/—tmpfs, data lost on reboot. To enable permanent storage:
mkdir -p /var/log/journal systemd-tmpfiles --create --prefix /var/log/journal systemctl restart systemd-journald Configuration in /etc/systemd/journald.conf:
[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=2G
SystemKeepFree=500M
SystemMaxFileSize=200M
SystemMaxFiles=10
MaxRetentionSec=1month
MaxFileSec=1week
RateLimitInterval=30s
RateLimitBurst=10000
ForwardToSyslog=no
ForwardToKMsg=no
ForwardToConsole=noSystemMaxUse=2G—disk limit. Without it, journald can consume the entire disk in days under heavy logging. RateLimitBurst=10000 protects against log storms: if a service emits more than 10,000 messages in 30 seconds, extras are dropped. For noisy apps (e.g., nginx debug), raise this limit via a drop-in config.
Configuration Comparison: Basic vs Production
| Parameter | Basic (tmpfs) | Production (persistent) |
|---|---|---|
| Storage | volatile | persistent |
| SystemMaxUse | 10% of disk | 2G (explicit) |
| RateLimitBurst | 1000 | 10000 |
| MaxRetentionSec | 3 days | 1 month |
| Compress | yes | yes |
Why RateLimit Matters
Without RateLimit, a log burst from a faulty service can clog journald, causing loss of critical messages from other units. Setting RateLimitInterval and RateLimitBurst is essential for stability. We recommend 10,000 messages per 30 seconds for most services, and up to 50,000 for noisy ones.
How to Filter Logs by Priority and Unit?
Viewing locally:
# Logs for a specific unit, follow mode
journalctl -u nginx.service -f
# Logs since last boot
journalctl -b
# Last hour
journalctl --since "1 hour ago"
# Errors and critical only
journalctl -p err -b
# JSON output for parsing
journalctl -u myapp -o json-pretty --since today
# Logs for a specific PID
journalctl _PID=1234For centralized forwarding, filtering is configured on the agent side: in promtail via relabel_configs, in Filebeat via include_matches. This sends only necessary logs, saving bandwidth and storage.
Why Limit Journal Size?
On a server with heavy log writing, without SystemMaxUse, journald can occupy up to 100% of free space in 2-3 days. This causes daemon failures, SSH lockouts, and data loss. A 2G limit preserves disk and keeps services alive. Proper rotation also cuts MTTR—you find errors faster. Our clients save tens of about $9–13 in savings on infrastructure through compression and rotation.
How to Forward Logs to Central Storage?
For Loki, use promtail with journal input:
---
scrape_configs:
- job_name: journal
journal:
max_age: 12h
labels:
job: systemd-journal
host: ${HOSTNAME}
relabel_configs:
- source_labels: [__journal__systemd_unit]
target_label: unit
- source_labels: [__journal_priority_keyword]
target_label: level
For Elasticsearch via Filebeat:
---
filebeat.inputs:
- type: journald
id: everything
include_matches:
- _SYSTEMD_UNIT=nginx.service
- _SYSTEMD_UNIT=postgresql.service
- PRIORITY=0
- PRIORITY=1
- PRIORITY=2
- PRIORITY=3
---Comparison: Promtail vs Filebeat for journald
| Feature | Promtail (Loki) | Filebeat (Elastic) |
|---|---|---|
| Target system | Grafana Loki | Elasticsearch |
| Filtering | relabel_configs | include_matches |
| Performance | 20% faster under same load | High (Go) |
| Parsing | pipeline stages | processors |
Example full promtail config for production
---
scrape_configs:
- job_name: journal
journal:
max_age: 24h
labels:
job: systemd-journal
host: "${HOSTNAME}"
relabel_configs:
- source_labels: [__journal__systemd_unit]
target_label: unit
- source_labels: [__journal_priority_keyword]
target_label: level
- source_labels: [__journal__hostname]
target_label: hostname
- replacement: "auto"
target_label: agent
pipeline_stages:
- drop:
selector: '{level="debug"}'
Turnkey journald Configuration Process
- Audit current journald configuration on all servers.
- Enable persistent storage and rotation with limits calibrated to your load.
- Integrate with centralized logging (Loki, Elasticsearch, Syslog).
- Create a runbook for on-call engineers and documentation.
- Load testing: run a stress test to verify no log loss.
Configuring a single server takes up to 4 hours. For a fleet of 50+ machines, rollout via Ansible takes 1-2 days. Pricing is per project—contact us for an assessment. We guarantee a reliable journal post-configuration.
What's Included in journald Setup
- Documentation: detailed configuration scheme, limit descriptions, on-call manual.
- Observability integration: connection to Loki or Elasticsearch, Grafana dashboards.
- Runbook and training: incident response scenarios, team training on journalctl.
- Post-deployment support: 2 weeks of monitoring and configuration adjustments.
Common journald Configuration Mistakes
- No persistent storage: logs lost on reboot — root cause impossible to find.
- Missing SystemMaxUse: under peak load, journald fills disk, services crash.
- Too low RateLimitBurst: a noisy service drops critical messages.
- No forwarding: logs stay local, central monitoring blind.
Our engineers eliminate these risks—after configuration, the journal behaves predictably.
Bottom line: correct systemd-journald configuration is the foundation of any observability stack. If you want to stop losing logs and reduce downtime, order journald setup from our team. Contact us for a consultation to tailor the config to your project. Our specialists have 10+ years of experience and have completed 200+ log configuration projects.