Tuning Linux Kernel (sysctl) for High-Load Servers

Tuning Linux Kernel (sysctl) for High-Load Servers

We integrate and tune Linux kernel parameters (sysctl) for high-load servers. Our team has 6+ years of DevOps experience and has completed 80+ optimization projects. Default kernel settings are a compromise between compatibility and performance. On a server with 10k+ concurrent connections or high UDP traffic, they become a bottleneck: the application gets EMFILE, sockets hang in TIME_WAIT, packets are lost. We've encountered this many times on projects: when metrics show a drop but CPU and RAM are free — the cause is almost always sysctl. With more than 5 years of kernel tuning, we've collected profiles for webserver, DB, load balancer. This article provides ready-made configs and practice, no theory. Our experience: on one project with 50k RPS, simply increasing somaxconn and enabling BBR reduced latency by 30% — a clear improvement over default settings.

Why Default Kernel Settings Are a Bottleneck

The Linux kernel out of the box is tuned for versatility: moderate backlog, conservative TCP, small buffers. On highload this turns into:

• Connection queue overflow. Every listen socket has a backlog. Under burst traffic, the queue overflows — SYN packets are dropped. Default net.core.somaxconn = 4096 chokes at 5k RPS. We set it to 65535.
• TIME_WAIT and ephemeral port exhaustion. Each closed TCP connection stays in TIME_WAIT for 2*MSL (60 s). At 10k requests/sec — 600k sockets in TIME_WAIT. The port pool 32768–60999 (28k) is exhausted in a second. Solution: expand the range, enable tcp_tw_reuse, reduce tcp_fin_timeout.
• Small receive/transmit buffers. Default rmem_default=87380 and wmem_default=16384 throttle high-latency connections (CDN, inter-region). For streaming servers this is critical.

# /etc/sysctl.d/99-network.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_tw_buckets = 1440000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

How BBR and Buffers Squeeze Maximum from the Network

BBR (Bottleneck Bandwidth and RTT) from Google is a congestion control algorithm that works based on measurements, not loss. On WAN links with high latency (100+ ms), BBR gives up to 40% throughput improvement over CUBIC, and in some scenarios BBR outperforms CUBIC by 1.5–2 times. Enabled with two lines:

net.core.default_qdisc = fq net.ipv4.tcp_congestion_control = bbr 

Check support via sysctl net.ipv4.tcp_available_congestion_control — should show bbr cubic reno. Available from kernel 4.9. For more details, see BBR on Wikipedia.

Buffers: for connections with high bandwidth-delay product (e.g., 100 Mbps × 200 ms = 2.5 MB), default buffers are insufficient. Increasing tcp_rmem to 16 MB allows full pipe utilization without losing throughput.

How to Avoid Hitting the File Descriptor Limit

The second common cause of degradation is fd exhaustion. Nginx with 65536 worker_connections consumes 131072 fd (each connection—socket and file descriptor). Default fs.file-max = 524288 is quickly exhausted. Tune:

# /etc/sysctl.d/99-fs.conf fs.file-max = 2097152 fs.nr_open = 1048576 

Important: fs.nr_open is the maximum limit per process. For nginx also check LimitNOFILE in the systemd unit (usually LimitNOFILE=65536). Check current limit: cat /proc/$(pidof nginx)/limits | grep "Open files". Usage: cat /proc/sys/fs/file-nr (first number—used, third—max). Check current process limits: cat /proc/<PID>/limits | grep 'Max open files'. Also check system-wide usage: cat /proc/sys/fs/file-nr. If approaching file-max, increase.

Memory: swappiness, dirty ratio, overcommit

# /etc/sysctl.d/99-memory.conf (optional, part of full profile)
vm.swappiness = 10
vm.dirty_ratio = 20
vm.dirty_background_ratio = 5
vm.overcommit_memory = 1
vm.overcommit_ratio = 50

• vm.swappiness = 10 — kernel swaps only when absolutely necessary. For Redis/Memcached set 0 or 1 (but 0 does not disable swap completely).
• vm.dirty_ratio = 20 + vm.dirty_background_ratio = 5: at 5% dirty pages, kernel starts background write; at 20% it blocks write(). For NVMe, you can lower dirty_background_ratio to 3.
• vm.overcommit_memory = 1 + vm.overcommit_ratio = 50 — allow overcommit by 50% of RAM. Needed for Redis (fork during bgsave) and Java with large heaps.

What Our Kernel Tuning Includes

• Audit of current parameters: collect sysctl -a, ss -s, cat /proc/sys/fs/file-nr, profile load (RPS, connection types).
• Load profiling: use wrk, iperf3, pgbench for precise tuning.
• Profile preparation: detailed config with justification for each parameter per your workload.
• Application and monitoring: phased deployment with metrics tracking (latency, throughput, error rate).
• Documentation: description of all changes, why and how to roll back.
• Warranty: 3 months of post-release support — if the profile does not deliver expected gains, we adjust free of charge.

Tuning cost is calculated individually, but payback is usually less than 3 months due to performance gains. Hardware savings can reach 30% — for a medium-sized project, that's $2,000 per month saved. Request a consultation — we'll assess your project.

Work Process

1. Analytics — audit current sysctl, identify bottlenecks (TIME_WAIT, fd, buffers).
2. Design — select profile for load type (web, DB, streaming).
3. Implementation — apply via /etc/sysctl.d/, test on staging.
4. Load testing — wrk, iperf3, compare metrics.
5. Deploy to production with monitoring (Prometheus + Grafana).

Our team has 6+ years of DevOps experience and 80+ optimization projects. We guarantee results. Contact us to get a detailed analysis of your system.

Apply all configs with sysctl --system, check a specific parameter with sysctl net.ipv4.tcp_tw_reuse. Before changes, take a snapshot: sysctl -a > /tmp/sysctl-before.txt.

Full production profile:

# /etc/sysctl.d/99-highload.conf
# Network — TCP
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_slow_start_after_idle = 0
# Buffers
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# BBR
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
# File descriptors
fs.file-max = 2097152
# Memory
vm.swappiness = 10
vm.dirty_ratio = 20
vm.dirty_background_ratio = 5

More details on BBR on Wikipedia.