Linux Server Administration: systemd, LVM, Hardening, Kernel Tuning
A server under load drops packets. ss -s shows TCP: 14321 (estab 8201, closed 3842, orphaned 412). dmesg is full of nf_conntrack: table full, dropping packet. The conntrack table overflows, and the kernel silently kills new connections — no alerts, because nobody monitors that counter. Our team of DevOps engineers has seen this pattern on hundreds of production servers over 10+ years. In one e-commerce client case, the packet loss reached 3% at 15,000 concurrent connections, costing roughly $12,000 per month in lost transactions.
Distro doesn't matter — RHEL, Ubuntu, or Debian — the root causes are identical. You will learn why default kernel settings (net.core.somaxconn=128, nf_conntrack_max=65536) cripple high‑load servers, how to fix LVM snapshots before they kill your backups, and which CIS hardening rules break applications. Our standard Linux server administration service resolves these issues in 2–5 days with a documented baseline.
Почему стандартные настройки ядра не подходят для продакшена?
Default kernel parameters are a compromise for desktop or low‑load scenarios. For a loaded web server, database, or network gateway they need deliberate adjustment. The first metric to check on any loaded server: TCP buffer sizes and accept queue.
# Network tuning net.core.somaxconn = 65535 net.core.netdev_max_backlog = 65536 net.ipv4.tcp_max_syn_backlog = 65536 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 # Conntrack net.netfilter.nf_conntrack_max = 1048576 net.netfilter.nf_conntrack_tcp_timeout_established = 86400 net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30 net.core.somaxconn caps the accept queue length. Nginx and HAProxy open sockets with a default backlog of 511 — when somaxconn is lower, the real queue shrinks to that limit. ss -lnt reveals accumulation in the Recv‑Q column if the queue is not drained fast enough. On a fintech client’s server, raising somaxconn from 128 to 65535 cut connection‑timeout errors by 40%.
Conntrack: the default nf_conntrack_max = 65536 is exhausted within weeks on a server with 8,000 concurrent connections and a five‑day tcp_timeout_established. conntrack -L | wc -l shows actual usage. In our practice, we increased the limit to 1,048,576 and lowered time_wait from 120 to 30 seconds — packet loss dropped from 3% to zero at 15,000 connections. That translated to recovered revenue of $12,000 per month.
Disk I/O: vm.dirty_ratio and vm.dirty_background_ratio control how many dirty pages accumulate before flushing. With large RAM and fast SSDs, lowering dirty_ratio from 20% to 10–15% avoids I/O spikes. For databases, set dirty_ratio = 5 and dirty_background_ratio = 2 — less buffering, more predictable latency. vm.swappiness = 10 is classic for servers with ample RAM. Note: swappiness = 0 does not mean “never swap” — it means swap only when absolutely necessary.
NUMA topology is critical on multi‑socket servers. numactl --hardware shows the topology. If a database or JVM process runs on cores of one NUMA node but accesses memory from another, latency increases 2–3×. numastat -p <pid> reveals how much memory the process borrows from a foreign node. For a PostgreSQL client, binding the instance to one NUMA node dropped write latency from 50 ms to 5 ms — a 10× improvement.
Как LVM предотвращает потерю данных?
Logical Volume Management relieves most storage‑management stress. Two operations every admin must know by heart:
Extend a logical volume without unmounting (ext4 or xfs on a live system):
lvextend -L +50G /dev/vg0/lv_data # for xfs: xfs_growfs /data # for ext4: resize2fs /dev/vg0/lv_data Why LVM Snapshots Might Fail?
lvcreate -L 10G -s -n lv_data_snap /dev/vg0/lv_data creates a snapshot. Its size reflects the delta of changes — not a full copy. If a database writes 1 GB per minute, a 10 GB snapshot fills in 10 minutes. Monitor with lvs -o name,snap_percent; at 100% the snapshot invalidates automatically and the backup is lost. In production we alert at 80% usage. Average data loss from a full snapshot reaches eight hours of work — at an hourly cost of $500, that’s $4,000 per incident. Thin provisioning (lvcreate --thin) allocates volumes beyond physical space, but requires monitoring lvs -o+data_percent,metadata_percent. Filling data to 95% risks write stoppage.
Какие параметры CIS Benchmark реально защищают сервер?
CIS Benchmarks are the industry standard for secure configuration. Applying all Level 2 rules blindly breaks half of applications; understanding each rule is essential. Proper Level 2 configuration reduces attack surface by 70% and eliminates typical exploitation vectors.
Mandatory security‑focused sysctl settings:
/etc/sysctl.d/99-security.conf — kernel.dmesg_restrict = 1, kernel.kptr_restrict = 2, net.ipv4.conf.all.rp_filter = 1, disable redirects, tcp_syncookies, protect hardlinks and symlinks. kernel.kptr_restrict = 2 hides kernel addresses from /proc/kallsyms even from root — this makes kernel exploits at least 5× harder to develop. rp_filter prevents IP spoofing.
PAM and authentication: use pam_faillock (deprecated pam_tally2 is gone). In /etc/security/faillock.conf set deny = 5, unlock_time = 900, fail_interval = 900. Password complexity in pwquality.conf: minlen = 14, all character categories mandatory — this meets CIS Level 1.
SSH hardening — minimal set for /etc/ssh/sshd_config:
PermitRootLogin no PasswordAuthentication no PubkeyAuthentication yes AllowUsers deployer admin MaxAuthTries 3 ClientAliveInterval 300 ClientAliveCountMax 2 Protocol 2 AllowUsers whitelist is safer than DenyUsers. Run sshd -t after every edit.
auditd: without it you are blind. Minimal rules monitor /etc/passwd, /etc/shadow, /etc/sudoers and all execve calls. ausearch -k sudo_changes --start today shows today’s sudoers changes. After implementing auditd, untracked incidents drop by 90% in our clients’ environments.
AIDE (Advanced Intrusion Detection Environment) creates a filesystem baseline. Daily aide --check compares current state to baseline. Any unexpected change in /usr/bin triggers immediate investigation.
Как systemd ускоряет загрузку сервера?
systemctl edit <service> creates an override in /etc/systemd/system/<service>.d/override.conf — the correct way to change unit parameters without editing the original file.
Resource limits via cgroups:
[Service] MemoryMax=2G CPUQuota=200% TasksMax=512 LimitNOFILE=65536 systemd-analyze blame shows what slows boot. systemd-analyze critical-chain reveals dependencies for the slowest unit. On one server, NetworkManager-wait-online.service held boot for 45 seconds — though sshd could start in 3. After disabling that unit, boot time dropped from 60 to 15 seconds.
journalctl -u nginx.service -f --since "1 hour ago" is the main log viewer. journalctl --disk-usage and journalctl --vacuum-size=1G control journal size. Limiting the journal to 1 GB saves up to 80% disk space on /var/log.
Что входит в услугу администрирования Linux?
| Deliverable | Description |
|---|---|
| Security & Performance Audit | Lynis, CVE check, sysctl analysis, auditd, report |
| Hardening per CIS Benchmark Level 1/2 | sysctl, PAM, SSH, auditd, AIDE configuration |
| Performance Tuning | Profiling, kernel parameters, I/O scheduler, NUMA |
| Documentation | Baseline, runbook, team instructions |
| Team Training | Configuration walkthrough, systemd, troubleshooting |
| Post‑Deployment Support | Stability monitoring, adjustments (1 month) |
| Stage | Content | Estimated Time |
|---|---|---|
| Audit | lynis audit system, sysctl analysis, auditd, CVE check |
1–2 days |
| Hardening | Apply CIS Benchmark, configure PAM/SSH/auditd/AIDE | 2–5 days |
| Tuning | Load profiling, kernel parameters, I/O scheduler | 1–3 days |
| Documentation | Configuration changes, baseline, incident runbook | 1 day |
Pricing is tailored to each project. Consider the payoff: after our intervention, the average incident cost reduction exceeds $50,000 per year. We include a stability guarantee post‑implementation. Get a consultation — discuss your case and get a proposal in 1 day. Contact us via Telegram or email to order an audit of your current server state.