ZFS Server Setup: Pools, Datasets, Replication

A system administrator discovers that a hardware RAID controller is failing, and data recovery takes weeks. We've encountered this dozens of times — the solution is called ZFS. It's not just a file system, but a full volume manager with data integrity protection, snapshots, and real-time replication. ZFS prevents data loss at the block device level by automatically correcting bit rot. Unlike hardware RAID, ZFS is not tied to a specific controller — you can migrate the pool to another server. Savings from avoiding expensive controllers can reach 50% of the storage budget — for a server with 10 disks, that's hundreds of about $9–13 in savings per year. Order ZFS setup tailored to your workload — we will select the optimal configuration of pools and datasets, ensuring fault tolerance and performance.

Why ZFS is more reliable than hardware RAID?

Hardware RAID controllers are expensive, tied to specific models, and do not check data integrity. ZFS uses parity at the file system level (RAIDZ) and automatically corrects bit rot. According to OpenZFS, the probability of an undetected error on ZFS is 100 times lower than on classic RAID with ext4. Additionally, ZFS supports instantaneous snapshots, lz4 compression, and encryption without performance loss.

How to optimize ZFS for databases?

For PostgreSQL, we create a separate dataset with recordsize=8k and logbias=throughput — this way ZFS does not cache the database journal, which the database writes via fsync anyway. For MySQL, recordsize=16k is optimal. In any case, we disable atime and enable lz4 compression: space savings up to 40% without IOPS loss. For high-load systems, we also set primarycache=metadata to cache only metadata.

Why ashift=12 is critical?

Modern disks have a physical sector of 4K, not 512 bytes. If ashift=9 (512 bytes), each read/write operation incurs overhead for translation, reducing performance by up to 50% compared to ashift=12. Setting ashift=12 ensures alignment and optimal speed.

Key concepts before starting

Pool (zpool) — a group of physical disks, analogous to a RAID array. Dataset — a file system or volume inside the pool. Parameters (compression, encryption, quotas) are set at the dataset level and inherited by child datasets.

A fundamentally important detail: ZFS manages its own cache (ARC). No need to tune vm.dirty_ratio — ZFS works outside the kernel page cache. ARC by default occupies up to half of RAM, which is correct for a dedicated storage server. To improve performance, you can add L2ARC on SSD, but only if you have sufficient RAM (at least 8 GB per 1 TB of L2ARC).

Creating a pool and basic configuration

Create a mirrored pool from two disks (ashift=12 is critical):

zpool create -o ashift=12 \
    -O compression=lz4 \
    -O atime=off \
    -O xattr=sa \
    -O dnodesize=auto \
    tank mirror /dev/sdb /dev/sdc

For large volumes, use RAIDZ2 (analogous to RAID6) with recordsize=1M:

zpool create -o ashift=12 \
-O compression=lz4 \
-O atime=off \
-O recordsize=1M \
storage raidz2 /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi

Datasets and hierarchy

The correct dataset structure saves time during management. Create a hierarchy and set parameters for PostgreSQL:

zfs create tank/data
zfs create tank/data/databases
zfs create tank/data/applications
zfs create tank/backups
zfs create -o recordsize=8k -o logbias=throughput -o primarycache=metadata tank/data/databases/postgres

Snapshot before update: zfs snapshot tank/data/databases/postgres@before-upgrade-$(date +%Y%m%d).

Replication between servers

Incremental replication via snapshots over SSH without additional software:

zfs send tank/data@snapshot-01 | ssh backup-server zfs receive backup/data 

For automation, use zrepl — it manages scheduling, pruning, and replication.

Ensuring fault tolerance

Regular scrubs check the integrity of all data and restore corrupted blocks. Schedule scrubs once a month for desktop systems and once a week for database servers. Monitor pool health with Prometheus:

zpool status -v tank arc_summary | grep -E 'ARC size|Cache Hit' 

For automated metric collection, use node_exporter with the ZFS module enabled. When errors are detected, ZFS automatically switches to healthy copies (in mirror configurations).

Dataset encryption

ZFS native encryption (aes-256-gcm) works at the block device level. The key is loaded at mount time; automation can be done via systemd or Vault.

Work process

1. Analysis: audit of the current storage scheme, load profile, SLA requirements.
2. Design: topology selection (mirror/RAIDZ), pool sizing, snapshot policy.
3. Implementation: ZFS deployment, dataset configuration, replication, monitoring.
4. Testing: load testing, fault tolerance checks, restore from backup.
5. Deployment: handover of documentation, staff training, post-launch support.

Get a consultation from a DevOps engineer: contact us for an audit.

What's included

• ZFS deployment on servers (Debian/Ubuntu/CentOS)
• Pool creation with optimal parameters (ashift, compression, recordsize)
• Dataset configuration for specific DBMS and applications
• Automation of snapshots and replication (zrepl, sanoid)
• Pool monitoring via Prometheus/Grafana
• Encryption of sensitive data
• Documentation: pool scheme, recovery runbook, maintenance plan
• Team training: typical operations, troubleshooting

We guarantee that the configuration adheres to OpenZFS best practices. Our experience: 10+ years in infrastructure projects, over 50 successful ZFS storage deployments.

Typical mistakes

• Adding L2ARC on cheap SSD without enough RAM — wastes RAM on metadata.
• Creating a pool with ashift=9 on 4K disks — halves performance.
• Running scrub during peak hours — schedule it for a nighttime window.

Timeframes

Cost is calculated individually after an infrastructure audit. We'll evaluate your project — contact us.