Configuring XFS/ext4 file systems for production servers

Why default XFS tuning drops writes at peaks

A production server with XFS "out of the box" holds only until the first serious write burst: nightly backups, database reindexing, archive unpacking.

At that moment write latency grows by an order of magnitude, the application catches I/O timeout errors, and disk metrics show it is not fully loaded—load goes into wait, not useful work.

The reason is how the file system allocates space. XFS splits the volume into allocation groups and hands out blocks in parallel, but with defaults free space quickly fragments into small non-contiguous pieces.

A new file gets blocks from different groups, metadata updates more often, and write streams queue on locks of these groups—instead of a linear stream, you get a choppy one.

The journal adds its own: buffer size, flush order, placement on media. When small synchronous operations and large sequential ones go through one flush point, large ones wait for small ones.

This is the I/O degradation seen on graphs as "the disk is slow", although it is the queue of write operations that is slow, not the media.

ext4 behaves differently and up to a point forgives carelessness, but hits its ceiling on multithreaded writes with large volumes. So the choice between them is not a matter of taste, but of workload profile.

Next—mount options for a specific profile, size and alignment of allocation groups, allocation policy for different file types, journal offload, and monitoring of free-space fragmentation. We break down how this is closed.

What correct layout and mount parameters give

Volume layout and mount options determine three measurable things: write latency spread under peak, the amount of operations lost on sudden power loss, and the duration of server return to service after failure.

In current Red Hat builds XFS is selected by default, but defaults are designed for "average" load—they are refined for the task. Below is what we check on every production node before commissioning.

• Aligning the start of volume and partitions to boundaries the drive understands: read-modify-write mode per block goes away, writes stop being many times slower than reads, and latency does not depend on data position.
• Reserve for metadata and inode size when creating the volume: space for them does not run out long before the volume itself is exhausted—no need to stop service and migrate data.
• File system journaling: after power loss metadata consistency is restored, and recovery reduces to replaying last transactions—seconds instead of a full volume check taking tens of minutes.
• Journal buffer size and count: write bursts during peak hours do not queue—write latency spread narrows instead of growing in steps.
• noatime or relatime in mount options: removes tens of thousands of unnecessary metadata updates per day, the drive does not write what no one reads.
• I/O queue for the workload profile: with a mixed stream, write wait is reduced, heavy backup stops delaying transactions.
• Regular fstrim and fill control: free blocks are returned to the drive, write degradation does not occur suddenly.
• Layout and mount configuration in the repository: recovery after server failure is reproducible, not "from the engineer's memory".

Comparison of approaches: option, applicability, limitations

No file system wins "in general"—comparison makes sense only together with how data is laid out across volumes. XFS handles parallel writes and large files better, ext4 is more predictable on small operations and simpler to maintain. Below are four layout schemes that are actually chosen between, with honest limitations of each.

In practice schemes are combined: root and system partitions are kept on a journaling file system, data and journals are moved to separate volumes, and rollback before update is done by snapshot.

Order is chosen before deployment—relayout on a live server costs downtime, so the decision is fixed in advance and verified in a load test.

Work order: from audit to handover to operations

Work follows one route: first—facts about storage, then configuration, and only after confirmed tests, production. No parameter is taken "as everyone else": it is either confirmed by the workload profile or not applied.

1. Storage audit. We capture volume utilization, file size distribution, share of sequential and random writes, peak windows. Output—a workload profile instead of generalities about "many small files".
2. Data placement scheme. We distribute data across volumes and mount points: journals and metadata separately, hot writes separately, archives separately. The scheme immediately shows which partition degrades first.
3. File system parameter calculation. For this scheme we set block size, journaling mode, inode limits, and mount options. We explain each item: what it gives and what we pay for it.
4. Disk load testing. We run synthetic and real-profile replay: look at latency and throughput over a long interval, not over a one-minute burst.
5. Fault tolerance check. We simulate disk failure, disconnection, volume overflow, and power loss, then recovery and integrity check. We record downtime and the amount of data brought up from reserve.
6. Production cutover. We apply in stages, with a maintenance window and rollback plan: if metrics go beyond bounds, we roll back to the previous configuration without data loss.
7. Handover to operations. We provide documentation, placement map, monitoring thresholds, and procedures for typical failures; we stay in touch until your team performs the first scheduled operation on its own.

Set of documentation, configurations, and procedures

We provide not a set of scattered commands, but a kit by which infrastructure can be maintained without us: an engineer opens the needed file and sees not only what is configured, but why exactly this mode was chosen. Below is what remains with your team after the work.

• Mount configuration — working files for each server with line-by-line explanation of options: why noatime is enabled, what justifies discard on drives, where the default journaling mode is deliberately left.
• Partition and volume scheme — map of disks, partitions, and mount points with alignment to block boundaries and reserved space for growth.
• Data access policies — owners and permissions on application and journal directories, separate service accounts instead of running as superuser.
• Volume expansion instructions — step-by-step order: add capacity, grow logical volume, then extend the file system online (xfs_growfs or resize2fs) without stopping service.
• Server maintenance procedures — frequency of fill control, alert thresholds, windows for scheduled integrity checks, and actions on power failure.
• Recovery order — what to do when metadata is damaged, how to restore from backup, and how to make sure data is intact afterward.
• Change log — every configuration edit with reason and rollback method.

Case: eliminating write degradation on a file server

A log collection server received a stream from several dozen services and at peak load stopped keeping up. The write operation queue grew, latency exceeded timeouts, and after reboot volume recovery took hours.

Losses were implicit: some events were dropped on the source side, and logs did not reach disk exactly when incident analysis was underway.

Metrics were captured in Prometheus — I/O queue depth, write wait time, and recovery time.

The picture was typical: the file system journal lay on the same device as data, array alignment was absent, and the application synchronized every event, turning a stream of writes into a stream of waits.

What changed

Data and the file system journal were separated onto different devices, moved to XFS with a block for the real write profile and array stripe alignment.

Added quotas and old segment cleanup policy, reduced the frequency of forced synchronizations at the application level, leaving delivery guarantee at the file system level.

Metrics before and after

Peak operation queue dropped from thousands of outstanding writes to hundreds and stopped growing over time. 99th percentile latency decreased by roughly twenty times, recovery window after failure—from hours to minutes.

Event drops at peak are gone, and load headroom allows connecting new sources without replacing hardware.

XFS or ext4 tuning: what to choose for current load?

There is no clear winner: both systems journal metadata, but lay out data differently and handle parallel writes differently.

File system choice is not by "newer or older", but by I/O profile: average file size, share of random access versus sequential, number of concurrent writers.

ext4 is sufficient where load is mixed and files are small—configs, sessions, directories with millions of objects, application logs. Delayed block allocation gathers small writes into one operation, and the metadata journal gives predictable recovery after failure.

The classic journaling scheme confidently covers web nodes and auxiliary services, and the volume can be shrunk if needed without recreation.

XFS reveals itself on large files and parallel writes: separate allocation groups let streams write to different areas of the volume without hitting a common lock. This is visible in streaming tasks—logs, backups, media, databases with large blocks. The flip side: mass deletion of small files is slower, and the volume cannot be shrunk.

We capture load criteria from the service itself: reproduce its profile on a copy of data and compare real operation times, not synthetic numbers. Then select mount options and block size for average write, check volume behavior when filling and emergency shutdown. The decision relies on measurements of specific load, not on a general argument.

Technical doubts: compatibility, risks, and rollback

The customer's engineering team rarely asks which file system is better than another. The question is more practical: what will happen to an already running server if it is changed.

Below is what we cover before work begins: data migration on a live system, impact on backups, rollback order, and integrity check.

Changing a file system is not reformatting a disk, but moving the tree to a prepared volume with a short service switch; the old volume remains untouched until the end of the observation period, so the way back always exists.

How are changes rolled back if something goes wrong?

Rollback is returning the mount point in configuration and restarting the service, without copying data back. The old volume is not changed or released until the agreed observation period on the new one ends. If a problem surfaces after cutover, return takes the same time as moving forward.

What changes for backups?

Usually nothing: state snapshots are made at volume level, not inside the file system, and the copy scenario remains the same. Before work we check that the backup agent correctly reads permissions and extended attributes on the new volume, and restore a control copy in full.

How do you verify data integrity after migration?

We compare the tree by file list with checksums before and after, check permissions, owners, and hard links, then selectively read large files and images. Discrepancies are resolved before service cutover, not after. Separately we look at metadata utilization—it shows headroom for load growth.

How to start: what to clarify and in what order to proceed

Storage work assessment starts not with numbers, but with inputs: without them the implementation plan comes out generic and accounts for neither workload profile nor data preservation requirements.

Below is what we clarify in the first conversation so we can return with a concrete sequence of stages and timelines.

• Workload profile: what writes to disk most often—database, application logs, message queues, container layer; what is the read/write ratio and are there bursts during reporting hours.
• Hardware layer: local disks or network storage, array and its fault tolerance level, disk reserve for future growth.
• Current state: which file system is deployed, with what mount and creation parameters, what manual edits were made after installation and by whom.
• Data requirements: whether state snapshots are needed, how often backup is taken, and what recovery time is considered acceptable for the service.
• Work window: whether putting the server into maintenance mode is allowed, whether migration without service downtime is possible, and who confirms the window.
• Volume forecast: how quickly data grows and whether partition expansion is planned in the foreseeable future—headroom in layout depends on this.

The sequence of stages after this is simple: fix inputs, prepare a work plan split into preparation, migration, and verification, run configuration on a test bench with a copy of data, only then touch production.

Describe the task—we will return with a storage estimate, implementation plan, and timelines, not with general words.