Configuring the I/O Scheduler on Linux: Choosing a Scheduler for NVMe, SSD, HDD

I/O Scheduler: Where the Task Queue Holds Back Disks

The kernel block layer accepts read and write requests and puts them into a queue of requests to the disk — the drive physically processes them sequentially, and the scheduler sets the order. Under a homogeneous load of large sequential operations, this layer is almost invisible.

Problems begin with a mixed profile: database transactions, the journal, backup dumps, and file synchronization compete for the same drive.

With the wrong scheduler, requests lose order. The device jumps between distant addresses instead of merging adjacent operations, and idles between them.

Storage response latency grows several times over while resources are formally free: utilization shows half, but the application waits for a response to a request sitting in the queue.

This is how an I/O operations bottleneck arises that is not visible in either CPU load or the amount of free memory.

Different media require different behavior. Fast NVMe with a deep internal queue is only hindered by reordering — there it is better to submit requests directly.

Hard disk arrays need a scheduler that coalesces adjacent operations and prevents one task from occupying the entire queue. Configuring the I/O scheduler is finding a trade-off between latency, throughput, and fairness of distribution for a specific workload profile.

Next — how to find this spot in metrics and how it is resolved.

What Changes in Disk Operation After Selecting the Mode

Mode selection begins with analyzing the disk workload profile: how many operations are in the queue at the same time, what proportion of reads versus writes, whether there are spikes.

While there is one mode for all cases, the kernel builds the queue according to an averaged pattern — and on a real stream this queue works against the application.

After selection, the I/O scheduler aligns with the actual profile: queue depth, priorities, and the order of issuing operations stop conflicting with how the drive physically processes requests.

The effect is most visible under mixed read and write load. Modes with queue separation prevent the read stream from waiting while the drive merges and reorders writes — operations get a time budget instead of falling into a common tail.

Throughput on a mixed stream grows without replacing hardware, and the waiting time of an individual operation is reduced because the queue stops being a dump of heterogeneous requests.

The second effect is evenness of drive wear. When the order of issuing operations takes into account the write structure and does not churn blocks pointlessly, rewriting is distributed more evenly, and unnecessary reordering does not burn resource.

This directly affects service life: for drives with a limited number of cycles, even wear pushes degradation further away rather than bringing it closer with peak spikes.

The third, most noticeable result for business logic is stability of application response. The class of failures "disk busy, application waiting" disappears: a request does not hang at peak because the queue is designed for it.

Tail latencies level out, the spread between fast and slow operations shrinks, and the service maintains even response time under load, not only when idle.

Choosing a Scheduler for NVMe, SSD, and HDD

The scheduler decides in what order the kernel block layer issues requests to the drive and how many it keeps in the queue. On NVMe, the controller reorders and merges commands itself, so software sorting only adds latency.

On HDD, order determines the number of head movements, and sorting by sector gives a multiple-time gain.

The choice is a trade-off between throughput, latency, and fairness between processes, not a search for the "fastest" option. In the Linux kernel, several schedulers are available, and they can be switched on a live device, without reboot.

Separately, we check the request reordering mode: on HDD and SATA SSD, merging adjacent blocks reduces the number of operations, while on NVMe it duplicates the controller's work and adds overhead.

We choose the final option based on measurements on the real workload profile — synthetic tests on an empty disk almost always lie.

Implementation Order: From Workload Audit to Handover of the Maintenance Procedure

The I/O scheduler cannot be set "by default": a node under a database, a service with logs, and image storage produce different disk access profiles. The order of work goes from measurements to decision — first audit, then mode, then consolidation in configuration.

1. Disk workload audit. We capture queue depth, read-to-write ratio, request size, and response time for each device — this is the baseline against which we compare everything else.
2. Queue layout. We label disks by purpose and calculate how many queues fall on a device and where the bottleneck is: the drive itself or the queue to it.
3. Mode selection. On NVMe, request reordering gives almost no gain; on SSD with parallel queues, a multi-threaded mode works; on HDD, one with request accumulation and sorting.
4. Test bench for mode validation. We deploy a copy of the environment, repeat the profile from the audit, and compare latency and throughput with the baseline. A mode with no gain on the bench does not go to the production node.
5. Switching by node. We change in stages, with one-step rollback, and immediately fix the rule in configuration — otherwise after a reboot or kernel update the value will return to the previous one.
6. Validation on real traffic. We look at behavior at peak and during low-load hours: response time, queue length, regression on neighboring services.
7. Handover of the maintenance procedure. We provide the document: which mode is assigned to which class of disks, how to check the current value, how to roll back, and which metrics to watch.

Documentation and Settings Package on Output

The package is assembled so that the state of the bench can be reproduced on a new node without us — from the repository, not from chat messages. Everything we changed on live systems is recorded as files and short instructions suitable for review by the on-call shift.

• I/O scheduler and device rule configuration files — under version control, with a comment for each exception: why a non-default profile was chosen for this volume.
• Queue and priority layout: which queues serve databases, journals, backups, what their weight and depth are, what happens on overflow.
• Map of task distribution across drives: volume, media type, assigned scheduler, and the measurement result on which the choice is based.
• Server maintenance procedure: what to recheck after a kernel update or disk replacement, at what point the profile stops being current.
• Instruction for the on-call shift: how to switch the scheduler on a running system, how to restore the previous value, and by which metrics to understand that things got worse.
• Measurement methodology: load scenarios, number of repetitions, set of captured latency and throughput indicators — so that disputed decisions are settled by numbers.
• Inventory of nodes with deviations from the reference and a plan to bring them to a common state.

Documentation for disk configuration is written for an engineer who will see the server for the first time: without references to our internal context and without steps that require the author.

Case Study: A High-Load Database Stopped Hitting the Disk Limit

The node combined a database and file tasks: dumps, static content ingestion, and backups ran on the same drives as transactions.

Every few minutes response dipped — the disk request queue grew, operation wait time jumped several times, and the share of CPU idle waiting for I/O reached a quarter with moderate CPU load itself.

The analysis showed that the issue was not drive speed but dispatching. The I/O scheduler mode remained from an earlier configuration with slow disks: it resorted requests and added latency where it was not needed.

PostgreSQL and background tasks ended up in the same queue without priority classes, and queue depth and request size did not match the workload profile.

What changed: volumes with the database were moved to a multi-threaded scheduler without extra sorting, traffic was separated into priority classes, and the bandwidth share for backups was limited; request size was increased for sequential reads.

Separately, the database journal and static ingestion were split across different volumes so that background operations would not displace transactions.

Metrics before and after: disk operation wait time at the 99th percentile decreased by 6–8 times, and the I/O wait share fell from a quarter to 2–3%.

Periodic response dips stopped — instead of spikes every few minutes, the graph became even, and the backup window shrank so much that it stopped overlapping with peak load. The node stopped hitting the disk limit, and load headroom remained on the same hardware.

What to Do If a Node Mixes a Database and File Tasks?

A single I/O scheduler on a mixed node always loses: it is tuned for one profile.

Enable bfq for fairness of file operations — transactions will start paying overhead on every operation; install a simple scheduler for minimal latency — background tasks will occupy the entire queue.

Therefore, mixed load on a node is separated across devices rather than searching for one universal rule.

Splitting disks by task gives predictability: the database journal and data — on fast drives with minimal reordering, while dumps, backups, and replicas — on separate volumes under bfq.

If there is physically only one drive, only priorities remain: the device queue is shared, and splitting into partitions for different profiles changes almost nothing.

I/O priorities are set independently of process priority: real-time, best-effort, and idle classes, as well as weights and operations-per-second limits in cgroup.

We keep background jobs in idle — such a class gets bandwidth only when the device is free and cannot displace transactions. A spike from backup is removed by an operation limit, not by raising priority.

We validate the result by measurement: we submit both loads simultaneously and look at the latency tail — the 95th and 99th percentiles, not the average, because the user feels the tail. We fix the rules in configuration so that they survive reboot and kernel update.

Technical Doubts Before Changing the Queue Processing Mode

The engineering team argues not about the idea itself, but about the cost of a mistake. Changing the queue processing mode is a rare case when a setting is applied on a running system and just as quickly reversed: the order and grouping of requests change, not the data layout. Below are the doubts we resolve before making the change.

Is there a risk of data loss?

The scheduler does not write to the drive — it decides in what order the kernel issues requests. The format, metadata, and contents of volumes are not affected, so there is no data loss from the mode change itself.

The risk is different: editing under peak load without monitoring latency and queue depth. Therefore, we first capture baseline metrics, then change.

What to do when drives are replaced?

A new disk comes with the default mode — hence "after replacement it became slower." We bind the rule to the media type, not to the device name: NVMe, SATA SSD, and HDD receive their own modes automatically. Replacing a disk or adding a node then does not require manual configuration.

How to roll back if things get worse?

Restoring the value is one operation, applied without reboot or downtime. We save a snapshot of the current parameters before editing so that rollback is not searched for under pressure.

Before rolling out to the entire fleet, we run the mode on one node and compare latency with the baseline.

Let's Discuss the Task and Come Back with an Estimate and Plan

To discuss the task substantively, it is enough to describe the environment: which nodes we service, which drives they have, and what exactly is slow.

We do not start with general questions — first we look at how the I/O queue behaves under your load and which parameters are currently set.

To start, send:

• composition of nodes and media types — NVMe, SATA SSD, HDD and their role in the system;
• load profile: database, message queues, file storage, virtualization;
• current scenario — which scheduler is selected and how queue parameters are set;
• symptom with numbers: write dips, increased response time, uneven core load;
• constraints — maintenance window, rollback requirements, and internal procedures;
• a bench or server where behavior can be reproduced before changes in the production environment.

Then the analysis proceeds step by step: we compare the configuration with the workload profile, take measurements before and after changing the scheduler on each media type, and check behavior under peak, not in a calm mode.

As a result, an estimate and work plan arrive: stages, target queue parameters for each class of media, rollout order with a rollback point, and a list of checks by which the result is visible.

Linux server configuration is done so that changes live in configuration and are transferred to new nodes with one command, rather than remaining an edit on one server.

Describe the task — we will come back with an estimate and plan.