Disk subsystem benchmarking: where performance is lost
The disk subsystem is almost always the link that hits the ceiling first when the load grows.
The database starts taking longer to return queries, the queue accumulates unprocessed messages, virtual machines on shared storage "hang" — and all this while the CPU and memory are only half loaded.
Disk subsystem benchmarking is needed not for pretty numbers in a report, but to separate the real I/O bottleneck from symptoms that merely resemble it.
The problem is that without measurements, the cause is sought at random.
An administrator sees growing read and write latency, adds memory, changes kernel parameters, migrates the service to another node — and the degradation returns a week later, because the root cause was different: exhaustion of the queue on the storage device, the wrong block size, or the fact that the write load profile does not match the profile the storage was selected for.
Each such iteration means downtime, lost team time, and risk for the service that has been running at its limit the whole time.
Disk subsystem performance cannot be assessed with a single number.
Sequential reads with large blocks, random writes with small ones, and mixed workloads dominated by reads are three different scenarios with different behavior, and optimizing for one easily makes the other two worse.
That is why a correct measurement always begins with the question "what is the profile of our real workload," rather than with launching the first test that comes to hand.
Next, we break down how this is addressed: which scenarios we reproduce, how we capture latency and throughput metrics, and how, based on the results, to distinguish a shortage of storage resources from an error in the application configuration.
What a correct measurement gives: load modes and metrics
A correct measurement begins with choosing a load profile, not with running the first command that comes to hand. Sequential reading in blocks of several megabytes shows the throughput ceiling — this is the scenario of backup and uploading large files.
Random access in 4 KB blocks with a queue depth of 32 reproduces the operation of a database: here the ceiling is set not by throughput, but by latency per operation.
Load profiles: what to measure and why
- Sequential access — large blocks, one thread or several. Shows how much data will pass during copying, streaming, replication.
- Random access — small blocks, high queue depth. Shows behavior under transactional load and with mixed read-write.
- Mixed mode — write share of 70/30 or 50/50. Closest to the real operation of a service where reading and writing happen simultaneously.
Metrics by which decisions are made
Throughput (MB/s) answers the question "will the data be transferred within the maintenance window". Latency — at the 95th and 99th percentile — answers the question "will database queries start hitting the disk".
Average latency lies: rare but long operations drown in the total count and are not visible, yet it is precisely they that cause spikes in application response. That is why we look at the tail of the distribution, not at the average.
What to measure with: comparison of approaches
| Tool | When it fits | Limitations |
|---|---|---|
| fio | Flexible profiles: block size, queue depth, write share, number of threads are all configurable. The main tool for reproducible measurements | Requires careful configuration — wrong parameters produce a beautiful but meaningless result |
| bonnie++ | Quick check of the file system and disk I/O "out of the box" | Less control over the profile, less suitable for precise comparison of configurations |
| dd | A one-off measurement of sequential write and read, a check that "the disk is alive at all" | Does not provide latencies and random access, is easily fooled by the cache |
For reproducibility, our engineers warm up the cache, set the file size larger than the amount of memory, disable caching at the file system level, and run each profile several times.
Otherwise the spread between runs outweighs the difference between configurations, and the measurement loses its meaning. We compare on the same set of parameters — this way the effect of changing the disk type, array settings, or file system is visible, rather than noise.
We summarize the results in a table "profile — throughput — latency p95/p99", and it is on that basis that the configuration decision is made.
fio
How we conduct measurement: stages of engineering work
Before starting measurements, we break down the task: which load profile matters — random reads in small blocks, sequential writes in large blocks, a mixed stream dominated by reads.
This determines the choice of tool and parameters; otherwise the numbers look good but have nothing to do with real work.
Next, we fix the target configuration — media type, caching mode, file system — so that the test bench replicates production rather than abstract "hardware in general."
- Task audit and hypothesis formulation. We gather the application requirements: I/O block size, read-to-write ratio, queue depth, latency sensitivity. We formulate exactly what we are testing — throughput ceiling, behavior under load, or degradation over time.
- Preparing the test bench for measurements. We deploy a clean configuration, disable background processes and excess cache, align partitions, and warm up the media before the series. We run the same scenario on different settings, changing only one variable at a time.
- Capturing the load profile. We run a series of measurements on several tools to eliminate methodology error: one provides a detailed picture of latencies, the second — a steady stream, the third — a quick check on sequential operations. A discrepancy between them is a signal to re-verify the conditions.
- Analyzing results and finding the bottleneck. We look not at averages but at the latency distribution: the ninety-ninth percentile shows what the user will feel, whereas the average hides it. We correlate the measurements with the file system, scheduler, and cache settings.
- Verifying recommendations with a repeat run. We confirm each setting with a control measurement on the same scenario — otherwise it is a guess, not a conclusion. We record which parameter produced a gain and which turned out to be neutral.
- Delivering the report with measurements. We hand over tables for each scenario, latency distribution charts, a list of changed parameters, and the expected effect. We separately note which conclusions carry over to production and which are tied to the test bench.
- Configuration recommendations. We consolidate the result into specific configuration changes and the order in which to apply them, so the team can implement them without a repeat analysis.
This order eliminates the main problem with measurements — when the numbers exist but no decision can be made based on them. Describe your task and current configuration — we will get back to you with an estimate and a work plan.
How does a synthetic test differ from a real workload?
A synthetic test feeds a predefined stream of operations to the disk: a fixed block size, queue depth, read/write ratio, and no contention for the resource.
A real workload does not look like that — it is non-uniform over time, depends on the file system cache, on how the application groups requests, and on what is happening in adjacent layers: the network, the database, the scheduler.
That is why the numbers from a utility's report and the behavior of a production system diverge — not because the tool is lying, but because they measure different things.
The key source of divergence is the cache. If a test reads a file that is already in the page cache, the measurement shows the speed of memory, not of the storage medium.
A correct measurement requires either direct I/O bypassing the cache, or a preliminary cache flush before each run; otherwise the reliability of the measurements drops to zero.
The second factor is queues: the queue depth in a test is set manually, while in production it is formed by the number of simultaneous requests from applications. The same storage medium at queue depth 1 and at queue depth 32 behaves like two different devices.
The third layer is the conditions for a correct comparison. Two storage options can be compared only if the block size, read/write ratio, queue depth, cache bypass method, and run duration all match. Changing any parameter between measurements makes the conclusion invalid.
Separately, it is worth remembering warm-up: during the first seconds of a test, the storage medium and the controller are still reaching their operating mode, so short runs show an inflated or deflated result.
The practical takeaway: a synthetic test answers the question "what is the upper limit of this storage medium under a given profile," not "how will the system behave under load." It is needed to filter out obviously unsuitable options and to catch degradation after changes, but the decision about the configuration is made based on a mixed scenario close to the application's profile.
We build such scenarios from logs of real operations and keep the conditions identical between runs — then the comparison yields an answer that is confirmed on the production system.
Case study: how measurements eliminated storage degradation
A virtualization cluster of six nodes served a product environment: databases, cache, queues. Complaints came in waves — "everything slows down in the evening" — but CPU and memory metrics remained normal, so the problem was blamed on the application.
Until an I/O profile was taken at the volume level: it turned out that peak read operations were hitting the disk array queue, and storage latency grew in steps upon reaching a certain level of concurrent requests.
What the measurements showed
First, a baseline was taken: sequential and random read and write operations at different block sizes, different numbers of concurrent threads, and different queue depths.
The runs were performed on an empty volume and on a production one to see the difference between "datasheet" speed and the real picture under load. Separately, a mixed workload profile was checked — the read-to-write ratio matched production.
The picture turned out to be non-linear: up to a certain number of concurrent operations, latency held steady, then grew severalfold. This was the source of the evening failures — the application increased parallelism, and the storage responded with a queue instead of data.
What was changed
| Option | When it fits | Limitations |
|---|---|---|
| Tuning the scheduler and queue depth | Latency grows with parallelism | Doesn't help if it's hitting the physical limits of the media |
| Separating volumes by workload profile | Mixed reads and writes on a single array | Requires rebuilding volumes and migrating data |
| Write cache at the controller level | Writes dominate | Risk of data loss without protected power |
| Replacing media with faster ones | Resource and latency headroom exhausted | Capital costs, downtime for migration |
A combination was chosen: volumes were separated by profile, the scheduler was tuned for mixed workload, and parallelism was limited on the application side.
Measurements were repeated after each step — to see the contribution of that specific change rather than the cumulative effect.
Result
Storage latency under peak load leveled out, and the spread between minimum and maximum response time shrank severalfold. The evening failures disappeared as a class: the bottleneck was no longer hidden.
A side effect — the application stopped "catching up" to the queue with retries, and the load on the cluster decreased.
The measurement methodology was codified in a regulation: the same runs are now performed before every storage expansion, and degradation is caught at the planning stage rather than in production.
The toolset — fio for I/O profiles and bonnie++ for file operations; the combination covers both synthetic benchmarks and behavior under real load.
Doubts before starting: compatibility, access, rollback
Disk subsystem benchmarking on a production server is a legitimate concern: measurements create load, and some utilities write test files directly to the working file system. We do not run measurements "blindly" on the production environment.
First, we capture a profile of the current load, select a window of minimal activity, and choose a mode in which the test does not compete with working processes for the disk queue.
What if the measurements still affect the production system?
Before launching, we establish a rollback point: a snapshot of the configuration, current mount and I/O scheduler parameters, and the list of running services. We write test files to a separate directory with size and time limits, rather than over working data.
If the load exceeds the agreed threshold, the measurement is stopped, changes are rolled back according to a pre-defined plan, and the service returns to its original state.
What if access to production is needed? We work on the principle of least privilege: reading metrics and configuration, running utilities within an agreed window, without permanent administrator access. All actions are logged, access is granted for the duration of the work and closed upon completion.
What if the measurements show degradation? That is the result: a report with the load profile, bottlenecks, and configuration recommendations — with no changes to the configuration until you approve them.
Based on the results of the work, you receive:
- measurement launch scripts with fixed parameters and clear thresholds;
- a profile of the current load and measurement results in a readable format;
- a rollback plan for changes with rollback points;
- a list of granted accesses and permissions with the procedure for closing them;
- documentation on the measurement methodology so you can repeat them yourself;
- recommendations for configuring the disk subsystem with justification for each change.
Describe the task and the current environment — we will come back with an estimate and a work plan, including a scheme for safely conducting measurements.