Block storage for VM: where the bottleneck is and what breaks
Block storage for VM rarely breaks on its own: a drop in disk operations comes from the storage layer.
When several machines share one volume or a common pool, their requests queue up together, and any peak load — copying, a heavy report, synchronization — slows down all neighbors. The application sees this as increased response time, even though CPU and memory are free.
The second source is the caching mode. If the hypervisor caches writes, while the guest has its own journal, acknowledgments return faster than the data actually lands on the medium.
Until the first failure, this looks like acceleration; after it — like loss of the last transactions. The opposite mistake is also costly: excessive write-through caching consumes volume resources and adds latency to every operation.
The third cause is the lack of queue separation. Data, transaction journal, snapshots, and copies on one volume compete for the same bandwidth and the same IOPS limit.
Separating them across different volumes or groups removes the storage bottleneck from the equation, and VM degradation no longer depends on someone else's activity.
Diagnostics begins with measurements: read and write latency, I/O queue length, volume busy time percentage, block size, read-to-write ratio. Then — selecting the volume type, cache mode, and monitoring policies for the load profile; we explain how this is addressed.
What moving VM disks to separate storage gives
When virtual machine disks reside on the hypervisor's local arrays, response time depends on neighbors: one machine doing background copying eats up the queue, and everyone else's read/write latency grows.
Finding the culprit is difficult, and explaining fluctuating service slowdowns to the business is even harder.
Moving disks to separate storage breaks this coupling. Each volume gets its own IOPS and bandwidth limits, the scheduler distributes them across nodes, and response time remains predictable across the entire load range, including peak windows.
The main effect is that a whole class of failures disappears. Data resides not on a specific server's disk, but on distributed storage at the Ceph level: a node failure does not mean volume loss, the machine restarts on another server.
Recovery time is measured in minutes instead of hours of troubleshooting and restoring from a copy — the cost of downtime falls accordingly.
Storage costs stop growing with the fleet of machines. Thin provisioning gives the volume only actually written blocks, a clone for a test environment is raised without a full copy, and outdated snapshots are cleaned up by a retention policy.
Then everything relies on monitoring: operation latency, queue depth, timeout percentage, and replica state. These metrics reveal degradation before it turns into a failure.
Disk connection modes: comparison of options
The disk connection mode determines what happens to a virtual machine when a node or network fails, and how freely it can be migrated between servers. Below are the real alternatives chosen at project start, along with the limitations of each scheme.
| Option | When it fits | Scheme limitations |
|---|---|---|
| Local server disks | Single node, minimal write latency needed, machine migration not planned | No replication: hardware failure means state loss; machine is tightly bound to the node |
| Network volumes | Machines are migrated between nodes, snapshots and rapid expansion needed | Network latency on every write, dependency on the channel, separate limit on IOPS |
| Distributed storage | Many nodes, need to survive failure of one or two without service interruption | Requires at least three nodes and a stable internal network; diagnostics are more complex than for a single volume |
| Replication over network volumes | Critical services where fast failover to a copy is important | Double space consumption; in synchronous mode write latency increases |
In practice, a hybrid is more common: the system and caches on local disks, data on network volumes. Then speed does not hinder machine migration, and backup and recovery remain predictable.
We record the choice in the placement scheme and test it with load before going live, so limitations show up on the test bench, not in production.
How we select and configure block storage for the load
Storage selection begins not with choosing a hardware model, but with the load profile: we collect metrics from running virtual machines and see what exactly hits the disk limit.
The read-to-write ratio, nature of peaks, latency requirements, and queue depth on the volume determine the media type, redundancy level, and topology — an error at this step does not show up immediately, but at the first load increase. Then follows an engineering procedure that we keep the same across all projects.
- Capture the load profile — collect statistics from hypervisors and guest systems over a work cycle, not a single day: where the flow is steady, where short bursts occur, which volume degrades first.
- Select storage — match the profile with the volume type and redundancy level, discard options that formally fit by capacity but won't withstand peaks.
- Design the volume scheme — allocation by purpose, file system, mount parameters, backups and their order, so the volume doesn't have to be stopped.
- Configure and connect — enable multipathing, set caching and queueing for the load pattern, verify that volumes assemble automatically after a node reboot.
- Test for failure — sequentially disconnect the storage path, a node, half the channels, and record how long failover takes and whether writes are lost.
- Set up monitoring — alerts on latency, capacity, wear, and I/O errors, so degradation is visible before a user notices it.
- Hand over procedures — placement scheme, volume expansion, rollback, and disk replacement procedures, incident response order — as a document the on-call shift works by.
What the customer receives: schemes, policies, procedures
The project closes not with a presentation, but with working documentation: the on-call shift and your engineer must be able to expand a volume or roll back a change without our prompting. Therefore, the set of artifacts is recorded based on the actual storage state, not drafts in chat.
- Volume map — which volume is assigned to which node, capacity, media class, mount point, purpose, and owner on the customer side.
- Placement scheme — pools, fault-tolerance levels, failure domain boundaries: you can see whose failure will affect a specific workload.
- Access policies — roles, zones, masking rules, who can change configuration and who can only read metrics.
- Replication settings — recovery point objective, copy retention depth, schedule, volume rollback procedure.
- Monitoring — metrics (free space, response latency, queue depth, I/O errors), warning thresholds, and alert recipients.
- Storage documentation — volume and network path configuration, node connection diagram, instructions for typical operations.
- Maintenance procedures — planned windows, media replacement, firmware updates, recovery testing from copies, responsible persons.
Each document is verified during an acceptance run: we simulate a failure and ask your engineer to perform the operation strictly according to the procedure. If a step cannot be reproduced, we fix the document rather than explain verbally.
Case study: how we eliminated I/O drops on a virtualization cluster
Where I/O was lost
A virtualization cluster of several dozen nodes, machine disks resided in block storage on Ceph, volumes grew in thin mode, and backup snapshots were written to the same pool where production traffic flowed.
At first everything was smooth, but as the number of machines grew, short drops appeared: I/O latency spiked tens of times during nightly backup, node migrations, and array rebuilds.
The share of operations longer than the target threshold grew from 1–2% to 15%, and some machines went into disk wait for minutes.
The cause was competing queues: long recovery shared bandwidth with production traffic, thin volumes fragmented, and the swap file and temporary directories lived on the most loaded pool.
What we changed and how we measured
We separated storage into pools for different load classes and left the common pool only for block storage for virtual machines.
We limited queue depth and background rebalance speed so background tasks wouldn't displace production traffic, moved the swap file and temporary files to separate volumes, enabled write cache, and aligned guest system partitions.
Telemetry was collected with the same load profile before and after: we looked not at averages but at percentiles and the share of operations above the threshold — averages are exactly what hide drops.
Result: the share of slow operations returned to 1%, latency during backup stopped growing, node recovery accelerated threefold, updates and migrations proceed without service pauses.
Monitoring was left with three metrics — latency, queue depth, and free space in the pool; alerts with a clear on-call engineer response procedure are configured for them.
How does block storage differ from file and object storage?
The difference is not in hardware, but in the data access model. Block storage gives a virtual machine a raw volume: there is no file system inside, it is created and maintained by the guest OS.
File storage, conversely, publishes a ready file system with paths and permissions — a single directory can be mounted by several machines at once.
Object storage is structured differently: data resides as objects with keys in a flat address space, access is via API, not through mounting.
Hence different tasks. A block volume is a VM root disk, a partition for a database, anything requiring predictable response to random operations and its own file system.
The price is binding to an availability zone and care with simultaneous writes: you cannot attach one volume to two machines for writing.
File storage is appropriate where several virtual machines must see the same data: shared directories, profiles, exchange folders. The limitation is metadata and locking: under small-file load they become a bottleneck.
Object storage is not suitable for a VM disk: an object is overwritten entirely, access latency is higher. But it is the right place for images, backups, and archives — versioning and retention are set by policy.
The choice of storage type comes from the load profile: share of random operations, simultaneous access from multiple nodes, object size. In practice, models are combined — block for machine disks, object for copies and images, file for shared directories.
Technical concerns: fault tolerance, encryption, migration
Objections to block storage for VM rarely concern speed — more often they are operational risks: volume behavior on node loss, migration without service interruption, encryption, and compatibility with an already deployed array. We address each.
What happens to the volume when a node is lost?
We distribute volume replicas across different failure domains, so a node failure does not stop the guest: I/O switches to a live copy.
We test volume fault tolerance with a separate scenario — in synchronous writes, acknowledged data is not lost; in asynchronous, the recovery point is known in advance.
How does migration without downtime work?
Within a cluster, the volume migrates together with a running machine: content is copied in advance, leaving a short switchover. Between storages, we enable staged replication at the volume level, verify checksums, and only then move the load. The unavailability window is seconds.
Can existing storage be connected?
Yes, an already deployed array connects as a separate pool — there is no need to rebuild it entirely. We migrate volumes in groups, verifying integrity and behavior under load for each, and only then migrate the rest.
Original copies remain until the end of acceptance, so rollback to the previous scheme remains possible.
Will volume encryption slow things down?
We perform encryption at the volume level, store keys separately from data, and set up scheduled rotation. The CPU handles acceleration, so overhead stays within single-digit percentages and is unnoticeable under typical load. But removal or disposal of the medium no longer exposes disk contents.
Let's discuss your task: we'll select a storage mode for your load
To make the initial assessment substantive rather than guesswork, we need input data about the load and current storage.
Let's discuss the task in this format: you send the load profile and constraints, we return an assessment and plan — with a specific volume type, settings, and monitoring scheme.
- I/O profile: read-to-write ratio, block size, required latency, daily and weekly peaks, seasonal spikes.
- Number of virtual machines, their roles, and which ones have storage critical for application response and which only for data preservation.
- Volumes and growth rate: current volume size, free space, expected growth per quarter.
- Platform and hypervisor, volume type, connection mode (shared or locally bound), requirements for snapshots and replication between availability zones.
- What already hurts: drops at peaks, growing queues, application timeouts, unexpected storage costs, manual operations when expanding a volume.
- Environment constraints: maintenance window, backup policy, encryption requirements, and access separation requirements.
In response, you will receive an assessment and plan: recommended storage mode for your profile, volume and queue parameters, snapshot and lifecycle settings, a set of metrics and thresholds for alerts, and a migration procedure with a rollback plan.
The next step is a short call based on the data sent: we clarify contentious points and fix the scope of work.