Kubernetes CSI driver setup: Ceph, NFS, EBS, GCE PD

Why Kubernetes CSI drivers slow down under load

A volume mount failure looks the same from inside and outside: the pod hangs for hours in ContainerCreating, kubelet waits for the driver's response, and the logs show a timeout and repeated connection attempts.

The cause is usually not the application: Kubernetes detaches a volume from a node only after the driver has confirmed detachment, and if the previous attempt did not complete, the next operation is queued.

The cost of such a failure is not just one stuck replica. While Kubernetes CSI drivers build a queue of operations, the scheduler cannot place pods, autoscaling does not help, and a version update rolls back.

Incident analysis begins with the question "why was the volume not released" rather than "why did the service go down."

Then storage timeouts in the cluster come into play: as the number of volumes grows, operations hit backend limits, and each next one waits for a slot to free up.

A storage node failure turns this into an avalanche — volumes do not move to another node, an attempt to attach them to two nodes at once blocks startup, and recovery turns into hours of manual checks. Let's break down how this is addressed.

What managed storage provides: effect on timelines and failures

While a volume for a database or queue is provisioned manually, a new service waits: coordinate a block device, attach it to a node, write the mount point into the manifest, repeat the same in the test environment.

With a driver, a volume is created from a storage request — volume provisioning time drops to seconds, and the same manifest deploys on any node without separate operations or downtime.

Storage performance in Kubernetes stops being a lottery when classes are described explicitly: for transactional load — a block class with synchronous replication, for logs and builds — a network volume, for shared files — a shared file system.

Our engineers deploy storage on Ceph with replication and distribute requests across classes so that a heavy directory traversal does not compete with transaction log writes.

Storage fault tolerance is tested not in words but on node restart: the volume detaches, the pod comes up on another node, incomplete writes are handled according to policy.

Service recovery time is set in advance — by the number of replicas, pod eviction order, and limits on simultaneous migration, rather than by how quickly the on-call engineer reacts at night.

Effect in operational terms: preparing an environment for a new service — minutes instead of hours, the share of failures during planned node maintenance drops to isolated cases, recovery after a node failure is predictable and fits within a window the team defines itself.

Storage connection options: comparison of approaches

The choice begins not with the storage array model, but with how the application accesses data. Block storage presents a volume as a separate disk — predictable latency and high write speed, but the volume attaches to one node. Network file storage removes this limitation at the cost of network latency and file-level locks.

Volume access mode is not a hint but a contract: the driver checks it when attaching storage to the cluster, and if the application requests shared access while the driver supports only single access, the pod will remain waiting and will not start.

That is why before choosing, you look at the number of replicas, the presence of shared state, and requirements for failover when a node fails: this determines whether ReadWriteMany is needed or ReadWriteOncePod is sufficient.

Work sequence: from cluster audit to handover

The work sequence here is not bureaucracy but a way not to lose data. Implementing a storage driver into a live cluster runs into the fact that applications already use other storage classes, and some volumes are tied to nodes and zones. Below is how we run the project so that the switch happens without service downtime.

1. Audit of the cluster and current storage: inventory of drivers and storage classes, list of volumes and snapshots, service sensitivity to latency, backup state. Output — a dependency map and bottlenecks.
2. Scheme design: a set of storage classes for different load profiles, volume binding mode, placement across zones, encryption, quotas. We agree on the switchover plan with a rollback point.
3. Storage driver implementation: controller and node components, permissions, access secrets. On the test bench we verify volume creation, expansion, and deletion.
4. Volume migration in waves: first services without disks, then databases and queues. We copy via snapshots, the old and new classes work in parallel — the switch is reversible.
5. Fault tolerance testing: we take down a node, break connectivity to storage, measure how long it takes for the volume to reattach to another node and what happens when a zone is unavailable.
6. Observability and procedures: metrics and alerts for driver errors, quotas, and I/O latency, instructions for the on-call shift.
7. Handover: documentation on the scheme, configuration, migration and recovery procedures, walkthrough with your team.

Handover package: configurations, schemes, procedures

It makes sense to consider storage configuration work complete not when the pods are up, but when your team has a set of materials by which an on-call engineer can add a volume, change a storage class, or recover data after a failure without our help.

Below is what we hand over at the end of the project as a repository with versioning and proofread documents.

• Storage documentation — description of the selected drivers, supported access modes, and volume types, indicating which workloads are tied to which storage class and why.
• Volume placement scheme — a map: node, availability zone, storage class, workload binding; separately marked what can be moved without downtime and what requires a stop.
• Storage access policies — roles and permissions for service accounts, mount rules, read-write restrictions by namespace.
• Backup procedure — snapshot frequency, retention depth, recovery verification order, and the person responsible for reconciliation.
• Manifests and configuration parameters — working files of storage classes, volume requests, and snapshots in your repository, not in an engineer's local folder.
• Standard operations guide — adding a new node, adding a storage class, expanding a volume, putting a node into maintenance.
• Monitoring and alert map — which storage metrics are collected, at what thresholds a notification fires, and what the on-call engineer does next.
• Recovery procedure — a step-by-step scenario with verification on the test environment, including rollback to the previous volume state.

Such documents age along with the infrastructure, so the repository also contains rules for updating them — a configuration change and a description change go in a single change.

Case: storage migration without service downtime

The customer is a product team with a self-managed cluster: several dozen services, state in volumes via a CSI driver, an old NFS array at the core.

The array was hitting the ceiling on write operations: during peak hours write response grew to seconds, nightly backups no longer fit into the window, and space exhaustion was handled as an incident.

The difficulty was not the new storage but the already created volumes: recreating them from scratch would mean downtime.

We brought up a second storage class on Ceph alongside it, enabled volume snapshots, and migrated services in groups — a consistent snapshot, volume restore in the new class, pod startup, checksum reconciliation, and only then decommissioning the old one.

Rollback remained available at every step: until the service confirmed operation, the old volume was not deleted, and reverting amounted to starting pods on the previous storage class.

After several weeks of observation, the array was decommissioned — without a single downtime window.

What to do if Kubernetes already has storage deployed in the cluster

If the cluster is already running and storage is deployed, you cannot change everything "head-on": reinstalling the driver breaks volume bindings, and applications get unavailable disks.

The working scenario is different — the previous CSI driver continues serving existing volumes until the load is removed from them, while the new one connects alongside and provisions new ones.

We start with inventory: which StorageClasses are used, how many volumes are pinned to each workload, where shared access to a volume is needed, and where single access is sufficient.

We separately check snapshots and cloning on the target storage — if they are absent, some environments consciously remain on the previous driver, and that is a normal outcome.

Lossless data migration goes through a copy, not a live switch: we take a volume snapshot, create a new volume in the target class from it, and move the application to it.

For databases with replication, the order is reversed — volumes are migrated replica by replica, writes to the new one are enabled last, when the data has converged; checksum reconciliation before deleting the old volume is mandatory.

Then comes phased storage switchover: by namespaces or individual applications, from non-critical to critical. The previous class remains available but stops being the default class — new requests go to the target storage.

Rollback is cheap: we return the previous default class, and the volumes and driver remain in place.

We stop where the switchover closes a specific class of failures. If storage handles the load and the only reason is "there is a more modern driver," redoing it will add risk without gain.

Technical doubts: fault tolerance, copies, compatibility

Before starting, an engineer checks not whether the volume will come up, but how storage will behave in a failure.

That is why before the first CSI driver installation we examine three scenarios: availability zone loss, data integrity after a failure, and compatibility with what already runs in the cluster. Below are the answers that usually determine whether we start or not.

General principle: the driver does not create fault tolerance by itself — it passes to the cluster what the underlying storage can do. Replicas, snapshots, and zones are configured there, and are reflected in the cluster through the storage class and volume access modes.

How do you make sure data was not corrupted after a failure?

We check block volumes with a file system check, Ceph verifies replicas itself with a background check, and snapshots provide a recovery point.

We consider recovery working only after a run on the bench: we deploy state from a snapshot, start the application, and compare checksums. Otherwise the check remains on paper.

Is this compatible with the current cluster?

The driver is installed alongside the existing one, old volumes continue to be readable — migration goes through csi-migration.

We check the API version, access to node services, and permissions to mount host paths: in managed clusters, some privileges are closed, and this changes the installation scheme.

Can we roll back to the previous storage?

Yes, if the volume reclaim policy is Retain rather than Delete: volumes survive deletion of cluster objects.

The new driver runs in parallel, the default storage class is changed without haste, and rollback means returning the previous class, while old volumes remain in place and readable.

Let's discuss the task: we will select a storage mode for your workload

To come back with an estimate of the scope of work rather than general wording, we need source data on the cluster and workload. The more specific the inputs, the more accurate the implementation plan: stages, volume decommissioning order, checkpoints, and rollback option.

• Cluster scheme: how many nodes, where control plane components are located, how many availability zones are involved.
• Workload profile: databases, message queues, file shares, static artifacts — what writes to disk and with what intensity.
• Availability requirements: is a short I/O interruption acceptable when switching a volume to another node, is synchronous replication between zones needed.
• What is already used: Ceph, NFS, cloud block disks, local drives — and what exactly is unsatisfactory in the current option.
• Constraints: access regulations, volume encryption requirements, maintenance windows, prohibitions on external storage.
• Symptoms: growing write latency, stuck volumes during switchover, lack of space for images and backups.
• Who on your side grants access and approves changes in the production environment.

Based on this data, we will estimate the scope of work, select a storage mode for your workload, and come back with an implementation plan — with stages, responsible parties, and criteria by which you will accept the result.

Describe the task and attach what you already have: schemes, metric exports, error logs. We will clarify the rest ourselves.