NFS server in Kubernetes: where storage breaks
A typical scheme looks like this: a single NFS server in Kubernetes is set up "quickly" — a single directory export, access by node address, mounting into a shared folder for containers.
In a test environment this works, but in production the first thing to appear is a stuck volume mount: a process on the node does not receive a response and cannot complete the operation.
A stuck volume mount is not a local problem of one application. The pod gets stuck waiting, the node cannot terminate it correctly and becomes unavailable — along with it, neighboring services that never accessed storage go down. At the same time, the logs look fine: the server responds, there are no obvious errors.
The second landmine is data loss on pod restart. If a write goes to a shared volume without lock coordination, a restart interrupts it midway, and after coming up the application reads half a file as if it were whole. Several replicas writing to one directory add name races and overwriting of each other's results.
Under peak load, the storage bottleneck manifests as a growing request queue: the server spends time on metadata operations, the application waits for a response and holds connections. Adding replicas in such a scheme does not help — they hit the same single node.
All this is solved not by a "correct server" but by separating roles: mount modes, access levels, limits, and observability for the volume. Let's look at how this is addressed.
What managed file storage gives a cluster
Order in storage is visible by how quickly a volume is provisioned.
With dynamic volume provisioning and a declaratively described storage class, the volume appears at the moment the pod is scheduled, and the container starts with an already mounted directory — without manual export on the server and waiting for an on-duty administrator.
The delay between "volume needed" and "volume ready" stops depending on a person.
The second effect is a predictable storage failure class. A single NFS server without replication stops all pods that hold data on it: application downtime equals server recovery time.
Managed file storage for Kubernetes separates node failure from service failure — the client reconnects to an available endpoint, the pod moves and receives the same volume at the same path, and the application does not rebuild state from scratch.
The third is write performance under load. The bottleneck shows up on metadata earlier than on capacity: many small files, frequent log rotation, simultaneous writes from dozens of pods.
The access mode and mount parameters are selected for the profile — shared write for shared directories, exclusive for databases and queues, separate volumes for logs so that heavy writes do not slow down reads.
Order is measured concretely: seconds to provision a volume, the number of operations left manual, the share of node failures absorbed without application downtime, and write smoothness at peak. Let's look at how this is configured for your cluster.
Own NFS nodes versus cloud file service
NFS in Kubernetes is connected as a PersistentVolume through a StorageClass and driver, and the access mode determines how many nodes can mount the volume simultaneously: ReadWriteMany for shared directories, ReadWriteOnce for a single consumer.
An error in choosing it manifests not at deploy time but at scaling: the new replica hangs waiting for the volume.
| Variant | When it fits | Limitations |
|---|---|---|
| Own NFS node | Need control over exports, permissions, and snapshots | Single point of failure: the machine goes down — all pods with the shared volume go down |
| Provider's managed service, AWS EFS | Multiple availability zones, don't want to maintain storage yourself | Higher latency on small operations, fewer settings, provider dependency |
| Local node disks | Databases, caches, sensitivity to I/O speed | ReadWriteOnce only, the volume is tied to the node and does not move with the pod |
| Block volume with a driver | One replica per write, fast rollback via snapshots | No shared access: a second pod will not mount the same volume |
An own node gives full control over exports and snapshots, but operation, updates, and backups remain on your side. A managed service removes this work but limits settings and adds latency on a multitude of small file operations.
A practical rule: shared data — on a shared volume, databases and caches — on local or block volumes. We check the access mode before rollout — on one node ReadWriteOnce works, but on three replicas the same manifest will no longer come up.
How we implement file storage: stages of work
We structure the work so that each next step relies on confirmed data from the previous one.
First, a storage audit: we look at which volumes are already mounted in the cluster, which volume claims remain from deleted applications, who writes to the shared directory and with which access modes.
At this stage, two or three silent bottlenecks usually surface — an overloaded export and an application holding a mount in hard mount mode.
- Storage audit. Inventory of volumes, access profiles, mount points, and permissions; measurement of read and write latency under real application load.
- Scheme design. Choice between a shared export and separate ones per application, division by storage classes, calculation of capacity and throughput headroom.
- Server-side preparation. Deployment of the service, exports restricted by subnet, synchronous write mode for critical directories, per-user quotas.
- Cluster connection. Storage driver, storage classes and volume reclaim policies so that a deleted volume returns space instead of remaining stuck.
- Fault tolerance testing. We shut down a node with an application, break connectivity to storage, and see how pods behave: migration to another node, remounting, speed of return to work.
- Load runs. We look at the level of parallel writes at which queues begin, and select mount parameters for the application profile.
- Handover to operation. Metrics for latency and volume usage, failure alerts, scaling regulations, and a brief instruction for the on-duty shift.
The audit takes more time than configuration, and we do not hide this: redoing the scheme after launch costs more than two days of inventory before it.
As a result, the team is left with a working environment, a reproducible configuration, and a clear procedure for action when a node or the storage itself fails.
Configurations, schemes, and regulations: what you get
At the output you get not a set of scattered manifests, but a kit by which storage is deployed, tested, and handed over to the on-duty shift.
Everything lies in the repository next to the applications: changes go through review, and the state of storage classes and volumes is reproduced without manual edits on nodes.
- Volume configurations — StorageClass and PVC descriptions for different access profiles: ReadWriteMany for shared data, ReadWriteOnce for loaded databases; mount parameters, the directory inside the volume, and the volume limit.
- Storage access policies — export rules on the server side, separation by subnets and namespaces, separate read and write permissions, anonymous access closed.
- Placement scheme — which NFS node serves which directories, in which zone it lives, how the volume gets into the pod, where the failure points are and how they are separated from each other.
- Backup regulations — what is backed up, at what frequency, how long copies are stored, who starts restoration and how it is verified that the copy is actually readable.
- Infrastructure documentation — procedure for connecting a new volume, actions when the storage server is unavailable, metric map, shift handover checklist.
- Observability — metrics for operation latency, used capacity, and errors on clients, trigger thresholds, and alert routes.
- Migration and rollback plan — how to move data from the current storage without downtime and how to return to the previous scheme if application behavior diverges from expectations.
Case: migrated storage without service downtime
Before the migration, all namespaces in the Kubernetes cluster worked with a single NFS mount point. Volumes were issued manually: a request, editing the export list, waiting for an engineer — from several hours to a day.
Failure of the single server meant stuck volumes: pods did not restart, and the request queue accumulated while the on-duty engineer dealt with the configuration.
What we changed
We assembled two storage nodes with export replication and automatic failover, described storage classes declaratively — the volume is created and mounted without human involvement.
We added namespace quotas, I/O latency monitoring, and a node replacement regulation tested on a training failure.
The migration proceeded in parallel: the new environment was brought up next to the old one and data was transferred with checksum verification.
Services were switched one by one — the mount point changed in the manifest, the pod was recreated, and the previous volume remained available for rollback. The old environment was kept in reserve until the load had completed a full cycle on the new storage.
Result in numbers
Volume provisioning decreased from 4-6 hours of manual work to 30 seconds automatic. There were no data access failures over six months of observation — before the migration they accumulated at 2-3 per month.
Manual export operations decreased by approximately 90%, leaving only quota expansion and scheduled updates. Recovery time after node failure — from 40 minutes of manual troubleshooting to automatic failover in a minute.
Dynamic volumes or static connection: what to choose
In Kubernetes, a volume can be issued in two ways, and the difference is not in convenience but in the control point.
In dynamic mode, the claim references a storage class, and the component responsible for provisioning itself creates a volume of the required size, binds it to the claim, and mounts it into the pod.
In static connection, the export is prepared in advance, and a volume description with an exact address and parameters enters the cluster — the application finds it by label.
The choice depends on who owns the data and how long it lives.
| Variant | When it fits | Limitations |
|---|---|---|
| Dynamic volumes | Services with variable load, queues, caches, many similar environments | Need a working storage class and permissions to create volumes; volume life is governed by the reclaim policy |
| Static connection | Shared export for a group of services, data that survives cluster recreation, external arrays | Size and address are fixed; expansion and rotation are manual |
The key detail is volume lifecycle management. The storage class sets the reclaim policy: when a claim is deleted, the volume is either released together with the data or retained and requires manual connection. For databases and queues, this decides whether state survives environment recreation.
In practice, both modes coexist: dynamic for stateless services and temporary environments, static for data that must survive the cluster. We arrange them across storage classes so that no volume remains ownerless.
Technical questions before implementing storage
Before implementation, the engineering team has a short list of doubts on which the fate of the project depends: volume compatibility, simultaneous writes, access permissions, and behavior during failure. Let's go through the ones that come up most often during approval.
Are the volumes compatible with current manifests?
We test in the test environment the same manifest that will go to production: the pod receives the directory and sees a regular file system.
The difference from a block volume is one thing — there is no formatting, so migration and rollback are done by copying the directory, not by restoring a state snapshot.
Is it safe for two pods to write to one directory?
ReadWriteMany mode allows mounting on multiple nodes, but consistency is ensured by the application, not the storage. For shared uploads and artifacts this is normal; where write order matters, we leave a single writer or separate data into subdirectories with locks.
How are access permissions issued?
Through user IDs inside the pod and the root permission mapping mode on the server. If they are not configured, the container writes from an unprivileged account and is denied. We set the directory owner and group in the template once — new services inherit the working scheme.
What happens if a node or storage server fails?
If a node fails, the volume detaches and moves with the pod — it is important that the mount point is available from the new node. If the server itself is unavailable, pods go into I/O wait and are not deleted gracefully, so we set timeouts and the reclaim mode in advance.
Let's discuss the task: what to prepare for the conversation
The work plan is made up of several inputs: how access to volumes is structured now, what the load profile is, and what environment limitations exist. The more specific the answers, the fewer clarifying rounds — the estimate and plan come in one pass, not after three calls. There is no need to prepare a full technical specification in advance.
What is useful to have on hand:
- Kubernetes cluster version, installation method, number of nodes and availability zones;
- current storage scheme — NFS server, cloud file storage, or local volumes — and what is unsatisfactory about it;
- load profile: volume sizes, number of simultaneous readers and writers, whether a shared volume for several pods is needed;
- sensitivity to latency and throughput in specific scenarios;
- fault tolerance requirements: what must survive a node or zone failure, acceptable downtime and volume of loss;
- environment limitations: network policies, private subnets, audit requirements, infrastructure access;
- timelines and the window in which production may be changed.
With this set, we return with an estimate and work plan: stages, risks, and the procedure for connecting storage without stopping services. If some data is still unknown, we will go through it in the conversation and supplement along the way.