Turnkey CephFS Deployment for Distributed File Systems

Turnkey CephFS Deployment for Distributed File Systems

Imagine a hundred servers simultaneously writing and reading files from a single storage system with instant data consistency. RBD won't work here: it provides a block device to only one client. You need a POSIX file system on top of Ceph, and it exists — CephFS. This article shares our turnkey deployment experience: from MDS configuration to Kubernetes integration and monitoring. With 5+ years in production, we've completed 30+ projects, and 70% of clients choose CephFS for shared storage. We reference the official Ceph documentation (CephFS documentation) for foundational concepts.

Typical scenarios: shared storage for Kubernetes (ReadWriteMany PVC), common storage for CI/CD runner caches, media assets for multiple application servers, home directories for hundreds of hosts. Each scenario has its own requirements for quotas, performance, and fault tolerance.

How CephFS Works and Why Consistency Matters

CephFS requires MDS (MetaData Server) — a service that manages metadata (directories, permissions, inodes). File data lives in RADOS; only metadata passes through MDS. As stated in the official CephFS documentation, MDS manages metadata. An active MDS handles requests, while standbys wait for failure. For production, a minimum of 1 active + 1 standby is needed.

# Deploy MDS via cephadm ceph orch apply mds cephfs --placement="3 ceph-node1 ceph-node2 ceph-node3" 

ceph fs volume create cephfs creates pools for data and metadata and starts MDS in one command. But a crucial detail: the metadata pool must be on the fastest disks. MDS caches metadata in RAM, but on a cache miss it queries the pool. A slow metadata pool means a slow ls in directories with thousands of files. Use a device class to place on NVMe: ceph osd pool set cephfs_metadata crush_rule nvme_replicated.

Practical Deployment: From Manual Pool Creation to Automation

For full control over pool parameters, we create them manually with these steps: Step 1: Create Metadata Pool — small, performance-critical.

ceph osd pool create cephfs_metadata 32 32
ceph osd pool set cephfs_metadata size 3
ceph osd pool set cephfs_metadata pg_autoscale_mode on

Step 2: Create Data Pool — main volume.

ceph osd pool create cephfs_data 0 0 replicated
ceph osd pool set cephfs_data size 3
ceph osd pool set cephfs_data pg_autoscale_mode on

Step 3: Create the File System.

ceph fs new cephfs cephfs_metadata cephfs_data 

Step 4: Mount on Clients. For automation, we use Ansible or Terraform. The kernel client (faster): specify MON addresses and secret in /etc/fstab. For controlled access, create a minimally privileged user:

ceph auth get-or-create client.cephfs-ro mon 'allow r' mds 'allow r' osd 'allow r pool=cephfs_data' -o /etc/ceph/ceph.client.cephfs-ro.keyring 

What's Included in Our Turnkey Service

Our turnkey CephFS deployment includes:

• Audit of current infrastructure and cluster design
• MDS deployment with multi-active if needed
• Configuration of data and metadata pools with device class awareness
• Integration with Kubernetes via ceph-csi-cephfs (ReadWriteMany PVC)
• Creation of subvolumes and quotas for projects
• MDS performance tuning and metadata pool optimization
• Monitoring via Prometheus/Grafana with ready dashboards and alerts
• Clear documentation and knowledge transfer sessions
• Technical support during initial production

Integrating CephFS with Kubernetes: ReadWriteMany PVC via ceph-csi-cephfs

For ReadWriteMany PVCs, we use ceph-csi-cephfs. A Helm chart creates a StorageClass, secret, and CSI driver. PVCs with accessModes: ReadWriteMany are impossible with RBD:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-cache
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ceph-cephfs
  resources:
    requests:
      storage: 100Gi

The CSI driver creates a subvolume — isolated space with a quota.

Subvolumes and Quotas: Project Isolation and Resource Control

CephFS supports directory-level quotas. For isolated projects, subvolumes are convenient:

# Create a subvolumegroup
ceph fs subvolumegroup create cephfs kubernetes
# Subvolume with a quota
ceph fs subvolume create cephfs pvc-build-cache kubernetes --size 50GiB

Quotas work via the attributes ceph.quota.max_bytes and ceph.quota.max_files. For Kubernetes PVCs, quotas are set in the StorageClass via parameters.

Performance and MDS Tuning: Avoiding Bottlenecks

MDS cache configuration is critical with large numbers of files. A typical config for small-file workloads:

[mds]
mds_cache_memory_limit = 8589934592 # 8 GB
mds_log_max_segments = 128
mds_tick_interval = 5
mds_dir_max_commit_size = 10485760 # 10 MB
mds_session_timeout = 60

Multi-active MDS (2 or more) splits the namespace by directory — helps with hot spots but doesn't scale linearly. Monitor latency: ceph_mds_request_latency_sum — if >100 ms, expand MDS or optimize the metadata pool. In our internal benchmarks, the kernel client outperforms ceph-fuse by up to 3x for sequential reads. Compared to NFS, CephFS delivers up to 5x higher throughput for concurrent writes in a multi-client scenario.

CephFS Monitoring: Key Metrics and Dashboards

For production operation, we use Prometheus + Grafana. Key metrics:

We configure alerts in Alertmanager and create dashboards with SLO/SLI.

Real-World Case: Media Company Deployment

In one project for a media company (our client), we deployed CephFS across 50 servers. Initially, metadata latency was 150ms due to an HDD-based metadata pool. After migrating to NVMe and tuning MDS cache, latency dropped to 30ms. The client now runs over 1000 concurrent CI/CD jobs without bottlenecks. The cluster now handles over 10,000 metadata operations per second with sub-30ms latency. This project demonstrated the impact of proper metadata placement and MDS sizing. This case is from our own practice, demonstrating our hands-on expertise.

Typical Problems and Their Solutions (Postmortem)

• df -h shows incorrect free space? CephFS reports total cluster capacity. Use ceph.dir.rbytes attributes with quotas for accurate per-directory usage.
• NFS client doesn't see changes? Disable directory caching with dcache_timeout=0 at mount time, but expect a 15–20% performance drop.
• MDS not switching to standby? Check ceph health detail — often the standby cannot reach the MON. Verify network access and rank configuration.

Estimated Timelines and Cost

Cost is calculated individually after an audit. Typical project costs range from $5,000 to $20,000, depending on the scope and complexity. Contact us for a consultation and project estimate. Order a turnkey CephFS deployment for your infrastructure. Our engineers with 5+ years of experience will help you implement a fault-tolerant storage system with a guarantee.