Storage Design and Administration Services

Why does a Ceph cluster start to slow down?

A Ceph cluster with 15 OSDs shows health: HEALTH_WARN at night. ceph -s returns too many PGs per OSD (400 > max 250). RBD I/O latency doubles — a classic symptom of designing storage without PG calculation. This is not a disaster, but over several years of practice (more than 50 deployed Ceph clusters) we have learned to design so that it does not occur.

One common mistake in storage design is incorrect estimation of placement groups. Our engineers select pg_num for the current topology and enable autoscaling in Ceph Nautilus+, guaranteeing SLA 99.9% on data availability. Experience shows: proper PG calculation reduces peak I/O latency by 40–60% compared to default configuration.

Ceph: architecture and pitfalls

Ceph is a software-defined storage with the CRUSH algorithm that distributes objects without a central catalog. This gives horizontal scalability, but requires accurate PG calculation.

CRUSH Map and data distribution. Each object is hashed into a PG, and the PG is mapped to OSDs via CRUSH. The failure domain is configured so that replicas do not land on the same host, rack, or datacenter.

# View CRUSH map and calculate pg_num ceph osd getcrushmap -o /tmp/crushmap.bin crushtool -d /tmp/crushmap.bin -o /tmp/crushmap.txt ceph osd pool get <pool_name> pg_num # Recommendation: (OSD_count * 100) / replica_count → round to power of 2 # Enable autoscaling ceph osd pool autoscale-status 

For 12 OSDs with 3x replication: (12 * 100) / 3 = 400 → 512. If there are five pools, the total PG per OSD increases sharply, and too many PGs per OSD appears. Enable ceph osd pool autoscale-status — this solves the problem.

RBD performance. RBD (RADOS Block Device) is the basis for VMs (Proxmox, OpenStack). Critical parameters:

• rbd_cache = true with rbd_cache_size = 128 MB — write-back cache on the client (speeds up random writes, but risk of data loss on crash). For databases use rbd_cache_writethrough = true or disable cache.
• ms_async_op_threads — for NVMe increase to 8–16 instead of default 3.

# Pool benchmark rados bench -p <pool_name> 60 write --no-cleanup rados bench -p <pool_name> 60 seq rados bench -p <pool_name> 60 rand 

Typical Ceph cluster problems:

• slow ops — operations wait >30 s. ceph daemon osd.<id> dump_ops_in_flight will show the cause: overloaded OSD, network latency (check MTU, jumbo frames), low bluestore_throttle_bytes.
• Rebalancing after adding OSD slows down the entire cluster. We adjust:

ceph osd set-backfillfull-ratio 0.90 ceph config set osd osd_max_backfills 1 ceph config set osd osd_recovery_max_active 3 

Reducing osd_max_backfills to 1 reduces load but increases recovery time — we find a balance for the specific cluster.

Monitoring. ceph mgr module enable prometheus enables an exporter on port 9283. Metrics: ceph_health_status, ceph_osd_stat_bytes_used, ceph_pool_rd_bytes. In Grafana use dashboard ID 2842.

PG calculation: how to avoid OSD overload

When designing storage, we always calculate PG so that each OSD has no more than 250 PG. If there are many pools, we use autoscaling (pg_autoscale_mode = on). According to the Ceph documentation (Placement Groups), exceeding 250 PG per OSD increases latency by up to 50%. For a hybrid cluster (CephFS + RBD) with 12 OSDs, 3 replicas, and 4 pools, the formula (12 × 100) / 3 = 400 → 512 gives 512 PG per pool. Total PG per OSD = (512 × 4) / 12 ≈ 171, well below the 250 limit. Autoscaling is enabled.

NFS: when it just works and when it stops

NFS v4.1 with pNFS for parallel access is overkill for most tasks. For shared storage of web servers or home directories, NFS v4 without pNFS is sufficient.

/etc/exports:

/data/shared 10.0.0.0/24(rw,sync,no_subtree_check,no_root_squash) 

• sync — confirmation of write after flush to disk (production).
• no_root_squash — necessary for Kubernetes Persistent Volumes with NFS provisioner.

Client mount options:

10.0.0.1:/data/shared /mnt/shared nfs rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,nfsvers=4.1 

rsize/wsize 1 MB are optimal for large files. hard — the client waits for the server indefinitely (unlike soft).

S3-compatible systems: MinIO and RadosGW

MinIO in single-node — for dev/staging. Distributed mode with erasure coding — from 4 nodes, resilient to loss of half the disks. For Ceph RadosGW data is stored in RADOS, scales horizontally:

radosgw-admin user create --uid=s3user --display-name="S3 User" --access-key=KEY --secret=SECRET radosgw-admin quota set --quota-scope=user --uid=s3user --max-size=100G 

MinIO setup: mc alias set myminio http://localhost:9000 minioadmin minioadmin && mc mb myminio/backups.

Why RAID 6 is preferable to RAID 5 for disks ≥16 TB?

Software RAID via mdadm lives on bare metal. During rebuild of RAID 5 with modern disks (16–20 TB), recovery takes 20–40 hours. The probability of a second disk failure during this time is 3–5 times higher than with a 4-hour rebuild of old disks. RAID 6 tolerates simultaneous failure of two disks — safety is 2 times higher in terms of data preservation.

# Create RAID6 from 6 disks mdadm --create /dev/md0 --level=6 --raid-devices=6 /dev/sd[bcdefg] # Monitor cat /proc/mdstat mdadm --detail /dev/md0 

Be sure to configure MAILADDR in /etc/mdadm/mdadm.conf for alerts.

How to choose a storage type?

The choice between Ceph, NFS, and MinIO is dictated by the workload profile. For block access (VMs, databases) — Ceph RBD. For file access — NFS v4 or CephFS. For objects — MinIO or RadosGW. Hybrid schemes (e.g., CephFS + RBD) require careful planning of CRUSH roots.

What is included in the storage design work?

• Architecture documentation (CRUSH map, network topology, PG calculation).
• Configuration files and automation scripts (Ansible, Terraform).
• Integration with monitoring (Prometheus, Grafana, dashboards).
• Runbook for administration and disaster recovery.
• Team training (2–3 sessions of 4 hours).

How we design fault-tolerant storage

1. Requirements analysis — workload profiles, RPO/RTO, budget.
2. Architecture selection — Ceph, NFS, MinIO or hybrid.
3. Design — CRUSH map, PG, network, monitoring.
4. Implementation — deployment, configuration, performance tests.
5. Go-live — runbook, team training, SLA 99.9%.

Cost is calculated individually after analysis of storage requirements. Order an audit of your current infrastructure — we will propose a solution with TCO 30–50% lower than commercial SAN/NAS. Contact us for a consultation; we will evaluate your storage and provide a detailed plan within one week. For a custom storage architecture tailored to your workload, schedule a consultation with our engineers.