Ceph Cluster Setup (Standalone) for Distributed Storage

Users complain about slow writes to S3, and virtual machines periodically freeze during I/O. The reason is an improperly configured Ceph: OSD on HDD without NVMe cache, CRUSH without rack separation, 'cluster network' not set. A successful Ceph cluster setup involves hardware planning, deployment, and tuning. According to Wikipedia, Ceph is a distributed storage system that combines block (RBD), file (CephFS), and object (RGW/S3) storage in one cluster. One cluster covers the needs of virtual machines, Kubernetes, backups, and applications working with S3. The price of this universality is complexity: a misconfigured Ceph runs slowly and loses data in unusual ways during failures. Our experience (10+ years in production, 50+ deployed Ceph clusters) guarantees a fault-tolerant architecture. Savings on commercial storage licenses can reach $15,000 per year, and a properly configured Ceph works 3-5 times faster than a standard configuration. For example, a 5-node cluster with NVMe costs about $12,000. Typical cluster deployment costs range from $8,000 to $15,000 depending on complexity.

How to set up a Ceph cluster?

Setting up a Ceph cluster includes hardware planning, cephadm deployment, CRUSH topology configuration, Bluestore tuning, and monitoring integration. Each stage affects performance and fault tolerance. An average three-node cluster with monitoring is deployed in 5-7 days. The cost is determined individually after an audit of your infrastructure.

Advantages of Standalone Ceph

Standalone means Ceph without a third-party orchestrator (without Proxmox, without Rook for Kubernetes, without OpenStack). It is managed via cephadm — the official orchestrator that deploys services in containers. Everything regarding topology, disks, and network is decided by us. This gives maximum control and requires maximum understanding of the architecture.

Solving Common Ceph Issues

A misconfigured Ceph is a source of chronic headaches: slow writes (latency up to 50 ms on RBD), data loss when a single node fails, uncontrollable backfill during peak hours (network load up to 10 Gbit/s). We solve these problems through:

• Precise hardware planning with network and disk separation;
• CRUSH topology that isolates failures at the host or rack level;
• Fine tuning of Bluestore and recovery parameters.

Ceph with correct Bluestore tuning shows 70% lower latency than the standard configuration, and write speed increases 3-5 times. This is confirmed by our case studies: after tuning, RBD write speed increased 3-5 times.

How to choose the right disk topology?

Minimum for production: 3 nodes. With 3 nodes, we lose one without data loss at size=3, min_size=2. With 5 nodes, we tolerate two simultaneous failures at size=3 with CRUSH rules on different nodes. Disks: do not mix OSD of different classes (NVMe and HDD) in the same pool without explicit separation via device classes. Ceph balances data evenly by OSD weight, and a slow HDD will determine the speed of the entire pool.

Recommended configuration for 3 nodes:

• System disk: SATA SSD 240GB;
• 2-4 x NVMe for OSD (capacity);
• Dedicated NVMe for WAL/DB (bluestore_block_wal, bluestore_block_db);
• 2 x 25GbE: public network + cluster network.

Separating public and cluster networks is not optional. Replication between OSDs generates traffic up to 25 Gbit/s, which when mixed with client traffic degrades latency by 30-50%.

Disk configuration comparison — Ceph cluster setup

Deployment via cephadm

1. Install cephadm on the bootstrap node using curl.
2. Run bootstrap specifying MON IP and cluster network.
3. Copy the SSH key to the remaining nodes.
4. Add hosts to the cluster.
5. Apply the ceph-init.yaml specification.

curl -s --remote-name https://download.ceph.com/rpm-latest/el9/noarch/cephadm && chmod +x cephadm && ./cephadm install cephadm bootstrap --mon-ip 192.168.10.1 --cluster-network 192.168.20.0/24 --apply-spec /tmp/ceph-init.yaml
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node2
ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node3
ceph orch host add ceph-node2 192.168.10.2
ceph orch host add ceph-node3 192.168.10.3
ceph orch apply -i /tmp/ceph-init.yaml

File /tmp/ceph-init.yaml:

service_type: host
hostname: ceph-node1
addr: 192.168.10.1
---
service_type: host
hostname: ceph-node2
addr: 192.168.10.2
---
service_type: host
hostname: ceph-node3
addr: 192.168.10.3
---
service_type: mon
placement:
  hosts:
    - ceph-node1
    - ceph-node2
    - ceph-node3
---
service_type: mgr
placement:
  count: 2
---
service_type: osd
service_id: default_drive_group
placement:
  hosts:
    - ceph-node1
    - ceph-node2
    - ceph-node3
data_devices:
  paths:
    - /dev/nvme1n1
    - /dev/nvme2n1
db_devices:
  paths:
    - /dev/nvme0n1

CRUSH Map Configuration for Fault Tolerance

CRUSH (Controlled Replication Under Scalable Hashing) determines how Ceph distributes data. The default rule is replicated_rule with a host level. This means replicas of the same data will never be on the same host. For more complex topologies (racks, datacenters), create a rule with rack-level replication. Example: 3 racks, one node per rack — losing a rack is not a problem.

ceph osd crush add-bucket rack01 rack
ceph osd crush add-bucket rack02 rack
ceph osd crush add-bucket rack03 rack
ceph osd crush move rack01 root=default
ceph osd crush move rack02 root=default
ceph osd crush move rack03 root=default
ceph osd crush move ceph-node1 rack=rack01
ceph osd crush move ceph-node2 rack=rack02
ceph osd crush move ceph-node3 rack=rack03
ceph osd crush rule create-replicated rack_replicated default rack
# Create pools
ceph osd pool create vm-data 0 0 replicated rack_replicated
ceph osd pool set vm-data size 3
ceph osd pool set vm-data min_size 2
ceph osd pool application enable vm-data rbd
rbd pool init vm-data
ceph osd pool create .rgw.root 32 32
ceph osd pool create default.rgw.log 32 32
ceph osd pool create default.rgw.control 32 32
ceph osd pool create default.rgw.meta 32 32
ceph osd pool application enable .rgw.root rgw
ceph mgr module enable pg_autoscaler
ceph osd pool set vm-data pg_autoscale_mode on

Bluestore parameters in /etc/ceph/ceph.conf:

[global]
osd_pool_default_size = 3
osd_pool_default_min_size = 2
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[osd]
bluestore_block_db_size = 107374182400 # 100GB
bluestore_block_wal_size = 10737418240 # 10GB
bluestore_cache_size_hdd = 1073741824 # 1GB
bluestore_cache_size_ssd = 3221225472 # 3GB
osd_op_num_threads_per_shard = 2
osd_op_num_shards = 8
osd_recovery_max_active_hdd = 3
osd_recovery_max_active_ssd = 10
osd_max_backfills = 1

[mon]
mon_osd_down_out_interval = 300 # 5 min before out

Monitoring and recovery

For monitoring, use the built-in Prometheus module via MGR:

ceph mgr module enable prometheus 

When a node fails, mark the down OSDs as out (ceph osd out osd.X), wait for backfill to finish (state active+clean), then remove them (ceph osd purge osd.X --yes-i-really-mean-it).

Deliverables and What's Included

• Audit of current infrastructure and configuration selection;
• Cluster deployment (3+ nodes) with cephadm;
• CRUSH topology for your fault tolerance scheme;
• Bluestore and recovery parameter tuning;
• Integration with Prometheus/Grafana;
• Documentation and runbook for maintenance;
• Team training on basic operations.

Contact us for a consultation — we will evaluate the project and offer an optimal solution. Get a preliminary estimate of timelines and costs based on your hardware configuration.

Timelines and cost

Cost is calculated individually after an audit of your infrastructure. Typical deployment costs range from $8,000 to $15,000 depending on complexity. A standalone Ceph cluster setup requires careful planning. Request a consultation — we will show how effective Ceph can be when properly configured.