Configuring Proxmox Ceph Storage for Distributed Storage

Let's be clear: when virtual machines on Proxmox experience I/O bottlenecks due to disk operations, and Ceph logs show persistent degraded PGs, it's not Ceph's fault—it's a configuration deficiency. Many administrators attempt to deploy distributed storage on three servers with a single gigabit link and inexpensive HDDs, only to witness cluster failure upon disk failure. Our team has analyzed dozens of such scenarios over 7 years of deploying Ceph in production. This guide covers Proxmox Ceph configuration for high-performance environments. We offer turnkey services from audit to monitoring. Contact us for a project assessment. Warranty on work—12 months, certified engineers.

Official Ceph documentation: for production clusters, a 25 Gbit/s or higher network is recommended.

Planning and Preparation

Ceph requires two networks: public (client traffic from VMs to Ceph) and cluster (replication between OSDs). Combining them on one interface leads to contention between VM IO and replication traffic under load, degrading performance.

Minimum production configuration:

• 3 Proxmox nodes, each with a 10G NIC for public network and a separate 10G or 25G for cluster network
• Dedicated NVMe/SSD for OSD—system disks are unsuitable. NVMe delivers up to 500,000 IOPS per OSD.
• Separate SSD for Ceph WAL/DB (Write-Ahead Log and metadata) if using Bluestore with BlueStore DB

Without a dedicated WAL/DB device, Ceph places WAL directly on the OSD. It works, but slower. For SATA SSDs, placing WAL on an NVMe accelerates writes by 2-3 times. Bluestore is 3 times faster than the outdated Filestore on random writes and uses less space.

Installation and Configuration

Installing Ceph via Proxmox GUI

Datacenter → Storage → don't go there yet. First: Proxmox host → Ceph → Install Ceph.

Choose the version (Quincy or Reef — current stable), specify public and cluster networks. After installation on all nodes, create MON and MGR. MONs must be on an odd number of nodes (3 or 5) for quorum. On a 3-node cluster, quorum = 2 MONs. If one node fails, the cluster continues to work.

Creating OSDs

# Create OSD with dedicated WAL/DB
pveceph osd create /dev/nvme1n1 --wal-dev /dev/nvme0n1 --db-dev /dev/nvme0n1

# Without dedicated device
pveceph osd create /dev/sdb

When creating via GUI, select the disk and optionally the WAL/DB device. OSDs are automatically formatted as Bluestore. Bluestore is mandatory. Filestore is deprecated and not supported in new versions.

Pools and Replication Configuration

After creating OSDs, create a pool for RBD (VM block disks):

pveceph pool create vm-data --pg_autoscale_mode on --size 3 --min_size 2 

size=3 — three replicas ensures high availability and data replication across nodes. min_size=2 — minimum replicas for write. On three nodes, this is the most reasonable configuration. pg_autoscale_mode=on — Ceph auto-manages PG count.

Assign the pool as a Proxmox datastore. Via GUI: Datacenter → Storage → Add → RBD. Or via CLI:

pvesm add rbd vm-ceph --pool vm-data --monhost '192.168.20.1,192.168.20.2,192.168.20.3' --keyring /etc/pve/priv/ceph.client.admin.keyring 

CephFS for ISOs and Templates

RBD is for VM disks. CephFS is for ISO images, container templates, shared storage between nodes. Create MDS (MetaData Server)—required for CephFS. On a second node, set up a standby MDS. Create the filesystem:

pveceph mds create
pveceph fs create --name cephfs --pg_num 64
pvesm add cephfs iso-cephfs --path /mnt/pve/iso-cephfs --content iso,vztmpl

Performance Tuning

To avoid performance degradation, careful tuning is required. Common issues include network latency jitter and clock skew. Ensure inter-node delay < 0.5 ms, ideally < 0.1 ms. Clock skew must be < 500 ms using chrony or ntpd. For databases, disable RBD snapshots at the Ceph level and use VM agent snapshots to avoid copy-on-write overhead. Use CRUSH map customization to separate OSDs by failure domain. Proxmox Ceph tuning also involves adjusting OSD settings like osd_op_num_threads_per_shard.

Avoiding Performance Degradation

Problems often arise when using snapshots. For production databases, disable snapshots at the Ceph level and take them via an agent inside the VM. Also monitor network latency—inter-node delay should not exceed 0.5 ms, optimally under 0.1 ms. Clock skew must be no more than 500 ms, otherwise MONs lose quorum. Ceph high availability is ensured by proper replication and monitoring.

Monitoring and Troubleshooting

# Overall status, OSDs, performance, problematic PGs
ceph status
ceph health detail
ceph osd tree
ceph osd df
ceph iostat 2
ceph pg ls-by-state degraded
ceph pg deep-scrub <pgid>
ceph osd pool scrub vm-data

PG states to watch: degraded, undersized, inconsistent. Run scrub to restore consistency.

Prometheus metrics are available via the Ceph MGR module (enable with ceph mgr module enable prometheus, metrics on port 9283). In Proxmox, use Grafana dashboards with IDs 2842 and 5336. This provides comprehensive ceph monitoring prometheus integration.

Typical Issues

Cost and Services

Configuration Comparison

How to Configure Ceph in 5 Steps

1. Install Ceph on all nodes via Proxmox GUI.
2. Create MON and MGR on all nodes.
3. Add OSDs with dedicated WAL/DB on NVMe for maximum performance.
4. Create an RBD pool with auto-scaling PGs and replication 3.
5. Connect the pool to Proxmox as a datastore and set up monitoring.

What's Included

• Audit of current infrastructure and network
• Design of Ceph cluster (network scheme, disk selection)
• Installation and configuration of Proxmox VE and Ceph
• Creation of RBD and CephFS pools
• Monitoring setup (Prometheus + Grafana)
• Documentation and staff training
• 12-month warranty support

Timeline and Cost

Cost is calculated individually after an infrastructure audit. Our engineers have 10+ years of experience and have completed over 50 projects. Contact us for a preliminary assessment. Migrating to Ceph can reduce storage costs by up to 40% compared to traditional SAN solutions.