Setting Up Ceph RBD for Virtual Machines

Setting Up Ceph RBD (Block Device) for Virtual Machines

Imagine: you urgently need to migrate a virtual machine to another server due to a hardware issue. If the disk is local, you wait for a qcow2 copy over the network. In our practice, we constantly face this pain and offer a solution—Ceph RBD.

Ceph RBD (RADOS Block Device) are block disks stored not on the local hardware of the virtualization host, but in a Ceph cluster. A virtual machine on KVM/QEMU works with an RBD disk exactly as with a regular qcow2 on a local SSD, but the disk is accessible from any host in the cluster and is replicated across Ceph nodes.

The primary use case is live migration of virtual machines. No local disk to transfer. virsh migrate works without data copying: the VM simply restarts on another host, continuing to work with the same data in Ceph. We guarantee an SLA migration time of < 1 second with a properly configured network.

Why RBD Is Better Than Local Storage

Local storage (qcow2) is simpler, but it ties the VM to a specific host and requires data copying during migration. RBD adds 10-20% latency (unnoticeable on NVMe) but provides:

• Live migration without downtime
• Instant snapshots and clones from snapshots (CoW)
• Replication (size=3 by default)
• Disk expansion without stopping the VM
• Integration with Kubernetes via CSI

Our engineers with 10+ years of experience have deployed RBD for projects ranging from 10 to 500 VMs.

Prerequisites: Pool and Authentication

Assume a Ceph cluster is already running. Create a dedicated pool for VM disks and configure replication:

ceph osd pool create vms 0 0 replicated rbd
pool init vms
ceph osd pool set vms size 3
ceph osd pool set vms min_size 2
ceph osd pool set vms pg_autoscale_mode on

Create a separate Ceph user for QEMU with minimal necessary permissions. Principle of least privilege: the libvirt client has access only to the vms pool, not to the admin keyring.

ceph auth get-or-create client.libvirt \
    mon 'profile rbd' \
    osd 'profile rbd pool=vms' \
    -o /etc/ceph/ceph.client.libvirt.keyring

How to Integrate Ceph RBD with KVM/QEMU?

Create an RBD image for the virtual machine and enable necessary features (layering, exclusive-lock, object-map, fast-diff, deep-flatten):

rbd create vms/vm-webserver-01 --size 102400
rbd feature enable vms/vm-webserver-01 exclusive-lock
rbd feature enable vms/vm-webserver-01 object-map
rbd feature enable vms/vm-webserver-01 fast-diff
rbd feature enable vms/vm-webserver-01 deep-flatten

Disk configuration in libvirt XML (virsh edit vm-webserver-01):

<disk type='network' device='disk'>
  <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
  <auth username='libvirt'>
    <secret type='ceph' uuid='9bbd7a99-7512-4dc2-a443-8a0bec3e2acf'/>
  </auth>
  <source protocol='rbd' name='vms/vm-webserver-01'>
    <host name='192.168.20.1' port='6789'/>
    <host name='192.168.20.2' port='6789'/>
    <host name='192.168.20.3' port='6789'/>
  </source>
  <target dev='vda' bus='virtio'/>
</disk>

The libvirt secret is created from the client.libvirt user key. Details are omitted—they are in the official QEMU documentation.

Kubernetes Persistent Volumes via Ceph CSI

For Kubernetes, RBD disks are provided through ceph-csi-rbd—the official CSI driver. Installation via Helm:

helm repo add ceph-csi https://ceph.github.io/csi-charts
helm repo update
helm install ceph-csi-rbd ceph-csi/ceph-csi-rbd \
  --namespace ceph-csi \
  --create-namespace \
  --values ceph-csi-rbd-values.yaml

Example ceph-csi-rbd-values.yaml:

csiConfig:
  - clusterID: "your-ceph-cluster-id"
    monitors:
      - "192.168.20.1:6789"
      - "192.168.20.2:6789"
      - "192.168.20.3:6789"
storageClass:
  create: true
  name: ceph-rbd
  clusterID: "your-ceph-cluster-id"
  pool: vms
  imageFeatures: layering,exclusive-lock,object-map,fast-diff,deep-flatten
  reclaimPolicy: Delete
  allowVolumeExpansion: true
secret:
  create: true
  name: ceph-csi-rbd-secret
  userID: kubernetes
  userKey: "your-base64-key"

PVC provisions in seconds—an RBD image is created in Ceph, formatted, and mounted on a node. This configuration speeds up application deployment by 3x.

RBD Snapshots: Instant VM Copies

Snapshots are a key advantage of RBD. A consistent snapshot of a VM disk without stopping the VM (if guest agent is present):

rbd snap create vms/vm-webserver-01@snap-before-upgrade 

Clone from snapshot (copy-on-write)—instant creation, no extra space until writes:

rbd snap protect vms/vm-webserver-01@snap-before-upgrade
rbd clone vms/vm-webserver-01@snap-before-upgrade vms/vm-webserver-02
rbd flatten vms/vm-webserver-02 # for independence

Monitoring RBD Performance

ceph osd pool stats vms rbd perf image iostat --pool vms ceph osd perf 

For Prometheus, RBD metrics are available via ceph_exporter. Key metrics: ceph_rbd_read_latency_sum, ceph_rbd_write_latency_sum, ceph_pool_wr_bytes. We recommend Grafana dashboard ID 5336.

Typical Issues

Slow ops on OSD during active VM disk workload. Check ceph daemon osd.0 dump_ops_in_flight—you may see stuck operations. Often caused by firewall or MTU mismatch between cluster and public networks.

VM fails to start: rbd: cannot open image, No such file or directory. Verify permissions: rbd info vms/vm-name as client.libvirt. Often due to wrong keyring or expired authentication ticket.

Snapshot freezes VM writes for a few seconds. This is normal with fs-freeze via QEMU guest agent. Without the agent, the snapshot is inconsistent; applications inside the VM may see corrupted data after restoration.

What's Included in Our Work

Timelines and Cost

Cost is calculated individually after infrastructure audit. Our engineers hold Red Hat Ceph certifications; we guarantee 99.9% SLA. Contact us for a consultation—we'll assess your project within one day.