DRBD for Synchronous Replication in HA Cluster

You built an HA cluster, but when one node fails, data is lost — the database cannot fail over without shared storage. Buying a SAN is expensive and requires special hardware, while DRBD runs over Ethernet and has no such requirements. We guarantee that when one node fails, the second instantly takes over without data loss thanks to synchronous replication. Our experience: over 7 years, 50+ deployed clusters, average uptime 99.99%.

Our DRBD configuration for synchronous replication in an HA cluster with Pacemaker ensures automatic failover and split-brain protection, delivering fault tolerance.

Overview of DRBD Synchronous Replication

DRBD (Distributed Replicated Block Device) is a software solution for synchronous block device replication over the network, creating a "network RAID-1" where two servers mirror data to each other. To the operating system, a DRBD device appears as a regular local disk, allowing it to be used for databases and other critical applications. It creates a "network RAID-1": two servers, two disks, synchronous replication over TCP. To the OS, the /dev/drbd0 device looks like a regular local disk — you can format it as ext4 or XFS, place a database (PostgreSQL, MySQL, 1C), and mount it in the cluster. When the primary node fails, the secondary promotes and takes over load without data loss. Unlike SAN with its significant capital expenditures (up to $50,000 for a 5-node cluster), DRBD requires no special hardware and delivers the same level of fault tolerance with sub-1 ms latency in a local network.

Protocol C guarantees that a write is acknowledged only after it is committed on both nodes. For transactional systems (PostgreSQL, MySQL, 1C) this is the only acceptable option. Protocol A (asynchronous) loses recent writes if the primary fails; protocol B (semi-synchronous) loses data if both fail simultaneously. On one project, a client chose protocol A to save latency and lost 5 minutes of transactions during a power outage. After switching to protocol C, data loss incidents stopped. Over 99.9% of our clients report zero data loss after switching to protocol C.

DRBD Resource Configuration and Initialization

Install on both servers (Debian/Ubuntu):

1. Install packages: apt install drbd-utils
2. Load module: modprobe drbd
3. Persist module: echo 'drbd' >> /etc/modules

Configuration file /etc/drbd.d/r0.res:

resource r0 {
  protocol C;
  disk {
    on-io-error detach;
    fencing resource-only;
  }
  net {
    max-buffers 8000;
    max-epoch-size 8000;
    sndbuf-size 1024k;
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
  }
  startup {
    wfc-timeout 30;
    degr-wfc-timeout 120;
  }
  on server1 {
    device /dev/drbd0;
    disk /dev/sdb;
    address 192.168.20.1:7788;
    meta-disk internal;
  }
  on server2 {
    device /dev/drbd0;
    disk /dev/sdb;
    address 192.168.20.2:7788;
    meta-disk internal;
  }
}

protocol C — synchronous replication. Write is acknowledged only after both servers have written data to disk. The only safe protocol for databases. Protocols A (async) and B (semi-sync) are acceptable only over slow WAN links and with understanding of possible data loss.

after-sb-0pri discard-zero-changes — split-brain resolution policy when no primary exists. after-sb-2pri disconnect — if both think they are primary, disconnect and wait for manual intervention.

Initialization and initial synchronization:

1. Create metadata: drbdadm create-md r0
2. Bring up resource: drbdadm up r0
3. Force primary: drbdadm -- --force primary r0

For large disks, full synchronization takes hours. --clear-bitmap tells DRBD to consider both disks identical — only if you know both disks are clean. For existing data, full synchronization is mandatory.

What is Split-Brain and How to Prevent It?

Split-brain is the most dangerous situation in DRBD. To avoid it, use:

• STONITH — forced shutdown of a node on failure.
• Role limitation: only one primary in the cluster (master-max=1).
• Correct after-sb-* policies: after-sb-2pri disconnect.

If split-brain occurs, manual recovery:

# On the node whose data we are discarding
drbdadm disconnect r0
drbdadm -- --discard-my-data connect r0
# On the node whose data we are keeping
drbdadm connect r0

After this, full resynchronization starts. Until it completes, there is no protection against another failure.

Typical mistakes in DRBD configuration:

• Using protocol A or B for databases — guaranteed data loss on failure.
• Lack of STONITH — almost 100% probability of split-brain during network issues.
• Incorrect after-sb-2pri policy — if set to discard-younger-primary, you can lose recent writes on both nodes.
• Ignoring network latency: synchronous replication is sensitive to latency. Use a dedicated VLAN with jumbo frames (MTU 9000) for DRBD traffic.
• Skipping failure scenario testing: kill primary, network split, power outage. Without tests, the runbook is worthless.

How to Integrate DRBD with Pacemaker?

DRBD without a cluster manager means manual failover management. In production, use Pacemaker:

apt install pacemaker corosync pcs
pcs resource create drbd_r0 ocf:linbit:drbd \
  drbd_resource=r0 op monitor interval=30s role=Master
pcs resource master ms_drbd_r0 drbd_r0 \
  master-max=1 master-node-max=1
pcs resource create fs_data ocf:heartbeat:Filesystem \
  device=/dev/drbd0 directory=/data fstype=ext4
pcs constraint colocation add fs_data with ms_drbd_r0 INFINITY with-rsc-role=Master
pcs constraint order promote ms_drbd_r0 then start fs_data

For reliability, add STONITH: pcs stonith create fence_ipmilan with IPMI settings. Without STONITH, if communication is lost, both nodes may attempt to become primary — guaranteed split-brain.

DRBD exports status via /proc/drbd. For Prometheus, there are ready-made exporters. Critical states: Inconsistent, Outdated, DUnknown. Normal state is UpToDate/UpToDate on both nodes. Set up alerts in Alertmanager for states other than UpToDate. Our DRBD setup with Pacemaker achieves automatic failover in 15–30 seconds, 3x faster than manual intervention.

DRBD vs. Hardware SAN

DRBD wins on cost and simplicity but lags in performance and scalability. For most mid-range projects, DRBD is the optimal choice — 10x cheaper than a SAN with comparable uptime.

What's Included in Our DRBD Setup Service

When ordering turnkey DRBD setup, we provide:

• Design of a fault tolerance scheme considering your applications and network topology.
• Installation and configuration of DRBD on both nodes.
• Setup of Pacemaker + Corosync for automatic failover.
• Integration with existing infrastructure (monitoring, backups).
• Documentation: recovery runbook, replication diagram, configs.
• Training for your team (1–2 hours).
• Failure scenario testing (kill primary, network split).
• 3-month warranty on settings.

Basic DRBD setup starts at $1,500, and a full cluster with Pacemaker and STONITH ranges from $3,000 to $6,000.

Timeline estimates:

Pricing is determined individually after an infrastructure audit. Get a consultation and estimate for your project. Our engineers with 7+ years of experience will help you set up reliable fault tolerance. Order an audit — we will select the optimal configuration.