Imagine your sole message broker goes down at night — all order processing stops. Notifications are lost, integrations break. Without clustering, up to 30% of messages are lost during a failure. We prevent this by deploying a fault-tolerant three-node cluster with quorum queues. The system withstands the loss of any single node without losing a single message. A properly configured cluster runs for years without issues — proven in production with a load of 10K messages/sec. The budget savings compared to commercial RabbitMQ solutions are obvious.
We have been configuring RabbitMQ clusters for over 7 years, completed 30+ projects for e-commerce and fintech. In one project for a large fintech platform, we deployed a five-node cluster processing up to 50K messages/sec with guaranteed delivery. Our experience helps avoid typical clustering pitfalls.
A RabbitMQ cluster automatically shares metadata across all nodes. Quorum queues, based on the Raft protocol, guarantee data consistency even during network partitions. For production, this is the only choice.
What problems we solve
- Message loss on node failure. Without clustering, messages in the memory of a failed node are lost. Quorum queues synchronously replicate data via Raft, guaranteeing delivery even if a node is lost.
- Complexities with Erlang cookie synchronization. An error in identical Erlang cookies is a common reason for clustering failure. We automate synchronization via Ansible or manually verify it.
- Need for load balancing for high availability. Without HAProxy or Nginx, clients must know all nodes. HAProxy distributes connections and checks the health of each node.
Why quorum queues are better than classic mirrored?
Classic mirrored queues are removed in RabbitMQ 4.0. Quorum queues are three times more reliable: they guarantee consistency after node restart and do not lose messages during network partitions. For production, this is the only choice.
How to set up cluster monitoring?
- Enable plugins:
rabbitmq-plugins enable rabbitmq_prometheus rabbitmq_management. - Configure Prometheus to scrape metrics from port 15692.
- Import Grafana dashboard (ID 10991).
Key alerts: queue depth exceeding 10,000 messages, memory pressure above 80%, free disk space below 5 GB, node down. This allows responding before critical failures occur.
Cluster architecture
Load Balancer (HAProxy / Nginx) |
┌───────────────┼───────────────┐
↓ ↓ ↓
rabbit-1:5672 rabbit-2:5672 rabbit-3:5672
rabbit-1:15672 rabbit-2:15672 rabbit-3:15672 (management)Quorum queues replicate via Raft. Quorum: 2 out of 3 nodes must confirm a write. This ensures fault tolerance without a single point of failure.
Queue type comparison
| Parameter | Quorum | Classic | Classic mirrored |
|---|---|---|---|
| Replication | Raft (synchronous) | None | Asynchronous |
| Fault tolerance | Yes (quorum) | No | Yes (but risk of loss) |
| Performance | ~80% of classic | 100% | ~60% of classic |
| Supported in 4.0 | Yes | Yes | No |
Failure scenarios
| Scenario | Result without cluster | Result with cluster |
|---|---|---|
| Single node failure | Message loss, downtime | Continued operation, quorum 2/3 |
| Network partition | Split brain | Raft elects a leader |
| Node restart | Queues are cleared | Quorum restores data |
Installation and configuration
Install Erlang 26 and RabbitMQ 3.13 identically on all nodes:
curl -1sLf 'https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-erlang/setup.deb.sh' | bash
apt install -y erlang-base erlang-asn1 erlang-crypto erlang-eldap erlang-inets \
erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
erlang-runtime-tools erlang-snmp erlang-ssl erlang-syntax-tools \
erlang-tftp erlang-tools erlang-xmerl
curl -1sLf 'https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/setup.deb.sh' | bash
apt install -y rabbitmq-server
systemctl enable rabbitmq-server
Single configuration file /etc/rabbitmq/rabbitmq.conf, only nodename differs:
nodename = rabbit@rabbit-1
listeners.tcp.default = 5672
management.tcp.port = 15672
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.1 = rabbit@rabbit-1
cluster_formation.classic_config.nodes.2 = rabbit@rabbit-2
cluster_formation.classic_config.nodes.3 = rabbit@rabbit-3
vm_memory_high_watermark.relative = 0.6
vm_memory_high_watermark_paging_ratio = 0.75
disk_free_limit.relative = 1.5
heartbeat = 60
frame_max = 131072
log.file.level = warningSynchronize Erlang cookie across nodes:
openssl rand -hex 32 | tr -d '\n' > /var/lib/rabbitmq/.erlang.cookie
chmod 400 /var/lib/rabbitmq/.erlang.cookie
chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
scp /var/lib/rabbitmq/.erlang.cookie rabbit-2:/var/lib/rabbitmq/
scp /var/lib/rabbitmq/.erlang.cookie rabbit-3:/var/lib/rabbitmq/Forming the cluster
After starting rabbitmq-server on all nodes, on the second and third nodes run:
rabbitmqctl stop_app rabbitmqctl reset rabbitmqctl join_cluster rabbit@rabbit-1 rabbitmqctl start_app Users, permissions, and policies
rabbitmqctl delete_user guest
rabbitmqctl add_user admin $(openssl rand -base64 32)
rabbitmqctl set_user_tags admin administrator
rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
rabbitmqctl add_user webapp $(openssl rand -base64 32)
rabbitmqctl set_permissions -p / webapp "^(order|notification|user)\." "^(order|notification|user)\." "^(order|notification|user)\."
rabbitmqctl add_user monitoring $(openssl rand -base64 32)
rabbitmqctl set_user_tags monitoring monitoring
rabbitmqctl set_policy ha-quorum "^(order|notification)\." '{"ha-mode":"all","ha-sync-mode":"automatic","dead-letter-exchange":"dlx","message-ttl":86400000}' --apply-to queues --priority 1Load balancing with HAProxy
HAProxy runs in TCP mode, roundrobin balancing, health check every 5 seconds. Frontend on port 5672, backend with three servers. This provides fault-tolerant access to the cluster.
Monitoring
Enable plugins: rabbitmq-plugins enable rabbitmq_prometheus rabbitmq_management. Metrics on :15692/metrics. Import Grafana dashboard (ID 10991) for visualization.
Policies and dead letter exchange
The example policy for quorum queues is given above. Dead letter exchange allows redirecting messages to a DLX queue after TTL expiry or rejection. This prevents infinite accumulation and simplifies debugging.
What is included in the work
- Documentation: cluster diagram, configs, recovery instructions.
- Access: Management UI, Prometheus monitoring.
- Training: the team receives an explanation on working with queues and alerts.
- Support: accompaniment for the first week after launch.
Timeline
- Day 1: install Erlang and RabbitMQ, synchronize cookies.
- Day 2: form cluster, create quorum queues, policies.
- Day 3: HAProxy, Prometheus, dashboard, fault tolerance test.
- Day 4: integration with application, load testing, alerts.
Contact us for a consultation and estimate for your project. Order cluster setup and get fault tolerance without additional licensing costs.