Corosync + Pacemaker cluster: where fault tolerance breaks
A fault-tolerant cluster on Corosync and Pacemaker looks reliable right up to the first real failure. Most often, a node doesn't go down entirely — it stays on the network, the resource is listed as running, but the application doesn't respond.
If the check only tracks process presence and runs once a minute, a silent failure is discovered after ten minutes of downtime, when some requests are already lost irretrievably.
Loss of quorum is the second most common cause of accidents when configuring a fault-tolerant cluster. On two nodes without a separate quorum device, after a communication break each half considers itself legitimate, and both continue to manage shared resources.
This is treated with a quorum disk or a correct waiting policy, while careless configuration results in double writes to the database and data divergence that takes longer to sort out than the cluster itself.
False failover occurs where the resource startup time exceeds the configured timeout. The database or queue takes longer to come up than Pacemaker waits, the resource is marked as failed and moves to the neighboring node, where the story repeats. The cluster starts cycling the service in a loop, and the load on the surviving nodes only grows.
And lastly — manual service recovery under load. If node isolation is not configured, during a split you have to shut down half the cluster by hand and compare data line by line. A properly assembled cluster handles such scenarios itself — we'll explain how this is addressed.
What managed fault tolerance provides: the effect in numbers
A cluster of two or three nodes with quorum survives a node failure without manual intervention: the resource manager notices the loss within seconds and brings the service up on a live node.
Service recovery time here is measured not in hours of on-call troubleshooting, but in tens of seconds — provided that dependencies and startup order are described in the configuration, rather than reconstructed from memory.
The share of failures handled automatically is the main metric of the implementation.
While failover is done by hand, each incident keeps the service unavailable until an engineer arrives; after configuring Pacemaker, a disk, network, or process failure is resolved on its own, and no ticket is opened for the on-call shift at all.
Service downtime stops depending on who from the team is on duty and how quickly they read the notification.
The load on the on-call shift drops accordingly: night calls remain only for scenarios that genuinely require a decision — not for restarting a resource or manually moving a service to another node.
We measure false positives separately. Quorum, check intervals, and recovery policies are selected so that the cluster does not fail over on short network degradation: otherwise automation does more harm than good.
Linux fault tolerance is a balance between reaction speed and resilience to short-term failures, and in the report we show both numbers.
Ways to organize fault tolerance: comparing approaches
The fault-tolerance scheme is chosen based on three inputs: how many nodes are available, whether there is shared storage, and how many sites the servers are located at.
These determine how the cluster gains quorum and what it uses to fence a node that has lost connection with the others.
| Option | When it fits | Limitations |
|---|---|---|
| Two nodes with shared storage | A pair of servers and a shared disk array | Quorum of two votes is unstable: a third vote or reliable fencing is needed, otherwise a network split gives two active copies |
| Three or more nodes | A service without shared storage, application-level replication | More hardware and failure points; some services still require a floating address |
| Two separated sites | Requirement to survive the failure of an entire site | Latency between sites limits synchronous replication; when communication breaks, quorum goes to one side, and the other is taken out manually |
| One node with cold standby | Non-critical services and rare downtime windows | Manual recovery, downtime measured in hours, data state lags behind |
The choice comes down to a trade-off: the fewer the nodes, the more acute the quorum question; the farther apart the sites, the more expensive synchronization.
For two nodes without shared storage, a third vote or an external arbitrator is not a complication, but a condition for correct automatic recovery.
Next we'll look at how this is assembled: quorum configuration, fencing policies, and resource startup order — the places where the scheme either withstands a failure or falls apart on its own.
Setup stages: from audit to handover to operations
Infrastructure audit begins with inventory: which services must survive a node failure, what their dependencies are, where service traffic flows, and what is already configured for quorum and fencing.
Without this picture, the cluster is assembled from a template, and the first real failure reveals that a critical resource was started without order constraints. Then the work proceeds step by step — from the scheme to handover.
Sequence of work
- Survey. We record the composition of nodes, shared volumes, and network segments, and gather recovery time requirements for each service.
- Scheme design. We determine the topology, the number of communication rings, the presence of a quorum device, the resource composition, and the rules for placing them across nodes.
- Node preparation. Time synchronization, service network, authentication between nodes, layout of shared storage — the foundation on which the cluster behaves predictably.
- Configuring Corosync and Pacemaker. Resources, groups, startup order constraints, recovery policies, a fencing agent for the specific infrastructure.
- Failover testing. We run a planned migration, node failure, communication channel break, and quorum loss — before it happens in production.
- Regulations and documentation. Cluster diagram, resource map, standard node maintenance procedure, diagnostic scenarios.
- Handover to operations. We go over the cluster's behavior on the test bench with the team, show how to read the state and what to do when fencing triggers.
Configurations, diagrams, and regulations: the handover package
The cluster is handed over not as a 'working system that is best left untouched,' but as a package: configurations, diagrams, and regulations according to which the on-call shift and your engineer can act without us. Below is what physically remains in your repository and internal knowledge base.
- Cluster configurations in version control: Corosync parameters, Pacemaker resource descriptions, quorum rules, and timeouts — with change history and a note explaining why each parameter was chosen.
- Access policies: administrator and on-call roles, rights to restart resources, edit configuration, and manually put a node into maintenance.
- Fault-tolerance diagram: which nodes, disks, and network paths back each other up, where a failure point is left intentionally, and how the cluster behaves when quorum is lost.
- Instructions for the on-call shift: step-by-step actions for typical events — a node did not come back after reboot, a resource moved to the wrong place, node states diverged.
- Recovery plan: procedure for bringing the cluster up from scratch, restoring configuration from the repository, checking data integrity after an emergency failover.
- Check regulations: planned drills with node shutdown, monitoring protective mechanisms triggering, updating diagrams after infrastructure changes.
- Decision log: which scenarios were worked out in drills, what limitations remain, and what was intentionally left outside the configuration scope.
Case study: how downtime during node failure was reduced
The customer is a service with a continuous flow of requests: three nodes, a database replica, a load balancer. Fault tolerance formally existed — resources were described in Pacemaker, a synchronization service monitored node availability.
But failover remained manual: the on-call engineer noticed the failure via monitoring, brought the service up on a surviving node, and only then investigated the cause.
Measurements over six months before the rework: average downtime during a node failure — 47 minutes, of which about ten went to detection, the rest to manual steps with checking the database state.
The shift supervisor spent about an hour restoring the service, and some failures occurred during peak hours, when every minute of downtime is visible to clients.
What was changed
We assembled the resources into a managed group with a strict startup order — first the database, then the application — and configured quorum on three nodes.
We added isolation of the failed node: the one that lost connection with the others is disconnected from shared resources before the second node starts working, so no split occurs.
We separately limited the number of restarts and prohibited automatic failback, so that resources do not 'wander' between nodes.
After switching the scheme to automatic mode, average service recovery time is about 25 seconds: two or three for detection, the rest for resource transfer and application startup.
Over nine months, not a single node failure required manual intervention, and planned checks — sequential node reboots during off-peak windows — proceed without losing requests.
A node failure has turned from an incident into a routine event that the team learns about from the log, not from clients.
How does a two-node cluster differ from a three-node cluster?
Quorum is arithmetic of votes, and the entire failure logic depends on it. In a three-node cluster, the majority is two votes: losing one machine leaves the cluster operational, and resources move to live nodes.
In a pair of nodes there is no strict majority, so Corosync uses a separate voting mode in which a node trusts its partner as long as it can see it.
The price of such savings is cluster split-brain. If the channel between two nodes breaks, each side considers the partner down and brings the service up on itself.
For a database or shared volume, this means data divergence and manual reconciliation, so forced node isolation here is not an option but a launch condition.
The third node does not have to carry load: as an arbitrator, it only participates in voting, so requirements for it are minimal.
An intermediate option is to put the deciding vote on a separate host: it recognizes a split, but will not take over services if the main machine fails.
Maintenance grows with the number of nodes: three machines mean three sets of updates, keys, and monitoring points, but fewer manual investigations after failures.
A two-node cluster is simpler to maintain, but it requires discipline: a separate communication channel between nodes, a working isolation device, and regular failure drills.
A typical mistake is choosing by the principle of 'fewer machines, less hassle' without checking whether a pair of nodes will survive a communication break.
The opposite mistake is three nodes loaded almost to the limit: after one fails, the remaining ones cannot handle the traffic. What should be counted is not machines, but the failures the service must survive.
Technical doubts: addressing common questions
Questions most often clarified before starting: the network scheme and node composition depend on the answers. Below is the system's behavior in emergency scenarios.
What happens if the nodes stop seeing each other?
The cluster loses quorum and stops resources on the isolated part — the service remains on the majority side. We configure liveness checks via a second channel, and the isolation mechanism forcibly powers off the cut-off node before it releases the resource. Without isolation, the quorum policy itself is meaningless.
Can we do without shared storage?
Yes. Data replication is handled by the service — a database replica or distributed storage — while the cluster manages the role and floating address. We configure startup order, readiness checks, and a prohibition on bringing up a copy on a node with outdated data. A shared volume is needed where the application cannot do replication.
How do we make sure failover actually works?
We run a routine migration, a hard shutdown of the active node, a channel break, and a crash of the service itself.
We record failover time, data integrity, and behavior when the node returns — it is the return that most often produces an unstable scheme: the resource comes up on a node without the current copy.
Let's discuss the task: what we'll clarify before starting work
So that the estimate is based on facts, not assumptions, we begin task analysis with inputs on infrastructure and services. We need specifics, not general words: how many nodes are planned, which services must survive a node failure, and within what time failover must complete.
Based on this data, we calculate the scope of work and the composition of the Corosync + Pacemaker cluster.
What we clarify in the first conversation:
- service composition: what we move into the cluster, what must run without interruption, and what can tolerate brief downtime;
- number of nodes and topology: distribution across racks and zones, a separate interface for service traffic;
- isolation of a failed node: managed power strips, remote management controllers, or hypervisor tools — without it, quorum does not protect against a split;
- storage: disk array, block-level replication, or distributed file system;
- failover requirements: target service recovery time and acceptable data loss;
- current state: distribution and current branch, network scheme, monitoring, access, maintenance windows.
You don't have to prepare the list in advance — some answers will be found right in the conversation. Describe the task, and we will come back with an estimate, a cluster diagram, and a staged implementation plan: with checkpoints, resource startup order, and rollback procedure.