Why MariaDB Galera Cluster loses synchronization under load
MariaDB Galera Cluster confirms a write not on one node, but after all participants certify the transaction. While nodes keep up equally, write confirmation and replication are inseparable: there is no data loss if one server fails.
The scheme breaks exactly when one node stops keeping up — the flow control mechanism slows the entire cluster to the speed of the slowest participant.
The second step to failure is node drop-out. The server disappeared over the network or got stuck on a full state transfer, the remaining nodes do not gain a majority, and quorum loss puts part of the cluster into a mode without write permission.
The application gets a node-not-ready error, connections hang waiting, and writes stop entirely, although the disks are intact and the data is in place.
More dangerous is the reverse scenario: while the cluster is split, the smaller part manages to accept and confirm transactions to the client, and after connectivity is restored they fail certification and are discarded.
Thus a confirmed transaction does not reach the other nodes — this is no longer database fault tolerance, but a divergence that has to be reconciled manually.
At peak load such a failure is more likely: certification queues grow, write locks are held longer, and a spike on one node through flow control accelerates delays on all of them. Any restart or state transfer on a hot cluster especially easily knocks out quorum.
We analyze how this is closed: how to calculate quorum and node weight, how to keep a lagging node in the cluster, and how to build a scheme in which node failure does not lead to write blocking.
What synchronous replication with write confirmation provides
A write is considered confirmed only when it has been accepted by a quorum of nodes, not by a single leader server.
The client receives a response only after commit on a majority — therefore the failure class "loss of confirmed data" goes away: if a node drops immediately after the response, the transaction still remains on the others.
MariaDB Galera Cluster closes this at the replication level itself, without application modifications.
If a node fails, quorum reassembles itself, and traffic routing moves clients to live nodes within seconds. Switchover time is measured in units of seconds and does not depend on database size — switchover does not wait for logs to be applied. The application sees a short connection interruption, not minutes of downtime.
A returning node recovers in a hot way: it receives a state snapshot and catches up with the change stream while the others continue accepting writes. Write is not stopped — downtime reduction is achieved because recovery proceeds in parallel with the workload.
The price for this is that write latency is determined by the slowest node in the set, so we choose the network between nodes and their number deliberately: odd quorum, without extra participants.
The weak link is visible in replication flow metrics, and this is the first thing we check during configuration.
MariaDB replication modes: comparison of options
Replication modes differ not in "completeness," but in who confirms the commit of a change and what happens when nodes diverge. This determines behavior during failure, sensitivity to latency between sites, and what rollback is even possible.
| Option | When it fits | Mode limitations |
|---|---|---|
| Asynchronous replication | reporting copies, distributed sites where loss of the latest changes is acceptable | the primary node confirms the commit without waiting for the copy: if failure occurs before log transfer, the difference is lost irrevocably |
| Semisynchronous replication | a balance is needed between write speed and the guarantee that the copy has accepted the log | confirmation arrives before application on the copy, so during switchover some changes are still catching up; behavior when the copy is unavailable is set by timeout policy |
| Synchronous confirmation (Galera) | critical data where node divergence is unacceptable, and nodes are in one network with low latency | commit passes only on a majority of nodes: loss of quorum stops writes, and an increase in the number of nodes increases write response time |
| Delayed-application copy | protection against an erroneous mass change | it does not speed up switchover, requires a separate node and space for logs |
The choice is not binary: in one installation, a synchronous cluster usually carries critical databases, while an asynchronous copy carries reporting and analytics. This way read load moves off the primary environment, and latency requirements between sites remain achievable.
Implementing a fault-tolerant cluster: work stages
A synchronous replication cluster on MariaDB Galera is not assembled "live": the order of work determines whether failover will be predictable or turn into manual operations under pressure. We finish each stage with an artifact that is checked before moving to the next.
- Audit of the current scheme. Topology, hot data size, latency between nodes, out-of-regulation schema changes, application behavior on write error. Result — a map of bottlenecks and a list of queries unacceptable in the cluster: long blocking transactions, mass inserts without splitting.
- Topology design. Number of nodes, quorum, a separate arbitrator outside the main failure zone, transaction confirmation mode with write to quorum. Here we also fix behavior on quorum loss: write stop automatically or by decision of the on-call engineer.
- Configuring synchronous writes. Replication flow parameters, transaction size limit for catching up changes, node behavior when lagging, switching the node to no-write mode. We check how the application survives transaction rollback and reconnection to another node.
- Failure tests. Planned node removal, abrupt stop, network break between zones, quorum loss, full node rebuild from a donor. We record write recovery time and the number of lost confirmed transactions — target zero.
- Backup and recovery. Snapshot from the donor node, recovery verification on a separate environment, full synchronization procedure when catching up changes is impossible.
- Handover to operation. Monitoring of node state and lag, instructions for the on-call engineer, procedure for node replacement and schema update in a maintenance window. After that — support until the team can replace a node itself.
Configurations, quorum schemes, and switchover procedures
After implementation, the customer's team has not only a working cluster, but also a set of artifacts by which it can be operated without us. Below is what we transfer to the customer's configuration repository and review during handover.
- Node configuration files. The full set of synchronous replication parameters: cluster name, write confirmation mode, node behavior on loss of connection with the group, limits on the change stream. Laid out by machine and duplicated in version control.
- Node placement scheme. How many nodes, in which availability zones, how quorum is distributed, what latencies between sites were assumed and why exactly such a composition of voters.
- Switchover procedure. Step-by-step order for controlled node removal, actions on quorum loss, the point at which a decision is made to move traffic to the backup site.
- Access rules. Roles for the application, service diagnostics, and recovery, from where connections are allowed, who has the right to change cluster parameters and how this is recorded.
- Recovery documentation. Scenarios for cold start of the group, bringing up one lagging node, procedure for working with a backup and reconciling state after return.
- Checklists. Checks before configuration change, after emergency switchover, and regular monitoring of data divergence between nodes.
- Metrics and alerts map. Which node and cluster indicators are considered normal, which require on-call intervention, and to whom the signal goes.
Case: an online store cluster without downtime during node failure
How it was before
The store ran on one primary database and an asynchronous replica. When a node went down, the replica lagged, and some orders placed at the moment of failure disappeared — the share of lost transactions reached 0.3% of daily flow.
Divergences were discovered from customer complaints, that is, after the client had not received order confirmation.
Switchover was done manually: the on-call engineer brought up the replica, changed the connection point in the application configuration, and only then removed maintenance mode.
Downtime was from 4 to 11 minutes, and every fourth failure ended with manual node rebuild from a backup.
What was changed and what became
We assembled a cluster of three nodes with synchronous replication: a write is confirmed only after it has been accepted by a majority of nodes, so order loss during failure of one node became impossible.
Failure is handled automatically — the remaining two nodes preserve quorum, and the cluster continues accepting writes without on-call participation.
Switchover time decreased from minutes to 3-8 seconds, the share of lost transactions — from 0.3% to zero, the share of manual recoveries — from a quarter of failures to one in ten, and only when the node disk is completely unavailable.
Incident analysis takes about 20 minutes: the log shows which node dropped, why, and when it returned to service — previously log reconciliation took hours.
We verified the settings against two independent sources — Galera Cluster documentation and community quorum reports: they confirm that the chosen mode gives predictable failure rather than silent data loss.
What to do if an asynchronous cluster is already deployed?
Migration is possible without a long unavailability window: data on replicas already exists, only the method of receiving it and confirming writes changes.
The order here is the reverse of the usual one — first we introduce one replica into the synchronous cluster, while the current master continues to work as before.
Before that, we check the current configuration. All tables must be InnoDB and have a primary key: without a key, synchronous row application works inefficiently.
We look at whether replica lag is growing, whether schemas on nodes match, whether there are direct connections to replicas for reads — after switchover these are already equal cluster participants.
Separately — long transactions and table schema changes: they are coordinated between nodes and on a large table can slow down writes.
Application backward compatibility is the main risk. The asynchronous scheme tolerates conflicts: divergences accumulate and surface later. The synchronous one returns them immediately as a transaction error, so the code must be able to retry the operation rather than fail.
The second is the connection point: a master specified by name is changed to the cluster address, otherwise write distribution will not work.
Migration plan: fix the settings, bring up a new node next to the replica, and transfer data with a physical copy — pouring a dump creates exactly that long window. Then we switch the application, add the remaining nodes, and monitor quorum.
The unavailability window itself is minutes for changing the connection point and checking writes and reads. Rollback is simple: return the application to the previous master; it remains a cold standby while the cluster proves itself under load.
Common technical questions before cluster implementation
At the design stage, it is important to check not replication itself, but cluster behavior in abnormal scenarios: connection break between sites, adding a node under load, write response. Below are five engineering-level questions with short answers.
How many nodes are needed for quorum?
Three is the minimum composition in which the cluster survives failure of one node and preserves quorum. Two nodes are fault-tolerant, but without reserve: loss of one leaves one vote, and the cluster goes into read-only mode. For two sites, we add a lightweight arbitrator without a data copy.
What happens during a network partition?
The smaller part loses quorum and goes into read-only mode — writes on it stop so that data versions do not diverge. The larger part continues serving transactions, and after connectivity is restored the lagging nodes catch up with state themselves.
Can a node be added on the fly?
Yes, service stop is not needed. The new node receives a copy of the data and joins voting only after catching up. During the full copy transfer, one node acts as donor — we distribute its load in advance so as not to degrade response.
Does synchronous replication slow down writes?
Write response is determined by network latency between nodes, not disk. Within a site this is fractions of a millisecond; between regions it is more noticeable.
Group commit combines parallel transactions into one coordination round, so throughput under mass load does not degrade.
How does a node recover after a failure?
After a short absence, it catches up with changes from the log and returns to service without a full database copy. If it has lagged more than the log stores, a full transfer is started. Data divergence between nodes is caught separately and requires node rebuild.
Let's discuss the task and come back with an estimate and plan
So that the estimate is to the point and not "by eye," initial data on the infrastructure and availability requirements are needed to start.
Describe the current replication scheme, number of nodes and sites, database volumes, and write load profile — this is enough to understand which quorum and synchronization modes will suit you specifically.
- The current topology scheme: how many nodes, how they are distributed across sites, where the failure points are now.
- Volumes and nature of the load: database size, read-to-write ratio, peak periods.
- Availability requirements: allowable downtime during switchover and whether automatic failover is needed when a node fails.
- Consistency requirements: which transactions must be confirmed on a majority of nodes, and where asynchronous replication is sufficient.
- How backup and recovery are organized — so that the work plan does not break existing procedures.
- Constraints on the work window: when the configuration can be changed without affecting users.
In return, we will come back with an implementation work plan: stage-by-stage node migration, cluster composition, order of checks, and acceptance procedure with measurable criteria.
We fix in advance which failure the cluster must survive and in what time it returns to normal after a failure. Describe the task — we will discuss the details and come back with an estimate and plan.