Automatic database failover: where node failure stops the service
Failure of the primary node rarely looks like a neat crash — more often it is a hung process, disk degradation, or a network partition where the replica is formally alive but the application is no longer writing.
Without automatic database failover, switching falls to the on-call engineer: they must notice the problem, confirm it is not a false alarm, manually promote the replica, and switch the traffic. While this investigation is underway, the service is down.
The cost of manual replica switching consists of two parts. The first is database downtime: while the engineer is figuring things out, write errors accumulate, queues grow, and some requests time out.
The second is transaction loss: if the replica has lagged behind the primary, upon promotion it becomes the new source of truth, and everything that did not make it across disappears irrevocably. The client does not see "database unavailable" but a missing order or a duplicated payment.
A separate class of failures is "silent" degradation. The node responds to checks but stops accepting writes; replication halts due to a conflict or log overflow.
External monitoring based on a single ping does not catch this, and manual investigation takes tens of minutes — exactly where every second counts.
Automatic failover is not designed as a "button" but as a set of conditions: who and by what criteria declares a node dead, how a new replica is chosen, how network partitions are guarded against so that two primaries are not promoted.
Below we break down how this is handled for PostgreSQL and MySQL — from choosing a new replica to returning traffic.
How Automatic Failover Works: Modes, Metrics, Choosing a Scheme
Automatic database failover is not "flipping a switch" but choosing a failover scheme for the load profile. Below are three working configurations that we deploy, and the honest limitations of each.
Scheme with a Quorum Arbiter
A separate lightweight arbiter node votes for promoting a replica and does not store data. The cluster survives the failure of the primary node without manual intervention: recovery time stays within tens of seconds, and split-brain is excluded.
The limitation is that the arbiter itself becomes a point of failure, so it is placed in a separate availability zone.
Synchronous Replication Mode
A transaction is confirmed only after being written to the replica — the share of lost transactions drops to zero. The cost: write latency increases by the confirmation time, and if the channel degrades, writes may be blocked.
It is suitable for financial and accounting systems where the loss of even a single transaction is unacceptable.
Asynchronous Replication with Fast Promotion
Writes do not wait for the replica, so latency is minimal. In the event of a failure, the last transactions may be lost — the window is measured in seconds. The choice for read-heavy services where speed matters more than strict consistency.
| Option | When it fits | Limitations |
|---|---|---|
| Quorum arbiter | Critical service, automatic promotion needed | Additional node in a separate zone |
| Synchronous replication | Finance, accounting, no transaction loss allowed | Increased write latency, risk of blocking |
| Asynchronous replication | Read-heavy, high throughput | Window of losing the last transactions |
The metrics by which we choose a scheme: recovery time, share of lost transactions, number of manual interventions per quarter. For orchestration we use Patroni — it manages promotion and maintains quorum through a distributed store.
We will analyze your load profile and select a scheme to match it.
Implementing automatic failover: stages of engineering work
Automatic database failover is not just about installing a single observer daemon: failover is the result of a correctly designed topology, replication, and voting rules.
That is why we start with an audit rather than with configuration, and we take it all the way to validation with a real failure.
- Database topology audit. We capture the current picture: where the master is, which replicas exist (synchronous or asynchronous), how the application connects — through a connection pool, a proxy, or directly by address. We identify single points of failure: a single coordinator node, lack of quorum, manual master promotion.
- Failover scheme design. We define the model: leader-follower with automatic promotion or a quorum scheme with distributed consensus. We set thresholds — how many seconds of master silence count as a failure, how many replicas must confirm promotion to prevent "two masters" and data divergence.
- Replication configuration. We bring the data flow into order: synchronous mode for critical transactions, replica lag monitoring, correct conflict handling after promotion. We move the application to an intermediary address that survives a master change without editing configurations.
- Failure testing. We simulate master crash, network partition, and the return of the old node to service. We measure failover time and verify that the application restored writes without manual intervention and that no data was lost.
- Documentation handover. We deliver the topology diagram, voting rules, instructions for planned failover and rollback, as well as an analysis of scenarios in which the automation intentionally does not trigger.
The work sequence is structured so that each stage is verifiable before moving to the next: unverified replication at the design stage turns into data divergence at the very first real failure. Describe your current configuration — we will get back to you with an assessment and a work plan.
How is automatic failover different from manual switching and load balancing?
Automatic database failover solves a problem that neither manual switching nor read balancing addresses. These are three different mechanisms, and confusing them means leaving the system unprotected at its most vulnerable point.
Manual switching is when an engineer notices a failure and manually promotes a replica to the leader role. Until they react, writes are stalled: applications get errors, transactions are rolled back, and the queue of accumulated operations grows.
At night, on weekends, at peak load, the reaction takes minutes — and all that time business processes are down.
Automatic failover removes the human from this chain: the system itself detects the loss of the leader, selects the most up-to-date replica, and switches it into write mode.
Read balancing is about something else. It distributes read requests across replicas to reduce the load on the primary node. It is about performance, not write fault tolerance: if the leader goes down, the balancer will keep serving reads, but there will be nowhere to write data.
Therefore, read balancing and automatic failover are not alternatives, but two layers of a single architecture.
The key concept here is the leader role. At any given moment, only one node accepts writes; the remaining replicas catch up with it via the change log.
Automatic failover monitors the health of the leader and, on failure, transfers its role to the next node, while the remaining replicas are reconfigured to the new leader. Read load distribution is preserved — only the write point changes.
The practical takeaway: read balancing scales the load but does not protect against failure. Write fault tolerance is provided precisely by automatic failover — and only it reduces downtime from minutes of manual intervention to seconds.
Describe your configuration — we will get back to you with an assessment and an implementation plan.
Case: reducing downtime during node failure in a database cluster
The client came with a classic picture: a cluster leader node and a replica, manual failover according to a wiki instruction.
When the leader failed, the on-call engineer brought the replica up manually — checking lag, switching it to write mode, changing the connection string in the application, warming up the cache.
According to incident logs, service recovery took from 25 to 40 minutes, and in a third of cases the failover was error-prone: the application kept writing to the failed node, some transactions were lost, and the rollback required yet another downtime window.
We analyzed the failure chain and removed the human from it.
The cluster got a quorum arbiter that confirms the state of the nodes and prevents two replicas from simultaneously considering themselves leaders — this closes split-brain when communication between sites is interrupted. Node state is checked by several independent signals rather than a single ping.
The application switches to the new leader node through a connection pool with liveness checks: the connection string points to a virtual address, and changing the node requires no configuration edits or service restarts.
What changed in the metrics
| Metric | Before | After |
|---|---|---|
| Service recovery time | 25-40 minutes | 40-90 seconds |
| Share of successful failovers | ~70% | 100% on training and real failures |
| Manual operations by the on-call engineer | 8-12 steps | 0 during a standard failure |
| Transaction loss during failover | up to several hundred | 0 in synchronous replication mode |
We separately checked the flip side: the automation must not trigger on short-lived network spikes.
The failure confirmation threshold and the delay before failover are configured so that a brief communication glitch does not initiate a leader node change, while a real failure does. Over six months of operation, not a single false failover.
Manual operations remained only where they are appropriate: scheduled maintenance and returning a node to service after repair. This reduced the load on on-call engineers and removed the main source of errors — haste during an incident.
Patroni
Doubts Before Implementation: Compatibility, Migration, Rollback, and Access
Automatic failover is an add-on to your current replication, not a replacement for your infrastructure.
If you already have streaming change propagation between the primary node and the replica set up, we add a control layer: a monitoring agent, a quorum for electing a new leader, and traffic routing.
Existing data schemas, indexes, and user permissions remain untouched — only who makes the failover decision and how quickly it happens changes.
What if the current database version doesn't support the required mode?
We check this during the scoping phase: we verify the branch, the presence of extensions, and the replica topology, after which we propose either an upgrade to the current branch or a scheme with an intermediate arbiter node. How does the migration work?
In stages: first the candidate replica is brought into the quorum, then the load is shifted to it in read mode, and only after a validation run does failover become automatic. What if something goes wrong after implementation?
The rollback procedure is described in advance: returning to the previous node, removing the agent from the quorum, and restoring manual control take minutes, not hours.
Who gets access to the control nodes and how is it segmented?
Access is granted by role: operations can see the cluster state and can initiate manual failover, while development gets metrics and logs only. Privileged operations on the quorum and agent configuration are available to a limited group with mandatory audit logging. Keys and tokens are stored in a secure vault, not in configuration files on disk.
The customer receives a complete package:
- Monitoring agent, quorum, and traffic router configurations with comments on each parameter.
- A diagram of the current and target topology indicating node roles and failure points.
- An access segmentation policy: roles, permissions, and the procedure for granting and revoking them.
- A manual failover regulation and rollback procedure with step-by-step commands.
- A set of checks for monitoring cluster state and trigger thresholds.
- Operations documentation: what to look at first during a failure and where to go next.
Describe your current topology and database version — we'll come back with an assessment, a migration plan, and a list of work tailored to your environment.
Automatic failover implementation timeline by phase
Timelines depend not on the size of the database, but on the number of failure points in the current architecture: a single node with manual switching is completed faster than a cascade of replicas with a load balancer and read/write splitting.
Below is an approximate duration by phase, without being tied to calendar dates. We provide an estimate for your specific environment after an audit.
The phases partially overlap: design runs in parallel with staging preparation, and runbooks are written during implementation.
The most unpredictable item is the trial runs: that is where version discrepancies, non-standard extensions, and queries that do not survive a role change come to light. That is why we do not shorten the staging — it is cheaper to catch split-brain on a copy than in production.
We separately allocate time for the observation window after handover: for a week the on-call shift works with the new scheme under our supervision, and only then do we close the project.
The total duration for a typical environment is three to six weeks; a cascade of replicas and geo-distribution add one to two weeks to that.
