PostgreSQL HA setup: where exactly high availability breaks
Database high availability does not appear just because a second server was placed next to the primary one.
Formally, a replica exists, the change stream flows, but the command to switch is given by the on-call engineer — and everything comes down to who noticed the problem first at four in the morning.
PostgreSQL HA setup begins not with installing packages, but with an honest list of cluster failure points: what exactly will stop working if a node goes down.
Primary node failure is rarely the only problem.
The switchover order, quorum, and control node are also failure points: with manual node switchover, downtime stretches to the on-call engineer's reaction time, while automation without coordination risks two nodes simultaneously declaring themselves primary. Recovering diverged data takes longer than waiting out the failure itself.
The third point is a replica that no one knows about. It lags in replication delay, has long lacked space for logs, and no one has checked its readiness for promotion. While the primary holds, this is not visible.
During peak hours, primary failure reveals that the standby node is missing part of the latest changes, and bringing it into working condition quickly will not be possible.
We examine how this is addressed: coordination instead of manual switchover, regular checks of replica promotion readiness, monitoring of lag and log storage headroom — plus a tool such as repmgr, which handles role switching and keeps a history of cluster topology changes.
What a highly available PostgreSQL delivers in practice
A highly available PostgreSQL cluster changes not so much the architecture as the nature of operations: failover to a standby node stops being an emergency nighttime operation.
repmgr monitors node state and, when the leader is lost, promotes a replica automatically — the application returns to service in tens of seconds, not tens of minutes of manual troubleshooting.
The class of eliminated failures is what previously guaranteed service stoppage: node failure, disk failure, loss of communication between sites, a hung database process.
Each such case required manual intervention; now typical scenarios are handled by automation, and the on-call engineer receives a notification after recovery has already occurred.
Under load it is important not just to bring up a standby, but not to lose confirmed transactions.
Synchronous replication mode with quorum preserves them on at least two nodes at the cost of a slight increase in write latency; the parameter is selected to match the workload profile, while read queries go to replicas and offload the leader.
Database availability becomes a measurable share of time, not a promise. The arbitration node protects against cluster split-brain during network partitioning, the former leader returns to service without being rebuilt, and monitoring shows replica lag and switchover history. We verify this with drills: we shut down a node under load and record the actual recovery time.
High-availability cluster options: comparing modes
The choice of scheme is determined not by a "desire for reliability" but by acceptable downtime and whether the business is prepared to lose the latest transactions.
Below are four schemes that are actually chosen between, and their cost in terms of switchover and data preservation.
The differences come down to three questions: where the copy resides, who confirms the write, and who makes the switchover decision.
Synchronous replication gives zero loss of confirmed transactions, but response time increases; asynchronous preserves speed, but if the primary fails, some of the latest writes may not reach the standby.
| Option | When it fits | Limitations |
|---|---|---|
| Single node + backups | Small workload, downtime within an hour is acceptable | Recovery takes minutes to hours, point of loss is the moment of the last backup |
| Two nodes + arbiter node | Automatic switchover without manual intervention is needed | The arbiter does not store data, it only votes: during network partitioning, an incorrect choice is possible |
| Synchronous replication | Transaction loss is unacceptable: payments, inventory, finance | Response latency increases; if the standby fails, writes are blocked or degraded |
| Asynchronous replication | Priority is response speed; loss of seconds of data is acceptable | The latest confirmed writes may not reach the standby if the primary fails |
Schemes are combined: a synchronous standby in an adjacent availability zone plus an asynchronous one in a remote data center.
The arbiter node does not store data; it only participates in voting during switchover, so a minimal machine is sufficient for it — in repmgr this is a separate witness node role.
Implementation order: audit, scheme, replication, verification
A high-availability cluster is not packages installed on several servers, but a sequence of checks where each step is confirmed by fact: node role, replication stream state, switchover result. Below is the work order we follow.
- Database configuration audit. We review the version and build, write-ahead log mode, existing replicas, backup state, and workload profile. The output is a list of bottlenecks and topology decisions.
- Topology design. We determine the number of nodes, whether a quorum witness is needed, where asynchronous replication is acceptable and where synchronous is mandatory, and which failures the cluster must survive without manual intervention.
- Replication setup. We bring up standby nodes, enable streaming log transfer, set write confirmation policies and a lag limit — losing a server must not mean losing already confirmed data.
- Automatic switchover. We configure repmgr: role tracking, standby promotion, split-brain protection when communication between sites is lost, metrics and alerts on lag and role changes.
- Switchover verification. We perform a planned switchover and simulate a primary node failure, measure service recovery time, and compare data after replica promotion. Without this check, the cluster cannot be considered ready.
- Handover to operations. We provide the topology scheme, switchover and update procedures, instructions for the on-call shift, and the procedure for handling an emergency — so the on-call engineer can distinguish normal lag from degradation without calling an engineer.
What we hand over: configurations, schemes, procedures
We hand over artifacts in a repository, not "verbally": every file is versioned, changes go through review, and deployment is reproduced in a clean environment.
Below is the set that covers both day-to-day operations and a nighttime incident when the on-call engineer sees the cluster for the first time.
- Node configurations: PostgreSQL parameters, switchover control, and timeouts — separately for the primary and standby node, with an explanation for each non-standard value.
- Cluster scheme: topology, node roles, standby promotion order, failure points, and their relationship to network segmentation.
- Access rules: database roles and privileges, access to monitoring and replication logs, the procedure for granting and revoking rights.
- Recovery procedure: step-by-step switchover scenarios for primary node failure, returning a node to service, and restoring from backup — with checkpoints and expected time.
- Instructions for the on-call shift: what to check at the start of the shift, what replication lag should be considered alarming, whom to escalate to and how.
- Monitoring map: which checks are configured, what triggers an alert, how to distinguish a false alarm from real degradation.
- Drill protocol: switchover test plan, run log, and results — failure must not be tested for the first time in battle.
Documentation lives together with the configuration: if a rule was changed in the cluster but not in the file, that is a defect that is closed during review.
How we reduced downtime: before and after measurements on a live cluster
Before: every failure woke the on-call engineer
A three-node cluster ran on manual switchover: the master went down at night, the on-call engineer connected, looked at replication logs, promoted the replica to master mode, edited connection endpoints in the application, and only then checked that the data had not diverged.
Recovery took from 40 to 70 minutes, and if the copies were out of sync, it stretched to several hours — in parallel, the lagging records had to be manually reconciled.
A separate cost of such a scheme is the human factor. The sequence of actions was kept in the on-call engineer's head and in a wiki instruction; one missed command led to the application continuing to write to the old master after it returned.
After: switchover in tens of seconds
We placed repmgr on top of streaming replication: the manager monitors node state, promotes a replica to master itself, and distributes the new topology to the others.
The application connects via a host-list connection string and reconnects to a live node without editing configuration and without restarting.
| Metric | Before | After |
|---|---|---|
| Downtime duration | 40–70 minutes | 20–40 seconds |
| Failures resolved without the on-call engineer | isolated | 9 out of 10 |
| Loss of confirmed writes | possible | eliminated |
Synchronous replication closed the residual risk: a transaction is considered complete only after confirmation from the copy, so switchover does not roll back already accepted orders.
Once a quarter we run drills — we forcibly shut down the master and compare the actual switchover time with the target.
What to do if the cluster is already deployed on a single machine?
An existing cluster on a single machine is a setup without a replica: disk failure is handled by restoring from backup, meaning hours of unavailability.
A zero-downtime migration of such a setup is possible, but it cannot start with role switching — first a second node is built, and only then is the primary role transferred. The work order determines exactly how long the service is paused.
The first step is moving the replica out: a copy of the node is deployed on a second server, streaming replication is enabled, and the node is registered in repmgr as a standby. The primary continues to accept writes, and we monitor lag until it reaches zero.
On the third, cheapest node, an arbiter is brought up: it does not store data, but it participates in voting and prevents two nodes from simultaneously considering themselves primary when communication between them is lost.
The transition to the new scheme proceeds via a controlled switchover, not an emergency one. The standby node is temporarily switched to synchronous write mode, they wait until there is no lag at all, and only then is the primary role given to it.
The application connects through a connection pool or virtual address, so the service interruption is measured in seconds, not minutes.
Then the former primary becomes the standby, and lag monitoring, backup, and planned switchover drills are added to the procedure — so the next failure is handled automatically, without manual decisions at the moment of the incident.
Technical concerns: replication, switchover, recovery
Concerns at this stage are predictable: how the replica will behave when it diverges from the master, who makes the decision when communication between nodes is lost, and what confirms that switchover works.
Below are mechanisms, not promises: everything described is tested on a stand before the cluster is brought into production.
The replica has diverged from the master — what then?
A diverged node is not promoted to master: it is marked as broken and removed from rotation. We return it via pg_rewind, and in case of deep divergence — recreate it from a base backup. The cluster continues to run on the current master the whole time.
Why is an arbiter needed in the cluster?
The third node without data participates in voting and prevents two nodes from simultaneously considering themselves master when communication between them is broken.
Without an arbiter, automatic promotion is disabled under uncertainty, and the engineer makes the decision — this is a conscious choice, not a failure.
How to make sure switchover will work?
We run a planned switchover on the stand: we measure write unavailability time, observe how the application restores connections to the new master, and review the replication log. Then we check the return of the former master to service and repeat the run on real traffic.
Does a replica replace a backup?
No: a replica repeats the master's errors and does not protect against accidental data deletion. Therefore we keep base backups and a log archive, and we regularly rehearse recovery on a separate stand, not at the moment of an incident.
How much downtime do planned works take?
Switchover to a healthy replica is a few seconds of write unavailability, and the application reconnects on its own. Updates and changes to replica settings proceed one at a time, one node at a time, without stopping the service.
Let's discuss the task: scheme, timelines, implementation plan
To come back with an estimate of work and an implementation plan, we need a snapshot of your configuration — not a general description, but specifics on nodes, workload, and procedures.
Below is a list of what provides maximum estimation accuracy: if some data is not at hand, we will reconstruct it together.
- Current topology: single node, streaming replication, cascade, or a cluster managed by repmgr; number of nodes and their placement.
- Database size, daily growth, and workload profile — write share, long transactions, peak windows.
- Data preservation and recovery time requirements: acceptable data loss and target replica switchover time.
- Installation and maintenance method: repository packages, containers, manual builds; how you upgrade major versions.
- Network segments, access rules, presence of a connection pool and load balancer in front of the database.
- Monitoring: what is already collected, where metrics go, who receives the degradation signal.
- Maintenance procedure: windows, who is authorized to initiate switchover, whether there is a rollback plan.
After discussing the task, we will come back with a cluster scheme, step-by-step implementation plan, timelines, and scope estimate. Describe the task — we will check it against your infrastructure.