Setting Up PostgreSQL High Availability with Stolon in Kubernetes

Failure of the primary database node takes down the service: Stolon in Kubernetes

A classical single-primary PostgreSQL setup rests on one assumption — that this node is alive. While it responds, writes proceed and replicas catch up with the stream.

Failure of the primary database node stops not only writes: some reads also hang, the application waits for a response, timeouts add up, connections accumulate and hit the pool limit.

Database-induced downtime in this case is measured not by the time the node is down, but by the time it takes someone to react.

Then manual DBMS cluster recovery kicks in. The on-call engineer brings the node up, determines which replica is the most up to date, changes roles, edits the entry point and connection string, and returns the lagging replica to service.

Each step is a human decision under pressure, and during this unavailability window a mistake costs more than the incident itself. If the application managed to write data and the new primary started from a different replica, the divergence is resolved manually by comparing logs.

This is exactly where the PostgreSQL single-node configuration point of failure lies: the question isn't whether the node will fail, but how long we'll take to recover and what we'll lose along the way.

While a human performs failover, the unavailability window predictably stretches, and the volume of lost records depends on how quickly the on-call engineer reacted and how fresh the selected replica was.

Stolon in Kubernetes removes this work from people: it maintains several database nodes, monitors the primary's state, selects a new one by data freshness, and routes traffic through a separate access point.

The application reconnects at the same address without waiting for an engineer to manually bring the cluster back into service. Below we break down how this is covered and what remains outside automation.

What a fault-tolerant cluster provides: measurable effect

Fault tolerance starts with an honest measurement of what is happening now.

A single database instance with manual replica promotion yields an unavailability window of tens of minutes: the engineer notices the failure, confirms it isn't the network, brings the node up from a backup, and switches traffic.

Each such failover is an operation under pressure, and that's exactly where integrity most often breaks: two nodes consider themselves primary, and part of the write goes to an outdated copy.

Stolon removes the manual step from this scenario. The cluster maintains a quorum of control components, role state is stored in a distributed configuration store, and the routing layer directs connections only to the confirmed primary node.

Losing the leading node triggers automatic role failover: the replica is promoted, routing is rebuilt, and the application reconnects at the same address.

In measurable terms, the effect looks like this: the database unavailability window is reduced from tens of minutes to single-digit seconds, the share of automatic failovers approaches one hundred percent — both for planned maintenance and for sudden failure — and the number of manual engineer interventions drops to zero in typical scenarios.

At the same time, an entire class of failures goes away: replica divergence and writes to a lagging node no longer depend on whether someone managed to press the right button at night.

Service availability stops being a promise and becomes a verifiable quantity. We include control drills in the operating procedure — controlled shutdown of a node during working hours — so that we measure recovery time on a live system rather than trust an estimated figure.

Cluster deployment modes: comparison of approaches

Database fault tolerance isn't a binary choice of "cluster or no cluster" but a spectrum of schemes with different implementation effort and different failure behavior. Below are the options that are actually chosen between at project start, and their limitations.

The main difference is who makes the failover decision. In the first two schemes it's a human, in a cluster managed by automation it's the service itself: it monitors node state and changes roles without operator involvement.

Replication limitations are common to all options: streaming transfer is asynchronous by default, so some of the latest confirmed records may not reach the standby, and standby lag directly affects recovery time.

The choice comes down to two questions: how much downtime you're willing to tolerate and how many of the latest records may be lost. The answers determine whether automatic failover is needed at all or whether a cold standby with a clear recovery procedure is sufficient.

Implementing Stolon in Kubernetes: stages of engineering work

Fault tolerance of a DBMS cluster in Kubernetes cannot be added on after launch — it is designed together with the topology, storage, and application connection scheme.

Below is the sequence of work for implementing Stolon in Kubernetes: from auditing the current scheme to handing the cluster over to operations with a verification protocol.

1. Audit of the scheme and load. We analyze how the application connects to the database — through a proxy, a connection pool, or directly, what the query profile and volumes are, and where the single point of failure currently is.
2. Design of a fault-tolerant scheme. We determine the number of replicas, placement across availability zones, the quorum for leader election, the synchronous replication mode, and the target failover time.
3. Configuration assembly. We deploy keeper nodes as a set with persistent volumes, sentinel for state monitoring and leader re-election, and separate addresses for writes and reads.
4. Storage and backups. We select disk classes, configure regular copies to object storage, and verify recovery on a separate test environment.
5. Failure testing. We kill the leader, disable an availability zone, break connectivity to the cluster state store — record the failover time and make sure applications reconnect without manual actions.
6. Handover to operations. On-call procedures, alerts for missing leader and replica lag, planned maintenance scenarios, and training for the customer's team.

The result is predictable recovery time after a node or zone failure, no manual failovers, and a clear set of actions for the on-call shift.

What we hand over: configurations, diagrams, procedures

A cluster that works but is understood only by those who built it is deferred downtime. Therefore handover isn't an archive of files but a set with which your engineer can promote a replica, perform a planned failover, and return to the previous state without our help.

• Fault-tolerance scheme — topology and node roles, failure points, cluster behavior when the master or a zone is lost, and the procedure for electing a new leader.
• Cluster configurations — manifests and environment parameters in the repository, resources and limits, pod placement rules, separate profiles for production and test environments.
• Database access policies — roles for applications, administrators, and support services, schema ownership, procedure for granting and revoking rights, credential rotation, secret storage.
• Backup procedure — schedule of full and incremental copies, retention depth, off-cluster export, scheduled recovery verification.
• DBMS cluster documentation — step-by-step scenarios for planned operations and emergencies, manual intervention procedure, responsibility boundaries.
• Monitoring and alerts — which metrics we look at, at which thresholds an alert is raised, and what to do in the first minutes for each signal.

We hand it over not as a repository link but as a walkthrough on a test environment: we show node failure, failover, and recovery from a copy, then your engineers repeat it themselves.

After that, the team has a single source of truth for the cluster, and they need to contact us only for architecture changes or scaling.

Case: how the cluster survived a node failure without manual actions

Before implementation, the cluster relied on a single master node: when it failed, the application caught connection errors, and the replica was promoted manually — an engineer connected, checked the lag, and switched traffic.

This took from twenty minutes to half an hour, and each time there was a risk of promoting a replica with unresolved lag to master and losing the latest transactions.

We deployed Stolon on top of Kubernetes: a state keeper runs on each node, separate sentinels monitor master health, and the entry point for applications redirects traffic based on the quorum's decision.

Promotion starts on its own as soon as the master stops responding — a human is no longer part of the scheme.

What the metrics confirm

Failover fits within tens of seconds, manual interventions over the quarter are zero, and replica lag stays within the bounds at which confirmed transactions are not lost.

Downtime during node failure was reduced from half an hour to a time the application can calmly survive on request retries.

Before moving to production, we ran scenarios in a load-test environment: we shut down the node with the master, the node with the sentinels, broke connectivity to the state store, and measured how long downtime lasted.

Each run produced a clear fault-tolerance limit — how many nodes the cluster survives without data loss and in which scenario intervention is still needed.

What to do if the database is already deployed on a single node?

A deployed single node doesn't need to be torn down and a cluster built from scratch — the transition is done on a live service. First, a replica is brought up alongside it that catches up with the primary node via the change stream, and only after that are roles changed.

A stop is required only at the moment the entry point changes, not for the entire migration: the window is measured in minutes, not hours of downtime.

The transition order is as follows: take a consistent state snapshot and record the current parameters, deploy the replica and wait for zero lag, promote it to primary mode, and make sure the former node remains readable and synchronized.

The last step is switching the application to the service name instead of the direct address of a specific pod. Such a transition plan is fixed before work begins, so that every step is verifiable and interruptible.

The key condition for seamlessness is that the application already accesses the database by service name, not by a direct link to the node.

Then changing the role in Stolon in Kubernetes is switching to another copy in seconds, without editing the service configuration and without restarting it. If the application still holds a direct address, this is addressed first — before we start touching roles.

The reverse rollback is designed in advance: the former node remains in service as a standby, the state snapshot is stored until the end of the observation period, and the return rules are described step by step for the on-call engineer.

If divergences surface after failover, the service returns to the former node in the same way — by changing the role and address. We review the specific configuration and propose a transition plan for your operating mode.

Engineering doubts we resolve before start

Before the start, the same set of questions comes up: what happens to writes during a cluster split, what replication this database needs, how to update it without downtime, whether the solution will fit into the current environment, and who is responsible for an incident. We answer directly, before work begins.

Synchronous or asynchronous replication?

Synchronous eliminates loss of confirmed transactions but ties writes to the number of live replicas. Asynchronous tolerates node loss more calmly, but the latest confirmations are lost during failover. We fix the choice based on the load profile and acceptable loss window.

What about compatibility with the current environment?

We check the database version and extensions, storage class, network access, monitoring, and backups. If something in the environment cannot survive a master role change, we say so immediately, not during rollout.

How to maintain a DBMS cluster without downtime?

Planned failover, major version upgrade, and parameter rollout proceed in waves: first replicas, then the master. The database accepts writes throughout the entire procedure.

Who is responsible for support after launch?

On our side — configuration, failover automation, and the redundancy scheme; on the customer's side — access, platform resources, and application changes. We fix the boundary in writing before the start.

Let's discuss the task: what we need from you and what we do

To come back with an estimate and work plan rather than general wording, we need a snapshot of your infrastructure. From it we calculate the cluster composition, failure points, and implementation timelines — and deliver this as a single document you can defend before your team.

• current scheme: a single node or manual replication, where the master lives, how applications switch during failure;
• cluster composition: number of database instances, data volume, daily growth, and read-to-write ratio;
• platform: Kubernetes version, availability of multiple availability zones, how persistent volumes are provisioned;
• network and access: how applications find the current master, whether there is a proxy layer, what network policies are in effect;
• fault-tolerance requirements: acceptable unavailability time for writes and reads, whether automatic rollback is needed;
• work windows: when the production environment can be switched, whether there is a test environment and a dataset close to real;
• observability: where metrics and logs are collected, what alerts are already configured.

This is enough to propose a concrete scheme: how many database instances, where to place the master and replicas, how applications find the new master after failover. Then we deploy Stolon and describe the data migration procedure without a long write stop.

The work plan is divided into stages: environment preparation, deployment, migration, failover drills, and production rollout. For each stage we record what we do and what is needed from your team — and only after that do we state implementation timelines. Describe the task — we'll come back with an estimate and a plan.