Why a service stays down for hours without automatic failover
It is night, and a database node has stopped responding. The replica is formally alive, but cannot promote itself: manual role switching is prescribed in the procedure, and there is only one master node in the configuration.
The application keeps writing, some requests go to a stale copy, the cache returns data that is no longer in the database — and the service looks operational until replica desynchronization shows up as returns and discrepancies in reports.
The cost of such a failure is not in the failed hardware.
While the on-call engineer gets to a laptop, obtains access, and manually switches the role, anywhere from half an hour to several hours passes; downtime during a node failure is measured not by restart time but by human reaction time.
The single point of failure here is not the server but a single on-call engineer with a phone: there is no one to handle a second failure in a row, and decisions are made based on indirect signs.
The hardest part is recovering from the incident. The returned node stores its own version of data, and you have to manually decide whose copy is primary, what to migrate, and what to discard.
An error at this step quietly makes its way into the production database: some records are lost, some are duplicated in orders and payments.
What changes after implementing switching without human involvement
The main change is that recovery no longer depends on who happens to be on call.
The cluster itself determines that the leader node is unavailable, removes its role, and promotes a new leader, while applications move to the current address through a connection manager without editing configs.
Patroni keeps state in a distributed coordination store, so the decision is made based on the actual loss of quorum, not on someone's command.
Target recovery time becomes a design parameter rather than a wish. Detection thresholds, timeouts, and voting conditions are configured for the workload profile: from failure to the new leader accepting requests, a few minutes pass instead of hours of manual troubleshooting.
After failover, the service is checked automatically — the connection pool comes up, the alert is cleared, and the event remains in the log with the exact time of each step.
An entire class of failures — "the on-call engineer did not react in time" — goes away.
The night shift stops being a point of failure: the engineer reviews an incident that has already been survived, looks at the switchover log, and decides whether planned work is needed. The on-call schedule is relieved, and incident analysis shifts toward prevention.
Service availability stops being assessed by eye. We expose metrics: time to detection, time to recovery, number of failovers and their causes, and the share of scenarios that completed without human involvement. They show where a manual step still remains — and it is closed in the next iteration.
Automatic failover modes: manual, automatic, hybrid
The choice of mode is determined by who makes the final decision — the on-call engineer or the cluster coordinator.
Patroni holds the leader key in a distributed store and, on loss of communication, removes the role from a node, while the cache watchdog mechanism (Redis Sentinel), when the master is unavailable, collects votes and promotes a replica.
Reaction speed is seconds versus minutes for a human, but automation works only with a live cluster quorum.
The second factor is topology. Across two sites, there is no way to assemble a quorum, and then automation is more dangerous than useful: both sides consider themselves alive and write in parallel.
A hybrid scheme helps here — the coordinator raises an alert and prepares a candidate, while an engineer confirms the switchover. When there is no in-house on-call team, some tasks are covered by a managed service, but the provider sets the failover parameters in it.
| Option | When it fits | Limitations |
|---|---|---|
| Manual switchover | One node, training environment, maintenance windows | Downtime lasts as long as it takes the on-call engineer to reach the console |
| Automatic (Patroni, Redis Sentinel) | Three or more nodes in different zones, a dedicated arbiter | During network degradation, a false switchover is possible; verification is needed that the replica has caught up with the log |
| Hybrid scheme with confirmation | Two sites, strict data integrity requirements | Switchover waits for a human: a notification channel and a written procedure are needed |
| Managed service | No in-house on-call team | Some parameters are unavailable, and the provider sets the rollback procedure |
We fix the mode at the design stage and verify it with a training failure: measure the switchover time until traffic is accepted and make sure the data has not diverged.
How implementation proceeds: from audit to handover to operations
The order of work does not depend on what exactly we are protecting — a database cluster, cache, or message queue.
First we record the current state and actual points of failure: without this, automatic failover triggers where it is not expected and adds incidents instead of resolving them.
- Infrastructure audit: inventory of nodes, replication topology, replica lag, current timeouts, and who switches the role today — an on-call engineer following instructions or a script.
- Fault tolerance design: quorum, number of synchronous replicas, behavior on loss of communication between sites. We separately analyze cluster splitting, when both sides consider themselves primary.
- Configuration of the control layer — Patroni: switchover priorities, liveness check intervals, staleness threshold, and the procedure for electing a new leader.
- Client access configuration: a single connection endpoint so the application survives a leader change without editing configurations and restarting.
- Failure testing in the production environment: shut down a node, measure switchover time, data loss, and application behavior. We repeat the drills until the result is reproducible.
- Operations procedure: alerts, on-call responsibility zones, the procedure for returning a node to the cluster, and rules for rolling back to the previous scheme.
- Handover: documentation, diagrams, shift training, and analysis of the first incident together with your team.
After handover, the scheme lives without us: on-call engineers have instructions for every scenario, and you have a clear boundary beyond which our support begins.
Configurations, diagrams, and procedures that remain with you
Automatic failover rests not on a single script but on a set of artifacts that can be read, verified, and handed over to another engineer.
We deliver them in a repository-ready form: configurations in a version control system, diagrams in the cluster description, and procedures in the team knowledge base.
- Working cluster configuration: Patroni parameters (quorum, timeouts, synchronous mode) and Redis Sentinel (quorum, time until a node is declared unavailable) with a comment explaining why exactly that value was chosen.
- Fault tolerance diagram: node topology, points of failure, consensus layer, behavior during network partitioning, and recovery procedure after split-brain.
- Switchover rules: under what conditions the role moves to a replica, who initiates the switchover, what happens with asynchronous replication and possible loss of the latest transactions.
- On-call procedure: alert severity levels, on-call responsibility zone, escalation points, and a checklist of first actions after a failure.
- Verification log: calendar of drills with leader shutdown, switchover time measurements, discovered issues with a note about resolution.
- Instructions for on-call engineers: scenarios "leader down," "no quorum," "replica lagging" with diagnostic commands and rollback procedure.
- Monitoring scheme: cluster state metrics, replication lag, consensus status, alert thresholds, and assignment to the on-call engineer.
Implementation case study: a database under load and a cache alongside
The primary database and cache cluster ran without automatic switching: when the leader node failed, the on-call engineer manually promoted a replica according to a checklist, and the application continued hitting the failed cache node until restart.
Over a quarter of observation — six failures, switchovers took from 20 to 40 minutes, and after each one there remained a risk of diverged replication.
What we changed
Automatic failover was assembled at two levels. The database was moved to Patroni: the cluster configuration lives in a distributed store, election of a new leader happens without a human, and quorum rules prevent a second leader from being promoted during a network partition.
The cache received a scheme with sentinels that vote to switch clients to a live replica — the application does not wait for a restart.
After configuration, switchover takes 20–30 seconds and happens without on-call involvement; incidents per quarter — one instead of six: a short increase in latency instead of service failure.
There is no loss of acknowledged transactions — synchronous replication acknowledges to the client only after receipt by the required number of nodes.
A separate effect is that manual verification of replicas after each switchover disappeared: data integrity is maintained by the cluster itself, not by an engineer with a checklist.
Hardware and network will still fail, but a failure has become a short event according to procedure, not an incident with a nighttime investigation.
How does role switching in a database differ from a cache?
In a database cluster, there can be only one leader node: if two leaders are promoted after a failure, they will start accepting writes independently, and data integrity will be at risk.
Therefore, automatic failover here is not just selecting a new node but proving that the previous leader has been stopped: quorum, confirmation from replicas, delay before making a decision. The cluster deliberately remains unavailable for writes longer than it could.
In a cache, the priority is the opposite. Data is recoverable from the database, so losing the cache is not an incident but a cold start: requests go to the data store all at once, and the load on it grows many times over.
Roles in the cluster are redistributed more easily, but the cost of an error is not record divergence but a surge in latency across the entire service until the cache warms up.
| What we compare | Database cluster | Cache cluster |
|---|---|---|
| Priority | Data integrity | Speed of return to service |
| Leader node role | Strictly one, through quorum | Re-election without waiting for confirmations |
| State after switchover | Records from the last acknowledged point | Cold start, part of the data is lost |
| Main risk | Data divergence with two leaders | Load spike on the database during warm-up |
Hence the different configuration requirements: for the database we set quorum, timeouts, and acknowledgment order; for the cache — switchover speed and correct warm-up.
We verify this with drills: shut down the leader node and see how long degradation lasts, whether the role changed correctly, and how quickly the cache recovered.
What is most often asked about fault tolerance
Before work starts, the concerns of the customer and their engineers are usually the same — from false triggers to who is responsible for operations after delivery. We address them in advance so we do not have to sort this out during an incident.
How do you distinguish a real failure from a false trigger?
A node is declared unavailable not based on a single missed response but on several missed checks in a row with a pause between them. A false failover costs more than no switchover: it breaks connections and leaves two nodes confident in their primary role. We select the threshold for the workload profile, not by default.
What happens during a cluster split?
Writes are allowed for exactly one node; the rest are switched to read-only mode and are prohibited from accepting changes. Having lost contact with the majority, the former leader steps down on its own. The scenario with two sources of conflicting data becomes impossible.
How do you test switchover without stopping the service?
We conduct drills on the live configuration with isolation of one node, not the entire environment — the application continues serving requests. We measure how long it took the new leader to accept the load and whether any incomplete operations remained.
We separately check lagging replicas: a node that has not caught up with the log must not be promoted.
Who is responsible for operations after launch?
We hand over not only the settings but also the rules: how to read cluster state, what to do when a replica is lagging, and when switchover is triggered manually. We run these scenarios with your team during drills. After that, support with agreed response times or an on-call engineer is possible.
Let's discuss your fault tolerance scheme
To calculate cluster behavior during a failure, we need facts, not the wording "we want fault tolerance." What matters to us is the workload profile, acceptable downtime window, and how applications survive a leader change — this determines the choice between synchronous and asynchronous replication, switchover thresholds, and the number of quorum nodes. Therefore, to assess and plan, send us:
- a diagram of the current environment: how many nodes, where they are located, how Patroni and Redis Sentinel are configured, and whether there is a separate voting node;
- a workload description: peak requests per second, share of write operations, data volume, and its growth rate;
- recovery requirements: acceptable downtime, acceptable loss of the latest operations, and behavior during a network partition;
- configurations and current branches of the DBMS and Redis in use, as well as the backup scheme;
- failure history: how the system behaved during node and network failures over the recent period, and where switchover did not trigger;
- access restrictions, maintenance windows, and requirements for data placement in specific environments.
In response, we will provide an assessment and work plan: the target node placement scheme, replication modes and switchover thresholds, and a phased implementation procedure with verification through training failures — communication break, leader shutdown, node return to service.
As a separate block, we will describe handover to operations: instructions for the on-call shift, a set of notifications, and criteria by which switchover is considered standard.
Describe the task in free form — we will come back with an assessment and plan.