Keepalived: why the failure of one node takes down the service
A typical setup usually looks fault-tolerant: the application is deployed in several instances, the database is replicated, and capacity is allocated for standby.
But the entry point to the service — a load balancer, reverse proxy, or gateway — is tied to a single node, and it turns out to be a single point of failure. All the redundancy behind it is nullified by the very first failure.
What breaks is not the application, but the address. A node goes down due to power, kernel, or a failed update — and the entry point that clients use stops responding.
The DNS record does not switch instantly: the record's TTL and resolver caches get in the way, and manually moving traffic takes minutes and is done at night according to an instruction.
Downtime when a node fails is measured in tens of minutes: users see errors and timeouts, while the on-call shift deals with the incident and edits configuration under pressure.
The cost of such a scheme is not only in hours of unavailability. Every manual failover is a risk of making a mistake: the wrong node, a forgotten route, lost sessions.
As long as the manual scenario remains the only one, service fault tolerance rests on the on-call engineer's speed, not on the architecture.
We break down how this is addressed. Keepalived keeps a floating address on a pair of nodes via VRRP, checks their state with scripts, and moves traffic without human involvement — with predictable failover time and no nightly manual edits.
What fault tolerance provides: measurable effect
The effect of fault tolerance is measured by three values: how long the service is unavailable when a node fails, what share of incidents reaches users at all, and whether servers can be maintained without stopping operations.
Automatic failover instead of manual intervention
Previously, a failed node was noticed by the on-call engineer — via a monitoring notification, sometimes via a complaint. While they connected and moved the address by hand, the service was down for tens of minutes, and every minute of downtime fell during working hours.
Failover is performed by Keepalived: the standby node takes the address within seconds, with no human in the loop.
Service recovery time shrinks from tens of minutes to 1-3 seconds, and the on-call engineer learns about the event from the log when the service is already receiving traffic.
The "node unavailable" failure class disappears from statistics
While one node reboots or is being repaired, the second one carries traffic — hardware failure ceases to be service failure. What matters is how liveness is checked: a network-level response confirms that the machine is on, but not that the application is responding.
An application request check filters out nodes that are formally alive but returning errors. A complete service failure then requires all nodes to fail simultaneously, that is, a common cause — network, power, or a failed release.
Service availability grows not "on average," but by eliminating an entire class of incidents.
Scheduled maintenance without downtime
Maintenance proceeds one node at a time: it is taken out of traffic processing, updated, returned to service, then the next one is handled. Availability does not drop, there is no need to coordinate a nightly window or rush — work is moved to convenient hours.
This scheme also pays off for rare operations: replacing hardware or changing configuration no longer turns into planned downtime for users.
Keepalived operating modes: comparison of options
The redundancy scheme is chosen before configuration: node priorities, the number of virtual addresses, and how the cluster behaves when a failed server returns to service depend on it.
VRRP allows several working layouts, and the difference between them is how many nodes actually carry load and how predictable failover is.
| Mode | When it fits | Limitations |
|---|---|---|
| Active and standby node | One service, simple incident triage, standby being idle is acceptable | The standby node sits idle; after the primary recovers, the address returns to it only if ownership takeover is allowed |
| Peer nodes with different priorities | A pair of servers where it is important to quickly return the address to the primary node | Priority is set statically — when adding a third node, the scheme is recalculated manually |
| Distribution of virtual addresses across nodes | Several services, both servers need to be utilized | Each node becomes primary for its own group of addresses: twice as many configurations and availability checks |
In any mode, node health is determined not only by service packet exchange, but also by an application availability check. A check script lowers the node's priority if the service does not respond — the virtual address moves to a live server.
Without it, Keepalived continues to hold the address where the process has already crashed, and failover does not occur.
Stages of implementing a fault-tolerant cluster
Work proceeds from inventory to handover: first we record what actually runs in the environment, then we design a scheme with redundancy for each critical service, and only then do we touch production nodes.
This order excludes the situation where fault tolerance is enabled for services whose network dependencies are not ready for redundancy.
- Infrastructure audit — we assemble a map of nodes, network segments, and services, mark single points of failure and connections that will survive moving the virtual address, and which ones will not.
- Redundancy scheme — we determine node pairs and roles, distribute priorities in Keepalived, choose the role return mode and VRRP advertisement intervals so that failover does not trigger on short network bursts.
- Configuration and availability checks — we write configuration and scripts that assess the state of the service itself, not just host liveness: otherwise a node with a crashed application will continue to hold the virtual address.
- Failover testing on the test bench — we reproduce failures: stopping the daemon, link failure, node reboot, service hang. We measure the time it takes for the standby to take the address and make sure sessions and queues are not lost.
- Rollout to production — we enable one segment at a time with an observation window, keep the previous configuration version for quick rollback, and monitor the state tables of neighboring devices.
- Handover of regulations — we provide the addressing scheme, configurations, manual failover scenarios, and the on-call procedure for node split-brain, so the operations team can manage the cluster without us.
Configurations, schemes, and failover regulations
The engineering part of the service ends not with service launch, but with a package of documents by which the cluster can be managed without us.
We hand over to the customer Keepalived configurations, the network scheme, and regulations — so the on-call shift understands what is happening, and a new engineer joins the project in hours, not weeks.
- Cluster configuration: files for both nodes with "primary/standby" roles, virtual router identifier, priorities, and advertisement intervals — with comments explaining why exactly these values are set.
- Network scheme: addresses of nodes and the virtual address, subnets, interface names, failure points, and a priority table — on a single sheet, not assembled piecemeal.
- Service liveness checks: scripts, startup parameters, trigger thresholds, and behavior during node degradation and complete failure.
- Failover regulation: normal scenario, emergency scenario, manual failover, and returning the role back — indicating who makes the decision.
- On-call instructions: step-by-step actions when the virtual address is lost, clearing the state, collecting logs, and criteria for when to call us.
- Monitoring setup: node state metrics, role-change alerts, and external service availability checks — so failure is seen before users call.
- Acceptance test protocol: failure scenarios that we reproduced and recovery time for each — as proof that failover works.
A separate item is the change checklist: how to edit configuration so as not to take both nodes down with a single action.
Case study: failover without traffic loss in production
The service's production environment ran on two nodes: the entry point lived on one server, and the second stood as a hot standby.
When the active node failed, the virtual address was raised manually — the on-call engineer received an alert, connected to the server, and reconfigured the address while clients saw errors.
Baseline: measurements before implementation
First we recorded the initial state — analyzed incident statistics and reproduced the failure on a test bench with production settings. We measured three values: failover time, share of lost connections, and number of manual interventions.
The test bench produced the same figures as production: 8–14 minutes to fail over, up to 6% dropped connections, 4–6 night calls to the engineer per month.
What was changed and what was achieved
Keepalived was installed on both nodes in VRRP mode: the virtual address is assigned to the active node, and the standby monitors advertisements and takes the address when they disappear.
Priorities were set so that return after recovery is controllable — without traffic flapping back and forth. Separately, we added an application health check: if the service is alive but not responding, the address moves without waiting for hardware failure.
During acceptance tests, failover completed within 1–2 seconds, the share of lost connections dropped to fractions of a percent, and manual interventions during the shift disappeared.
In production, we first ran the failure of each node separately, then the failure of both with return to service.
How Keepalived differs from a cluster manager
Keepalived is a VRRP protocol daemon: it holds a virtual address on the active node and moves it to the standby when the primary stops responding. Availability checks are configured for a specific service — checking a TCP port, HTTP response, or process presence.
As long as the check passes, the address remains on the current node; when it stops passing, the address goes to the standby node, and clients continue working via the same address without editing configs.
A cluster manager works at a different level. It manages resources: starts and stops services in a specified order, mounts shared disks, monitors quorum, and isolates a node that has lost connection with the others. Address redundancy is just one of the resources for it, not the main task.
Hence the selection criterion. If the application can already replicate and starts on its own, and only one thing is critical — that traffic goes to a live node — address redundancy and availability checks are enough.
If a managed lifecycle is needed — bring up the database, wait for synchronization, mount the shared disk, and only then open the address — a cluster manager is indispensable.
In practice, they are not opposed. The VRRP daemon is responsible for the address and traffic distribution, the cluster manager for resources; together they provide fast traffic failover and the correct order of starting services after failover.
Therefore, at the tool selection stage, we look at the application itself: how many states it has, whether quorum is needed, whether there is a shared disk, and whether a strict startup order is mandatory.
The answers determine whether we can get by with a single VRRP daemon or need to assemble a full cluster with resource management.
Availability checks, delays, and network split
A service availability check in Keepalived is not pinging an address, but checking the service itself: a local script connects to its own port or sends a test request and decides from the response whether the node is alive.
As long as the check passes, the node holds the address; on failure, it releases it and gives it to the partner. The trigger threshold and interval are selected so that a single application hiccup does not cause failover, otherwise the address will "jump" for no reason.
Two active nodes appear where service messages are lost in one direction: each considers the partner dead and takes the address for itself.
Protection is priorities, mandatory confirmation of connectivity before taking the address, and network separation: service exchange goes over a separate segment or via unicast delivery, not broadcast through production traffic.
What happens to already open connections during failover?
Connection state lives on the node and does not move with the address. The client gets a short failure and repeats the request, so failover is designed where retries are safe: idempotent operations, a load balancer with a shared state table. For long sessions, user re-entry is described in advance.
How to guarantee no two active nodes?
Priorities alone are not enough: a node with a higher weight must also make sure that the partner is really gone. That is why startup delay after launch, connectivity check, and a separate channel for service messages are added. In separated segments, this is the only way to distinguish loss of connectivity from node failure.
How quickly does a node return to service?
The node comes up on its own, but it does not take the address immediately — first the checks must stabilize, then a pause is maintained so the service does not flap.
Return by priority can be postponed to a maintenance window or disabled: then the role remains with the current active node, and the recovered node works as standby.
Let's discuss the task: implementing Keepalived for your environment
The first conversation is a review of your environment, not a presentation. We find out how many nodes participate in the fault-tolerant pair, which services hold the virtual address, how traffic is distributed, and whether there is an upstream load balancer.
Separately, we discuss change windows: whether nodes can be restarted during the day, whether there is access to network management and logs.
For the assessment to be substantive, we need inputs from you: network diagram, list of services behind the virtual address, failover time requirements, and maintenance mode. If there is no documentation, we will assemble the picture on a call with your engineers.
After the review, we send:
- an environment diagram with marked failure points and the place where Keepalived is installed;
- a stage-by-stage implementation plan, with work windows and rollback order;
- scope of work: VRRP setup, service availability checks, failover scripts;
- a list of configuration changes and what will be required from your side;
- test scenarios: how we will verify that the node actually takes over the address;
- roles and approval points during the work;
- stage-by-stage estimate — effort and dependencies, without binding to calendar dates.
Describe the task in two paragraphs — we will come back with an estimate and an implementation plan for your environment.