Why an etcd cluster loses quorum and takes down services
An etcd cluster stores configuration as a log of replicated records: a record is considered applied only after confirmation by a majority of nodes. Of five nodes, quorum requires three — the loss of a third puts the cluster into read-only mode, and then into complete failure.
A separate trap is configuration desync: a node with a slow disk lags behind the leader, accumulates a queue of changes, and drops out of voting, after which the remaining nodes begin electing a new leader in a loop.
From the outside, this looks like a configuration store failure with no single error in the application logs. Some services read keys from cache and continue running on stale values; some access directly and get a timeout.
Configuration desync spreads across service groups: some already see the new dependency address, others hold the old one, and traffic goes nowhere.
Then cascading unavailability kicks in. Service discovery, secret rotation, and launching new application instances depend on the store: while it is silent, replicas do not come up, routes do not update, and liveness checks start killing healthy instances. Quorum loss hits the entire environment at once, although the cause is local.
And almost always this is not a sudden disk failure but accumulated debt: all nodes in one failure zone, no control over record apply latency, backups that have never been tested by restore. We break down how this is closed — from topology and limits to node replacement procedure.
What a fault-tolerant etcd cluster provides: effect by metrics
A fault-tolerant etcd cluster is assembled from an odd number of nodes — usually three or five. Such composition survives the loss of one or two nodes: the remaining nodes retain quorum and continue serving configuration, so node degradation does not turn into a service outage. Below are five results visible in metrics after moving to such an environment.
Recovery time after node failure
A failed node is replaced by adding a new one to the cluster and syncing a state snapshot — without manually parsing configuration on the host.
Recovery time drops from hours of manual bring-up to tens of minutes, while configuration reads and writes remain available the whole time.
Eliminated failure class: configuration divergence between services
A single configuration in one store removes scattered files on hosts that caused environments to diverge and fail on parameter mismatch.
Revision history shows who changed a value and when, while rollback to a previous revision returns the service to a working state in seconds instead of searching for differences manually.
The remaining effects — no single point of failure when reading settings, identical configuration across all environments, and predictable speed of applying changes: an edit rolls out to all services in one pass, without walking hosts.
Configuration store deployment modes: comparison
Deployment mode determines two things: how nodes find each other and how many failures the cluster survives without losing configurations.
The difference between static node membership and dynamic discovery shows exactly at failure time — that is when it becomes clear whether the scheme was chosen correctly.
Single data center
Here static node membership is appropriate: addresses are specified in each member's configuration, the cluster assembles predictably, and debugging comes down to checking connectivity between nodes. An odd number of nodes provides reserve for one failure without stopping writes.
The scheme limitation is different: adding a node without editing configurations and restarting is impossible, and failure of the entire data center stops configuration writes entirely.
Multiple availability zones
Dynamic discovery is more necessary here: nodes find each other through an external registry or name service, and cluster membership changes without manual edits. The price is dependence on the network between zones and on the discovery source itself.
The higher the latency between zones, the longer write confirmation takes, so zones are placed with predictable connectivity.
| Mode | When it fits | Scheme limitations |
|---|---|---|
| Static membership, single data center | Small environment with a stable number of nodes | New node — configuration edits and restart; data center failure stops writes |
| Static membership across zones | Managed network between zones exists | Latency between zones increases write confirmation time |
| Dynamic discovery | Node membership changes often | External discovery source required; its unavailability hinders assembly |
Before choosing, we check latency between zones and how often the number of nodes changes — this determines the scheme and the order of replacing members.
Implementation stages: from audit to handover to operations
The order of work is arranged so each step relies on the previous data: first we record the actual state, then design, and only then change the production environment. Below is a typical route for an etcd cluster that stores application configuration and state.
Audit of current state
We look at what already works and which configurations must not be lost, and separately at current points of failure: a single node, replication without quorum, a backup that has never been tested.
- Configuration audit. Map of services that read and write state, key volume, change frequency, access rights. Output — bottlenecks and a list of configurations that will survive migration without edits.
- Scheme design. Number of nodes and quorum, placement across availability zones, backup procedure, certificate scheme, and access separation between applications.
- Node configuration. Deployment according to the designed scheme, configuration migration with key reconciliation, monitoring, alerts for quorum loss, and automatic restart.
Fault tolerance testing under load
- Fault tolerance testing. We shut down nodes one at a time, simulate leader loss and broken connectivity between zones: the cluster must retain quorum and continue serving reads and writes.
- Load runs. Profile close to production: write peaks, many simultaneous clients, long sessions. We measure response latency and new leader election time.
- Handover to operations. Recovery procedures, dashboards, access separation, drills with the customer's team, and support in the first days after cutover.
Handover package: configurations, diagrams, access policies
We assemble the package not "for a checkbox" but so that the on-call shift can bring up, expand, and restore the cluster without us. Below is exactly what goes to the customer along with the working environment.
Access policies and IAM roles
- Roles and bindings with separation of rights: reading configurations, writing, cluster administration — under separate service accounts.
- Procedure for issuing and rotating certificates with a process for revoking a compromised key.
- "Who can do what" matrix: mapping of applications, accounts, and allowed operations; writes to the production environment from test applications are prohibited by rules.
- Audit settings: which requests to the store are logged and how long the log is retained.
Maintenance and recovery procedures
- Configuration files of all nodes with comments — which cluster parameter is set and why exactly that way.
- Deployment diagrams: topology, quorum, order of adding and replacing a member, binding to sites.
- Maintenance procedure: scheduled health checks, update to the current branch, node removal without losing quorum.
- Recovery instructions: taking backups, validating their suitability, from-scratch bring-up scenario, and the point to which rollback is possible.
- Operations documentation: responsible parties, maintenance windows, on-call response time, and change log.
We hand over the package in a private repository and on a secure medium, with an accompanying description — what is in each folder and when this artifact needs to be updated.
Case study: restoring the store without losing configurations
After a disk failure, one of the etcd nodes failed, quorum broke, and applications stopped reading configurations.
Some services kept running on stale values, and recovery stretched to four hours: state was assembled from a daily backup and edits made directly on live nodes, reconciling keys manually. We break down what was changed in working with the store.
What it was before: manual edits and configuration divergence
Edits went directly into the live store through a client, bypassing the repository and review. Over several months, divergences accumulated: seven services read stale keys, and this was noticed only through an incident.
Backup was taken once a day and was never tested by restore, and the lost node was brought up manually by matching values from logs — on average, four hours of configuration write downtime.
What it became after: managed scheme and node switching
Now the state of the etcd cluster is described declaratively and lives in a repository next to Kubernetes manifests: changes go through review and are applied automatically.
Quorum is maintained on an odd number of nodes and survives the loss of one — removing a node for maintenance takes seconds, and applications do not notice the switch.
A snapshot is taken every fifteen minutes and once a week is verified by a full restore on a test environment.
Result: key divergence between nodes is zero, node recovery is twelve minutes instead of four hours, and switching is less than ten seconds instead of forty minutes of downtime.
What to do if the store is already deployed in one data center?
A working scheme with quorum in one data center is expanded without rebuilding the cluster. Adding a node is the joining of a new member to the existing quorum: the leader retains its role, and configuration writes continue continuously.
The new node first receives a snapshot and catches up with the change log, and only then becomes a voting member.
The order is as follows: nodes are introduced one at a time, and between steps we check log lag and read response. This way, topology change proceeds gradually — from three members to five in the same data center, and then some of them are moved to another site. Data remains the same; only the distribution of quorum across data centers changes.
Migration of the store to another data center follows the same principle: we bring up a member in the new location, let it catch up with the others, and only then remove the node in the old one.
At each step, the majority of votes remains alive, so no downtime window occurs — services continue to read and write configurations.
Before switching, we fix the plan: rollback points, fresh snapshots, acceptable lag, and order of node removal.
Separately, we monitor latency between sites — change agreement is sensitive to it, and a slow channel can turn a healthy cluster into one that constantly re-elects the leader.
After migration, we capture control metrics: write agreement time, leader stability, log volume. If values go beyond the boundaries, the prepared rollback to the previous layout is performed just as without downtime.
Customer technical concerns: node failures and updates
A node failure in a configuration cluster rarely looks like an instant crash: applications continue to read state while quorum is maintained, and the problem surfaces later — during planned hardware replacement or when trying to bring up an environment from a copy.
Below are three questions most often asked before work starts: behavior when a node drops out, procedure for changing membership, and suitability of backups.
What happens if one node drops out?
Quorum is the majority of members: in a three-node cluster, one is lost, two votes remain, and configuration writes continue. The leader is re-elected, and client applications see a short response delay, not an outage.
A two-node cluster provides no quorum at all — the loss of either one stops writes, so we convert even-numbered schemes to odd-numbered ones rather than "reinforcing" them by adding a second node.
How to change membership without downtime?
Nodes are introduced and removed one at a time, in strict sequence: first we enable the new member, wait until it catches up with state and confirms readiness to take load, and only then remove the old one. Leader role transfer is planned in a maintenance window.
Member addresses and roles are described in configuration as data — we do not edit live nodes by hand, otherwise membership diverges from the description.
How to make sure a backup is working?
A state snapshot is useless until it is restored: the copy is validated by deploying on an isolated test bench, reconciling completeness of keys and checksums with the production cluster.
Separately, we record recovery time — it determines how long it will take the environment to come up after an accident. We put backup validation into a regular procedure, and store the backups themselves outside the site where the cluster operates.
Let's discuss the task: we will return with an estimate and work plan
To make the estimate rely on facts, not industry averages, we need source data on your environment. Describe the task in free form — we will clarify what is missing and return with an estimate, work plan, and order of steps.
What we request for calculation:
- current etcd node placement scheme: how many machines, how they are distributed across availability zones, whether there are separate disks for the log;
- what currently manages application configurations and how they are delivered to nodes;
- load profile: number of keys, share of write versus read operations, peak periods;
- availability requirements — what downtime window for one node is acceptable, who initiates maintenance;
- how backup is organized and when restore from backup was last tested;
- restrictions on access, certificates, network segmentation, and change procedure;
- who on the customer side accepts the work and by what criteria.
Then the order of steps is as follows: analysis of the current scheme and bottlenecks, work plan with stages and control points, agreement on a change window, implementation with node failure testing, handover of configurations, diagrams, and maintenance procedure. Describe the task — we will return with a plan and timeline.