Why etcd service discovery degrades under load
An etcd cluster holds a service map: who listens where and which node is still alive. While everything is stable, etcd service discovery returns current addresses almost instantly, and applications communicate directly, without an intermediary.
Under load, it is not the service itself that breaks but the registry: the more clients and keys, the longer the write, and the cluster begins to lag behind the stream of updates.
If one or two nodes go down, quorum loss occurs: the cluster stops accepting writes and remains read-only. Then the loss of the cluster leader triggers an election, and during this window services cannot renew the key lease. Some instances re-register, some do not, and the registry diverges from the actual state.
Stale registry records are more dangerous than an explicit failure. A key with an expired lease continues to be served to consumers, traffic goes to a stopped instance, and instead of load the load balancer accumulates connection errors.
Discovery stalls: the time from address publication to the first successful request grows from seconds to tens of seconds, and some connections are not established at all.
Emergency response is usually manual: an engineer enters the address into manual service address lists, restarts consumers, and hopes they haven't forgotten half of them. This works until the next release, and sometimes until the next failed node. Next, we break down how this is addressed: quorum, leases, and observability instead of eyeballing.
Cluster resilience: quorum, leader election, reaction time
Quorum is calculated from the full membership, not from the number of live machines: a node without communication with the majority keeps running but does not confirm writes and cannot become leader until it catches up with the log.
Five nodes survive the failure of two, while four survive exactly the same single failure as three, so membership is chosen odd and spread across different failure zones.
Writes go through the elected leader, while reads are served by any node with an up-to-date log. When the leader is lost, the remaining nodes hold an election: the cluster returns to writing within seconds, and during this time new writes wait in a queue.
Election timeouts and heartbeat frequency determine this reaction — overly aggressive values cause re-elections on short-lived network failures.
For service discovery, the main thing changes: instead of periodic polling, clients hold a long-lived subscription to changes.
An instance registers a key with a lease and renews it while alive; if it crashes, the lease expires, the event goes to subscribers, and the load balancer removes traffic from the dead address.
Discovery time shrinks from tens of seconds to units — previously, some requests went nowhere for the entire polling window.
Registry fault tolerance is separate work: state backups with recovery verification, quotas on database size, scheduled defragmentation, nodes in different failure zones so that losing a zone does not take away quorum.
We build such a cluster on etcd with TLS between nodes and clients and alerts for quorum loss and leader change.
In operation, this is visible as a class of failures that has stopped being an emergency: the failure of a node or an entire zone does not create a window of service unavailability.
Service discovery modes: client-side, server-side, hybrid
In an etcd cluster, service discovery is assembled in three ways, and they solve different tasks: the difference is where the instance selection logic lives and what happens when a registry node or the service itself fails.
The choice is dictated not by fashion but by the acceptable failure zone, operational cost, and compatibility with the current environment — certificates, metrics collection, rollout order.
| Scheme | When it fits | Limitations |
|---|---|---|
| Client-side discovery | the service itself reads registry keys and chooses an instance; there are no intermediate nodes in the path | retry and balancing logic is duplicated in every service; changing rules requires rebuilding clients |
| Server-side discovery | traffic goes through a load balancer; the application knows nothing about the registry | an extra node in the request path is another point of failure; a separate list update loop is needed |
| Hybrid discovery scheme | the load balancer returns a list of addresses, and the client makes the final choice | harder to observe: metrics are needed on both the load balancer and service sides |
Both extreme schemes fit onto the existing cluster with TLS without restructuring: certificates are issued by the same authority, access to keys is separated by roles, and subscription to changes instead of periodic polling gives list update latency within a fraction of a second.
Balancing between services remains manageable — instance selection rules are described declaratively and are not smeared across application code.
During implementation, we keep a way back: switching between schemes is done by configuration, without rewriting services. This makes it possible to compare how the options behave on real traffic and roll back if the chosen scheme produces extra failures during node degradation.
Cluster deployment: audit, TLS, failure testing
Deploying an etcd cluster is work in which the order of steps matters more than speed: an error in node placement or certificates does not manifest immediately but at the moment when one node fails and quorum collapses. Below is the sequence by which we guide a project from audit to handover.
- Service inventory. We compile a list of what will register and be discovered through the cluster: keys, record lifetimes, who writes and who reads, which services depend directly on the address map. Without this list, it is impossible to calculate load or configure quotas.
- Node placement scheme. We determine the number of quorum nodes, distribute them across different failure zones, and check latency between sites. We keep quorum odd so that losing one zone does not stop writes.
- Node and storage preparation. Separate disks for the log and data, time synchronization, resource limits, and predictable behavior when the disk fills up — the most common cause of cluster degradation.
- Traffic encryption. We configure certificates for communication between nodes and for clients, enable mutual authentication, and describe the rotation procedure. We separately verify that an old certificate does not remain valid after revocation.
- Monitoring and alerts. We expose quorum metrics, write latency, and leader state, and set alerts for quorum loss and response time growth so that the on-call engineer notices a failure, not the user.
- Failure scenario testing. We plan node shutdowns, take an entire zone offline, break communication between sites, restart the cluster — and record how long it takes to return to a working state.
- Handover and procedures. We provide documentation, access, instructions for updates and taking nodes out for maintenance, and review typical incidents with the on-call shift.
Completed failure tests are what distinguish a working cluster from a configured one: from then on, maintenance is predictable because the system's behavior during failures is already known.
Configurations, diagrams, and procedures: what remains in your environment
After handover, the entire environment remains with you — not a "contractor's black box" but a set of artifacts that your team can use to deploy a stand from scratch, pass an audit, or take over on-call duty without calling us.
Every non-standard value in the configuration has a written justification: why the parameter is set this way, what will break if it is changed, and how to roll back to the previous state.
- Cluster configuration set: quorum parameters, storage quotas, snapshot schedule, and compaction rules — with comments on non-standard values and references to why they appeared.
- Cluster diagram: node topology, roles, distribution across availability zones, failure points, and system behavior when each of them is lost.
- Access policies: certificate chain, validity periods, rotation procedure, separation of read and write permissions by service, and on-call accounts.
- On-call procedure: what counts as an incident, which metrics and thresholds to watch, whom to escalate to and within what timeframe, when to restore from a snapshot.
- Operations documentation: step-by-step instructions for planned operations — adding a node, replacing a certificate, changing quotas, recovery after a failure.
- Checks and acceptance checklist: a set of tests confirming operability after any change, plus a drill report on the test stand.
We conduct handover as a walkthrough: we go through the items together with your engineers and demonstrate rollback and emergency recovery scenarios live.
After that, the environment runs without constant support — changes are made according to documented procedures, not from the memory of whoever configured it.
Case: registry of 40 services without manual edits
Initially, the addresses of forty services lived in static configurations and partly in on-call shift notes.
After an instance was moved to another node, the calling side continued to go to the old address: the service returned an error for 15–40 minutes until an engineer found the stale record and reissued the configuration.
The share of manual changes in routing was a full 100%, and the on-call shift received 12–14 calls per month solely because of incorrect addresses.
What changed
We set up a fault-tolerant registry on etcd — three nodes with quorum, mutual TLS authentication, and write permissions separated by path.
Each service received a small agent: on startup it publishes its address with a short lease and renews it; on shutdown it removes the record itself.
If an instance crashes and does not renew the lease, the key expires, and the calling side stops seeing the stale address within seconds.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Reaction to topology change | 15–40 minutes | 3–5 seconds |
| Share of manual route edits | 100% | 0 |
| Failures due to stale addresses | 6–8 per month | 0 |
| On-call calls about routes | 12–14 per month | 1–2, only when a registry node goes down |
| Recovery after node failure | manual configuration review | automatically; quorum preserves reads and writes |
The class of failures "the client knows an address that no longer exists" disappeared completely — when rolling back a release, the previous version's address remains published, so returning to it requires no edits.
The on-call workload shifted: instead of manually synchronizing address lists, engineers handle only liveness events of the registry itself.
What to do if the service registry is already deployed on another system?
Migration to etcd service discovery is not done with a single switch: first, two environments run in parallel. New instances register in both the old and the new registry, while consumers read from their usual source.
The coexistence of two environments does not break production services — discrepancies in output are visible before anyone is affected by them.
A small agent next to the service handles registration: it holds a short lease and updates state in both registries, so a crashed instance disappears from discovery within seconds rather than on a schedule.
Key names in the environments must match; otherwise, comparing output becomes manual reconciliation.
We secure access to etcd with mutual certificate verification and separate read and write roles so that a consumer cannot modify other services' keys.
The readiness criterion for a group of services is that the output of the two environments matches and the reaction to instance shutdown is the same; only after that do we switch readers in batches.
We keep the transition period until all consumers switch sources — usually several waves by service group.
Rolling back during this time costs one change to the read endpoint: the old environment remains alive and populated, no data recovery is needed, and we shut down its records and nodes last.
Observability is more important than speed: we monitor the share of successful resolutions, registry response latency, the number of records with expired leases, and counter divergence between environments.
A typical mistake is shutting down the old registry by calendar rather than by fact: as long as at least one consumer reads it, the environment must respond.
Common technical questions: fault tolerance and operations
Before work starts, the customer checks the same things: what happens if a node fails, how the registry recovers, and what is needed from the infrastructure side. Below are answers to the questions we review with the team before the first rollout.
What happens if cluster nodes fail?
The cluster is built with an odd number of nodes and quorum. Losing two out of five does not stop operation: the leader is re-elected in fractions of a second.
Losing the third switches the registry to read-only mode — services can see each other, but new ones do not register until the node returns.
How is recovery after a failure arranged?
The registry state is backed up on a schedule and stored outside the cluster. Recovery is performed to a clean node and takes minutes; we rehearse the procedure on a separate stand, not during an incident. After startup, services find each other without configuration edits.
Is there room for scaling?
What grows is not the number of nodes but the volume of keys: the registry stores service addresses, not their traffic. Tens of thousands of records are served by the same five-node membership.
Under high read load, we add observers — they do not participate in leader election and do not affect quorum.
What are the requirements for the environment and access?
Dedicated nodes with fast disks and time synchronization, predictable latency between them, a separate certificate authority for certificates and their automatic rotation. We grant access using short-lived certificates with roles — read, write, administer.
Let's discuss your task: registry design and implementation plan
To make the estimate concrete rather than "by eye," we need source data: a list of services and their dependencies, availability and recovery time requirements, the current network layout, space for the etcd cluster, and people on your side responsible for access and network rules.
In response, you receive a working document, not a letter with generalities: node placement scheme, draft configuration with TLS and permission separation, a set of metrics and alerts, and update and recovery procedures. Below is exactly what you get before work starts.
- Registry design: node topology, quorum size, failure zones, and cluster behavior when a node is lost.
- Estimate and work plan: stages, duration, control points, and risks at each step.
- Start procedure: what we need from you as input and which step we perform first.
- Team composition: infrastructure engineer, monitoring specialist, acceptance owner — roles and areas of responsibility.
- Operations procedure: backup, certificate rotation, updates, on-call duty.
- Acceptance criteria: checks the cluster passes before handover.
You can discuss the task in writing or in a short review session: what already exists, what does not work, and what the deadlines are. We will come back with an estimate and a plan that shows the scope and order of steps.