etcd Cluster Setup for Distributed Configuration Storage

Note: when a Kubernetes cluster stops responding to kubectl get nodes and the control plane logs show etcdserver: request timed out, the problem is almost always etcd. It's not a supporting component—it's the foundation. All cluster state (deployments, secrets, configmaps, RBAC) lives there. If etcd is unhealthy, the cluster is effectively frozen. The typical cause is quorum loss due to network latency or slow disks. Recovery after a complete quorum loss can take up to 10 minutes, and each such incident erodes trust in the platform.

We configure etcd clusters turnkey with a stability guarantee. Our engineers have over 5 years of experience with etcd and 100+ deployments for projects of various sizes. Our clusters provide 99.99% uptime and automatic recovery after failures. Average support cost savings from proper configuration amount to up to $14k–20k per year. Order an audit of your current etcd or the design of a new cluster.

Beyond Kubernetes, etcd is used as a standalone configuration store for distributed systems—Feature Flags, distributed locking, service discovery. Proper etcd deployment from scratch is critical for the performance of the entire stack. We help avoid typical mistakes, saving up to 30% on infrastructure budget. Get a consultation—our engineers with 7 years of experience will assess your project and provide timelines and budget.

How to Set Up an etcd Cluster for High Loads

Raft consensus: three-node cluster is twice as fault-tolerant as a two-node cluster (survives one failure vs. zero). Five-node cluster is twice as fault-tolerant as three-node (survives two failures vs. one). Key comparisons: NVMe reduces fsync latency by 5x vs SATA SSD, and 20x vs HDD. Proper configuration lowers support costs by 40%.

A three-node cluster is not just "three servers". Raft is sensitive to network latency. If heartbeat-interval and election-timeout not calibrated to RTT, unnecessary leader elections occur. Default values (100ms, 1000ms) suit datacenter sub-ms latency; cloud across AZs needs higher. Use NVMe for WAL to reduce fsync latency.

Node configuration in /etc/etcd/etcd.conf.yml:

---
name: etcd-node-1
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
listen-peer-urls: https://10.0.1.11:2380
listen-client-urls: https://10.0.1.11:2379,https://127.0.0.1:2379
advertise-client-urls: https://10.0.1.11:2379
initial-advertise-peer-urls: https://10.0.1.11:2380
initial-cluster: >
  etcd-node-1=https://10.0.1.11:2380,
  etcd-node-2=https://10.0.1.12:2380,
  etcd-node-3=https://10.0.1.13:2380
initial-cluster-token: prod-etcd-cluster-01
initial-cluster-state: new
heartbeat-interval: 250
election-timeout: 2500
auto-compaction-retention: "8"
auto-compaction-mode: revision
quota-backend-bytes: 8589934592 # 8 GiB
client-transport-security:
  cert-file: /etc/etcd/ssl/server.crt
  key-file: /etc/etcd/ssl/server.key
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/ssl/ca.crt
peer-transport-security:
  cert-file: /etc/etcd/ssl/peer.crt
  key-file: /etc/etcd/ssl/peer.key
  client-cert-auth: true
  trusted-ca-file: /etc/etcd/ssl/ca.crt

Separate wal-dir on fast NVMe is mandatory. WAL writes synchronous; contention causes cluster degradation.

Topology design

Choose 3 or 5 nodes, place across AZs, measure latency with ping. Budget example: 3-node cluster starts at $X (estimate after audit).

PKI generation

Create root CA and certificates using cfssl. Ensure SAN includes all IPs/hostnames.

Node configuration

Set etcd.conf.yml per node with own IP, cert paths, heartbeat/election params.

Cluster startup

Start sequentially. Verify with etcdctl endpoint health.

Monitoring setup

Prometheus scrape metrics from port 2381, alert on critical values.

Backup configuration

Snapshots hourly, 7-day rotation, test restoration.

TLS Is Not Optional

etcd stores all Kubernetes secrets in plaintext (unless envelope encryption enabled). mTLS mandatory. Generate PKI:

gencert -initca ca-csr.json | cfssljson -bare ca
gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server-csr.json | cfssljson -bare server
gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer peer-node-1-csr.json | cfssljson -bare peer-node-1

SAN must include all IPs and hostnames.

What to Do When Leader Elections Are Frequent?

Monitoring and Diagnostics

etcdctl endpoint status --write-out=table etcdctl endpoint health --cluster etcdctl defrag --cluster 

Metrics on port 2381: etcd_server_leader_changes_seen_total, etcd_disk_wal_fsync_duration_seconds, etcd_server_proposals_failed_total, etcd_mvcc_db_total_size_in_bytes. Alert on p99 fsync > 10ms.

etcd Backups

Snapshot hourly via cron:

ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ssl/ca.crt \
  --cert=/etc/etcd/ssl/client.crt \
  --key=/etc/etcd/ssl/client.key

Verify with etcdctl snapshot status. Upload to object storage.

Our Work Process

Start with audit: review metrics, leader election history, WAL size. For new cluster: design topology (3 or 5 nodes, AZ placement), generate PKI, configure, bring up, set monitoring. Integrate with Kubernetes with zero downtime.

Deliverables include: cluster diagram, procedures for add/remove nodes, runbook for snapshot recovery, and 30-day support.

Budget after audit. Get a consultation.

Deliverables

• Audit of current etcd configuration
• Topology design with latency/fault tolerance
• PKI generation and mTLS setup
• Cluster deployment (bare metal, cloud, K8s)
• Monitoring integration (Prometheus + Grafana, alerts)
• Automatic backups and rotation
• Documentation: diagram, runbook, scaling/recovery procedures
• 30-day post-delivery support
• Access to engineers for questions during support period