kubeadm cluster setup: where fault tolerance breaks
kubeadm setup according to a quick guide looks almost identical: kubeadm init on one machine, worker nodes join via kubeadm join, kubectl get nodes shows a neat green list. Pods roll out, services respond, the cluster looks fault-tolerant.
But the entire control plane is assembled in a single instance, and the safety margin of such a configuration is zero.
A single control node holds kube-apiserver, kube-scheduler, kube-controller-manager, and etcd all at once — everything in one instance.
If the machine or network goes down, the API server is unavailable: kubectl and deployment tools lose access, new pods are not scheduled, crashed ones are not restarted, service endpoints and certificate issuance freeze. The cluster is still alive but stops responding to changes.
For the first minutes the failure is masked: containers on worker nodes keep running, traffic flows, alerts are silent. So the outage is noticed later — when service downtime has already begun, and recovery has to be done manually, without a fresh state snapshot.
A separate trap is the etcd quorum: with one member, loss of the storage makes the cluster unavailable for changes, and restoring from a snapshot rolls back some recent operations.
Added to this are certificates with a limited validity period, the absence of a virtual address in front of control nodes, and rotation that is remembered only at the moment of failure.
We explain how this is addressed: multiple control nodes, a resilient quorum, a virtual address, placement across different machines, and a certificate rotation procedure.
What a fault-tolerant cluster provides: downtime and recovery time
Fault tolerance starts with the quorum of control nodes. Kubernetes keeps cluster state in a distributed storage, and writes succeed as long as more than half of the control nodes are alive.
Three nodes survive the loss of one without stopping: quorum is maintained, the API continues to accept commands, and the returned node catches up on the log and rejoins automatically.
On a single node, the same failure means an unavailable API and manual recovery from a backup — from tens of minutes to hours, with the risk of losing some recent changes.
For workloads the mechanics are different, and failover works here rather than recovery.
The scheduler notices the loss of a node and distributes its pods across the remaining ones, readiness checks remove unhealthy instances from load balancing, and controllers restore the required number of replicas without human intervention.
A service survives node loss within tens of seconds, whereas manual container migration and configuration edits would take hours of on-call time.
Which class of failures goes away
Previously, node failure meant calling the on-call engineer, analyzing logs, and manual restart — that is, recovery time depended on when the engineer noticed the problem.
In a fault-tolerant design this scenario disappears as a class: node crash, process hang, instance eviction — all these are routine events that the cluster handles itself.
Control nodes are spread across availability zones, so the failure of an entire zone does not deprive us of quorum.
We verify the effect using three metrics: service recovery time after node loss, the share of automatic recoveries, and the number of manual on-call operations.
In a configuration built according to this design, the last metric tends to zero — and with it goes the main cause of nighttime downtime.
Control node topologies: comparison of deployment options
Cluster topology determines what happens to workloads when one of the control nodes fails. With a single control node, failure means API unavailability: new pods are not scheduled, changes are not applied, and recovery requires manual intervention.
Three nodes remove this — quorum is maintained when one is lost, and kubeadm deployment continues to work without service interruption.
The second parameter is where the external state storage lives. If it is placed on the control nodes themselves, the design is simpler, but node degradation drags down the quorum.
External storage decouples the roles: nodes become interchangeable, and state survives their restart and replacement. The same choice stands between own hardware and rented — it affects network availability and which settings we can change at all.
| Option | When it fits | Limitations |
|---|---|---|
| Single control node | Development stands, pilots, internal services | Single point of failure, downtime during failure and updates |
| Three nodes, state storage on them | Small production where design simplicity matters | Quorum is sensitive to disk latency, scale is limited |
| Three nodes with external state storage | Production with continuity requirements | Requires a separate fault-tolerant site for storage |
| Own hardware | Data, network, and procedures under our control | Requires spare equipment and on-call duty |
| Rented nodes | Quick start without own infrastructure | Dependence on provider network and policies, some settings are closed |
At the planning stage, we check how the chosen topology coexists with your networks, storage systems, and maintenance window requirements — and only after that do we finalize the deployment design.
kubeadm setup: work stages from audit to handover
The order of work is not arbitrary: each stage relies on the recorded result of the previous one. A skipped audit surfaces already in production — the network or storage has to be redone after assembly, disassembling the cluster again. Below is the sequence by which engineers carry out kubeadm from the first meeting to handover.
-
Infrastructure audit. We examine nodes, disk subsystem, connectivity between racks and zones, address and permission restrictions. The output is a map: what is already ready for a fault-tolerant design, and what we change before starting work.
-
Design. We fix the number of control and worker nodes, placement of state storage, load balancing of traffic to the control interface, address ranges for pods and services. We agree on the decision in writing so as not to rebuild it along the way.
-
Node preparation. We bring machines to a uniform state: disk partitioning, kernel settings, time synchronization, network rules, one version of the container runtime. Inconsistency here is a typical reason why an assembled cluster behaves unstable.
-
Cluster assembly. We deploy the control plane in a fault-tolerant configuration, connect worker nodes, install the network layer, storage, and system services. All steps are reproducible: the same actions are repeated when adding a node.
-
Fault tolerance testing. We shut down control nodes one by one, then a worker — we check whether quorum holds and workloads migrate. Separately, we test node loss together with disk and restart of the entire control plane.
-
Handover to operations. We provide access, backup rules, the procedure for adding and removing nodes, updates, and failure analysis. Plus team training so that routine operations are performed without us.
Documentation, diagrams, and procedures at project output
After handover, the cluster does not live on its own: six months later, the value is not the set of machines but the recorded decisions.
Therefore we hand over a package of artifacts, according to which an engineer who did not participate in the assembly can bring up the stand from scratch and introduce changes without deciphering someone else's history in the terminal.
Configurations, diagrams, and procedures are stored in a single repository and versioned together — documentation does not diverge from the actual state of the cluster.
- Configuration files: kubeadm-config, control plane manifests, kubelet and container runtime parameters, charts for Ingress, DNS, and monitoring — the stand is deployed reproducibly on a clean environment.
- Network and addressing diagram: pod and service network ranges, API request balancing, interfaces with external networks, and routing between zones.
- Fault tolerance diagram: number of control plane nodes, etcd quorum, placement of state backups, cluster behavior upon zone loss.
- Access policies: RBAC roles and bindings, service accounts, namespace isolation, network policies, and workload admission rules.
- Update procedure: order of node updates, checks before taking out of service, rollback criteria, and retention depth of previous image versions.
- Recovery procedure and cluster documentation: scenarios for loss of a node and state storage, target recovery time, cluster map, alert settings, and a log of known limitations.
Case study: three control nodes, failure of one without downtime
Before the rebuild, the Kubernetes cluster was held on one control node and two worker machines assembled manually: the configuration lived in the engineer's head, and etcd backups were taken irregularly.
A failure of this node stopped the API — it was impossible to roll out a release or view pod status, and recovery from a snapshot took 40–50 minutes: bring up the node, restore certificates, verify addresses, wait for kubelet to see the worker machines again.
We rebuilt the control plane via kubeadm: three control nodes behind a load balancer with a common virtual address, etcd on all three — quorum holds as long as two are alive.
The installation is described declaratively, so a new node is brought up by the same scenario, without 'settings that only one person remembers.' Scheduled updates proceed node by node: pods are evicted to free capacity, the node is taken out of service, updated, and returned.
Metrics before and after
- Control node failure: before — up to 50 minutes of manual recovery and complete API unavailability; after — failover to a standby node in less than a minute, services continue to respond.
- Manual operations during failure: before 12–15 checklist steps; after zero — quorum and virtual address recover on their own.
- Scheduled updates without service interruption: before about half, the rest in a downtime window; after 100%, pod eviction fits into the normal load mode.
- Adding a new node to the control plane: from several hours to tens of minutes using a single reproducible scenario.
During operation, control nodes failed twice, and both times no on-call engineer intervention was required — the cluster handled the scenario itself.
What to do if the cluster is already assembled manually?
First, an inventory: how many control nodes, where etcd lives, how certificates and kubeconfig are issued, whether there is a load balancer in front of the API server.
A manually assembled cluster is almost always single-point: one control node, etcd without quorum, a set of configuration edits that only one engineer knows about. It is this state that determines which of the two paths is realistic.
The first path is bringing it to a fault-tolerant design: we add control nodes, switch etcd to quorum mode, place a load balancer in front of the API, reissue certificates for new addresses. We do not touch worker nodes, and workloads continue to run.
There is one limitation: if many manual edits have accumulated, the upgrade inherits the old discrepancies along with their causes.
The second path is to assemble the target cluster anew and migrate workloads. The new control plane is brought up in parallel, manifests, persistent volumes, and secrets are transferred to it, and traffic is switched pod by pod.
There is no downtime: switching happens on a live service, and rollback is returning traffic to the old control plane while it is still in place.
We make the choice based on the state of etcd, the volume of manual edits, and the criticality of data in persistent volumes. Upgrading requires less intervention; rebuilding yields a configuration reproducible from code and removes accumulated discrepancies at once.
Common technical questions: network, updates, node failure
Most questions at project defense concern not installation but operations: how nodes see each other, what happens when a machine is lost, and in what order to update the cluster. Below is an analysis of typical concerns.
How to ensure connectivity between nodes?
The control plane and worker nodes exchange service traffic over a fixed set of ports. If filtering or NAT is present between machines, the cluster assembles partially: nodes hang in NotReady, and etcd loses quorum.
We describe and verify the allowed traffic matrix before commissioning, not after the first failure.
In what order should the cluster be updated?
We go node by node: first control nodes, then workers, with workload eviction before restart. We upgrade minor versions sequentially, without skipping — otherwise component compatibility and rollback become unpredictable.
Before each step — an etcd snapshot; after — a check that all nodes have returned to Ready and workloads have redistributed.
What happens when a node fails or the network splits?
The loss of a worker node does not bring down the cluster: workloads migrate to the remaining ones if requests and limits are set and eviction is enabled.
More dangerous is a network partition between control nodes — without quorum, etcd stops accepting writes, and the cluster operates read-only. Therefore the number of control nodes is odd, and some are placed in a separate zone.
How is state backup organized?
Cluster state is etcd plus manifests and secrets. We take snapshots on a schedule and store them outside the cluster, and we rehearse recovery quarterly on a separate control plane: otherwise it turns out that a snapshot exists but the cluster cannot be brought up from it.
Let's discuss your task: we'll put together a plan and estimate
So that the implementation estimate is based on facts rather than guesses, we need a minimum of input. Describe them in free form — we will consolidate this into a work plan and come back with the composition of stages, rollout order, and a list of risks.
The first conversation is technical: we clarify the design, boundaries, and assumptions, without dozens-of-pages questionnaires.
- Cluster composition. How many control and worker nodes are planned, one site or multiple zones, whether a load balancer is needed in front of the control plane.
- Availability requirements. Acceptable downtime, behavior during node or zone failure, whether planned node replacement without service interruption is needed.
- Environment. Own servers or cloud, what networks and storages already exist, what security policies and access requirements must be met.
- Workload profile. How many services are migrating, whether there is state, what requirements for disk volumes and backups.
- Timelines and staging. By what date the working control plane is needed, whether there are windows for work, whether we assemble from scratch or migrate an existing cluster.
- Operations after handover. Who is responsible for the cluster afterward and whether knowledge transfer to the customer's team is needed.
Describe the task — we will come back with an estimate and a plan showing what we do at each stage and how we verify the result.