Kubernetes cluster setup with Kubespray

Building a cluster by hand: what breaks and how Kubespray helps

Manual cluster assembly starts the same way everywhere: bring up the control plane, add worker nodes, install the network plugin, configure storage.

By the third environment the states have diverged — each step was performed from memory, in a different order and with changes made along the way.

The first divergence between environment configurations usually surfaces during an incident: in the test environment the behavior reproduces, in production it does not, and the cause is a single forgotten line in the parameters.

From there the difference only accumulates: a different order of operations, a different container runtime version, different scheduler parameters, and a different set of extensions.

Each such difference has no author and no edit history — diagnosis takes hours, and after the fix no one can guarantee a new one won't appear tomorrow.

Configuring a Kubernetes cluster with Kubespray is built on a different principle: the state is described declaratively — by a node inventory and a set of variables.

One description is applied to the test, pre-production and production environments, while the differences that are genuinely needed are moved into an explicit layer on top of the shared part. The cluster becomes the result of applying a description that can be verified and repeated.

Hence the predictability: adding a node, upgrading, changing network parameters all proceed the same way across every environment. Rollback also becomes a procedure — restore the previous description and apply it. Below we break down how this works and where to begin.

Environment reproducibility: what a declarative cluster description gives you

A declarative cluster description is a set of files with node roles, networks, storage and add-ons.

Kubespray reads that inventory and brings the environment to the described state, so the test site and production are assembled from the same variables — divergences between environments disappear as a class.

A new node is brought in with a single operation: the host is added to the inventory, a run is started, and the node reaches readiness to take on workload.

Previously that procedure stretched over a day of manual steps and ended with a checklist reconciliation; now the first node and the hundredth go through identically.

Cluster recovery time is determined not by the on-call engineer's memory but by the files in the repository.

If the environment is destroyed, it is rebuilt from the description and you get the same topology, the same service addresses and the same policies — with no archaeology on leftover configurations.

Deployment automation makes the state reproducible: a change goes through review like a code edit, and a rollback is reverting a file to its previous version. Drift accumulated through manual configuration stops occurring.

Differences between environments are moved into variables: the number of nodes, machine classes and the set of add-ons change in a targeted way, while the shared part of the description stays unified. Cluster upgrades run through the same run as the initial build.

Kubespray, manual configuration and a managed cluster: comparison

The approaches differ not in their set of components but in where the boundary of responsibility runs. With Kubespray, the desired cluster state is described declaratively, and replacing or adding a node becomes a repeatable operation.

With manual assembly everything rests on your instruction — its quality is exactly what determines whether the configuration will reproduce on another environment.

In a managed cluster the control plane stays with the provider: you don't administer it, but you also don't influence the pace of updates or the available settings. Below are the conditions under which each option is appropriate, and what you pay for it.

The difference shows up in operations, not in installation. In the first two options, updating and restoring a node is your procedure and your responsibility for keeping it current; in the third it's part of the provider's rules. Hence the choice: it comes down to where you want to keep that procedure.

Implementation sequence: from environment assessment to handover

We agree on the implementation sequence before the first configuration command: until the design is locked down, any topology change costs a repeat run and downtime.

We start with an infrastructure assessment and finish with acceptance — with measurements and a document by which the on-call shift can maintain the cluster without us.

1. Infrastructure assessment. Inventory of nodes, network zones, storage and load balancers that the cluster depends on. We record what we reuse, what we bring up from scratch, and which addresses are already taken.
2. Cluster design and deployment plan. Topology of control plane and worker nodes, grouping, addressing, failure points. We agree on it before the first run: edits at this stage cost minutes, afterwards they cost rework.
3. Cluster configuration. Kubespray with parameters for the specific environment: custom inventory values, separate groups for high-load services, components from the current stable branch.
4. Failure testing. We shut down a worker node and a control plane node one at a time and check whether workloads migrate and service availability holds. We test storage and cross-zone networking separately.
5. Upgrade run. We upgrade the cluster on the test environment, including a rollback to the previous state, and drive it to a repeatable run with no manual edits on live nodes.
6. Work acceptance. Cluster health metrics, operations regulations, issued access and a walkthrough of typical failures with the on-call shift. We hand over the documentation and stay in touch for questions beyond its scope.

Build artifacts: configuration, roles, regulations

We hand over the cluster not as «it works and that's fine» but as a package that any engineer on the customer's team can use to maintain it.

Everything lives in your repository under version control: edits go through review, change history can be reconstructed, and rollback comes down to restoring the previous state.

• Infrastructure diagram — network zones, control plane and worker nodes, traffic entry points, external storage and queues; with a separate note on what remains outside the cluster.
• Cluster documentation — node inventory, group variable values, installed add-ons and API extensions, a list of differences from the standard Kubespray distribution with a rationale for each.
• Access policies — roles and permission bindings, namespace separation for teams and service accounts, network policy rules, and the procedure for granting and revoking access.
• Upgrade regulations — the sequence for upgrading the cluster and add-ons, checks before draining a node, return points and stop criteria.
• Roles and installation parameters — variables for the development, acceptance and production environments, so that redeploying an environment takes hours rather than weeks.
• Instructions for routine operations — adding a node, replacing a disk, scheduled certificate rotation, restoring from backup, and responding to monitoring alerts.
• Support agreement — response times for failures, the procedure for filing changes, and where incidents and post-mortems are recorded.

The result is an environment that doesn't need to be restored from one engineer's memory: upgrading, editing or handing the cluster over to another team follows a documented scenario.

Case study: Kubespray on 40 nodes — what changed in a quarter

The customer came with a 40-node Kubernetes cluster assembled by hand: configurations were edited directly on live machines, and divergences between the test and production environments surfaced during incidents.

Restoring a node after a failure took hours and rested on one engineer's memory — the state was described in personal notes rather than in a repository.

What we did

We moved the cluster description into a Kubespray inventory: component versions, network parameters, node settings and upgrade policies became part of the repository.

The cluster migration was carried out without downtime — new nodes were brought up by role, workloads were moved onto them, and the old ones were taken out of traffic. The environments were aligned: one inventory and different variable sets for test and production.

What changed in a quarter

The time to build a cluster from scratch dropped from a day and a half to a little over two hours — most of it is now taken up not by manual configuration but by installing updates on the nodes.

The number of divergences between environments fell from several dozen to zero: a deviation is visible in the diff report before the run even starts. Restoring a node takes minutes — the role is reinstalled from the repository, not from the engineer's memory.

Cluster reliability became verifiable: upgrades go to test first, then to production, with the ability to roll back the entire set at once. We carry this order into other projects — state in code, changes only through review, recovery as a routine operation.

How Kubespray differs from manual cluster assembly

Kubespray is a set of Ansible roles that describes a cluster declaratively: the node composition in the inventory, component versions, network and storage parameters — in group variables.

Manual assembly works the other way around, from a sequence of commands that lives in an engineer's head or in a script written for a specific case.

The difference shows up not on the first installation but on the third or fourth operation.

Adding a node, replacing a failed control plane node, upgrading the component branch — with a manual approach the order of steps is reconstructed from memory every time, and the cluster gradually drifts from its original state.

An existing cluster is no obstacle: a run brings running nodes to the described state without recreating them. First a backup of the state is taken, then changes proceed one node at a time with workload draining — rollback remains possible at every step.

The choice of build tool comes down to the number of environments and the lifespan of the cluster.

A single cluster for testing a hypothesis is faster to assemble by hand; several environments, reproducible setups and handing operations over to another team are arguments in favor of a declarative description.

Concerns about compatibility and cluster upgrades

A cluster built with Kubespray is a node inventory and a set of roles, not a black box: Kubernetes upgrades here are reproducible and rolled back by the same run. Below are the three concerns most often raised at the acceptance stage.

What about incompatible add-on versions?

We lock down the compatibility matrix for Kubernetes, the network layer, the secrets store and access controllers before work begins.

Before an upgrade we run a check on the test environment with the same set of workloads: incompatible add-ons are upgraded together with the cluster, not after.

How is the cluster handed over, and what does support include?

We hand over the inventory, roles, variable values and environment descriptions — everything lives in the repository from which the cluster is built from scratch.

Cluster support is built on upgrade regulations and monitoring: you can see which nodes and add-ons have changed, so a new version is installed predictably.

Let's discuss your task: what we'll clarify and what we'll send after the assessment

So that the estimate rests on facts, we'll start with an assessment of your current infrastructure: the node layout, availability requirements and what already runs in production. That's enough to understand the scope, bottlenecks and risks of deploying a cluster with Kubespray.

We'll clarify on a call:

• where the cluster will be hosted — physical servers, virtual machines or cloud, and how we split nodes into control plane and worker;
• what availability mode is needed: how many control plane nodes we keep and whether a load balancer is needed in front of the control plane endpoint;
• networks: which address ranges we allocate for workloads and services, and whether there are restrictions on routing between segments;
• storage: which disk classes are available and whether a driver is needed for workloads with persistent data;
• access: how roles are granted, where secrets are stored, and whether there are requirements for a change log;
• upgrades: who upgrades the cluster and how often, and which rollback plan you consider acceptable.

After discussing the task we send an estimate and an implementation plan — with the assumptions they depend on and checkpoints at every stage. Along with them, technical documentation and a pre-start checklist:

• the inventory configuration and node roles in the form you keep in your repository;
• the network and addressing diagram with reserved ranges;
• the upgrade and rollback policy, and the backup regulations for the control plane;
• a pre-start checklist: access, maintenance windows, and the people responsible on your side.

Describe your task — we'll come back with an estimate and a plan.