Azure AKS cluster: what breaks under growing load
An Azure AKS cluster behaves predictably exactly as long as the load stays within familiar limits.
At peak — a sale, a morning influx of users, batch processing — nodes hit the ceiling on CPU and memory, the scheduler cannot find room for new service replicas, and container restarts begin.
Formally the cluster is alive, all nodes are in Ready status, but the service responds many times slower. Some requests do not wait for a response, the user sees an error, and the on-call engineer receives an avalanche of identical alerts instead of one understandable cause.
The second bottleneck of the cluster is manual edits.
Someone adjusted resource limits through the console, someone manually added nodes at night, someone weakened readiness checks "to make it come up faster." All of this survives until the next deployment: the actual state diverges from the infrastructure description, and reproducing the same configuration on a clean environment no longer works.
Hence unstable environments. The test environment follows one set of rules, production another, and degradation under growing load can only be reproduced in production.
The downside of this scheme is night incident reviews, release rollbacks, and caution instead of product development. Below we discuss how this is addressed.
What a managed cluster provides: impact on timelines and failures
A managed cluster takes on what does not create product value: updating the control plane, replacing nodes, rotating certificates. The declarative description of Azure Kubernetes Service fixes the cluster state in configuration rather than in the on-call engineer's memory. Hence predictable operations: the same result in the test environment and in production.
Speed of rolling out a new service version
Rolling replacement of instances with readiness checks allows releasing without a downtime window: traffic goes to the new version only after it has confirmed it is operational.
The new version goes into service in minutes, not after nighttime approvals — the replacement order and compatibility are described in manifests. Returning to the previous version is also a command, not an emergency call.
Managed recovery after node failure
A node is marked as unsuitable, its workload moves to the remaining ones, and a replacement is brought up automatically according to the node group description. Recovery time is measured in minutes and does not depend on which engineer is on call.
As load grows, capacity is added, and after the peak it is removed, so reserve capacity does not sit idle.
Eliminated class of failures caused by manual edits
Edits through the web console are the most common cause of environment drift: someone adjusted limits, someone adjusted a network rule, and it can no longer be reproduced.
When changes pass only through configuration and review in the version control system, an entire class of failures disappears. Drift between the test environment and production is visible before release, not at the moment of an incident.
Azure AKS deployment options: what fits when
There are three approaches to deploying Azure AKS, and the choice between them is determined not by team size, but by how many environments need to be maintained and how often the configuration changes. Let us compare them, then examine the extreme cases in more detail.
| Option | When it fits | Limitations |
|---|---|---|
| Manual setup: portal and CLI | Pilot, one-off hypothesis validation, one environment | Configuration lives in command history; the cluster cannot be reproduced exactly; configuration drift is not tracked |
| Infrastructure as code on Terraform | Multiple environments, regular releases, reproducibility and audit requirements | Requires expertise and state management; initial setup takes longer |
| Hybrid approach | Core — network, nodes, roles — as code, application add-ons manually | Without a governance policy for the boundary, drift returns |
For which tasks a manual setup is needed
A manual setup is justified when the cluster is needed once and briefly: validate a hypothesis, run a load test, train the team to work with a managed setup. It is assembled quickly, and no expertise in infrastructure description is required.
There is one limitation, but it is strict: reproducing the same configuration a month later will not work, and rollback after an unsuccessful change comes down to manual reassembly.
Where full infrastructure as code is justified
As soon as there is more than one environment or releases happen regularly, manual maintenance stops scaling.
Terraform shows the change plan before applying, changes go through review, and cluster state is stored centrally — drift is visible before it reaches the production environment.
A second cluster is brought up with the same description and different parameters: environment recovery takes minutes.
Work stages: from cluster audit to handover of the setup
The work proceeds in sequential stages, where each next stage relies on the result of the previous one. That is why problems are uncovered during the audit and checks, not at the moment when the cluster is already carrying production load.
- Cluster audit. We capture the actual state: composition and sizes of node pools, update policies, access rights, what is described in code and what is configured manually through the console. The output is a risk map and a list of discrepancies between the declaration and reality.
- Environment and network design. We establish how many setups are needed, how subnets and service addressing are separated, and which routes are open externally and internally. We agree on the design before the first changes — redoing the network on a live setup is the hardest part.
- Cluster configuration. The configuration is transferred into the infrastructure description: node pools, autoscaling, update windows, workload isolation, secrets, and role-based access. After that, manual edits are not needed — the state is brought to the described one and reproduced from scratch.
- Fault tolerance testing. We controllably take a node out of service and observe behavior during a zone failure, during a version update, and during recovery from a backup. We record metrics and time to return to normal, not just the fact of a successful start.
- Operations policy. We describe typical operations: rollout, rollback, scheduled maintenance, and incident response. The team receives instructions for its tasks, not generic product documentation.
- Setup handover. We walk through the cluster architecture, access rights, and change procedure with the customer's engineers, and conduct training on real scenarios. After that, the setup operates without our ongoing support.
Each stage ends with an artifact that can be verified: a risk map, a design, an infrastructure description, a testing report, a policy. This order allows stopping at any step and understanding what has already been done and how it is confirmed.
Deliverables: configurations, designs, and policies
The work ends not with launching the cluster, but with handing over artifacts by which it can be maintained and reproduced from scratch.
Everything is stored in a repository under version control: changes go through review and are applied automatically, not manually from the console. Below is the set that remains with the customer.
- Cluster configurations — Terraform modules: node pools, autoscaling, zones, add-ons. Variables are separated by environment, and default values are documented.
- Terraform state file — remote storage, locks, separate state per environment: parallel applies do not break the infrastructure.
- Network designs — virtual network, node and pod subnets, routing, egress points: it is visible which traffic is allowed where.
- Access policies — roles and bindings, separate rights for people and services, network policies between namespaces, prohibition of privileged containers.
- Cluster documentation — deployment from scratch, adding an environment, where to view metrics and logs, which alerts are configured, and what to do when they fire.
- Update policy — maintenance windows, procedure for updating nodes and add-ons, testing in the staging environment, rollback criteria.
- Recovery plan — backups of state and storage, step-by-step return to the last working version, and verification of recovery in the staging environment.
Case study: a cluster for an online store's peak load
An online store with daily demand unevenness: quiet night hours and sale peaks, when the number of simultaneous shoppers grew several times over in a matter of minutes.
The cluster in Azure Kubernetes Service was assembled manually through the portal, some settings lived in chat, and any change before a promotion turned into a separate risk.
The task was formulated simply: the peak must pass without the cart going down and without an on-call engineer at the console.
What exactly was reworked
The infrastructure description was moved to Terraform: cluster, node pools, network policies, access rights. The configuration became reproducible, a change goes through review and is rolled back with one command, not "from memory."
Autoscaling was configured based on two signals at once — actual load and schedule: a nighttime minimum and pre-warming nodes for the morning peak, so that cold start does not consume the first minutes of the sale.
Readiness probes and pod distribution across availability zones were set up so that a node leaves rotation without dropping active requests.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Share of successful requests during sale hours | drop to double digits | stable at the target level |
| Recovery after node failure | several hours, manually | minutes, automatically |
| Manual interventions per month | dozens | isolated |
The cluster stopped being a bottleneck: peaks are planned in advance, and engineers focus on platform development rather than firefighting nighttime failures.
What to do if the cluster is already deployed manually?
A manually assembled cluster usually carries production load, so the logic of "tear it down and rebuild it" does not fit here.
The workable path is not rebuilding, but inventory: first we describe what actually exists, then gradually move resources under description management without stopping services.
While the description does not match reality, it cannot be applied — otherwise the tool will try to recreate objects that are already running.
We start by importing existing resources into the state file and reconciling the change plan with the actual configuration.
The readiness criterion for the transition is simple: the plan must be empty — that is the backward compatibility under which description management breaks nothing. Until the difference is reduced to zero, no change is applied to the cluster.
Then comes a gradual transition by layers. The first to go are things that do not carry traffic: quotas, access policies, configurations. Then workloads, and only at the end — what is riskiest to touch: network rules, external entry points, data disks.
Each layer is transitioned in a separate iteration, with a separate check and the ability to return to the previous set of descriptions.
A zero-downtime transition is ensured by the fact that new nodes are added in parallel with existing ones, the load moves as they become ready, and old ones are drained in a controlled manner and only after confirmation that everything works.
Rollback in such a scheme is returning to the previous descriptions: the state is restored, and manual edits made before the transition are not lost.
After several iterations, the cluster is fully described, without manual settings that cannot be reproduced. After that, any changes follow the same path: description, plan review, apply.
If you need to estimate the scope of such work for your setup — describe the task, and we will come back with a transition plan.
Technical concerns: network, access, and updates
Network, access, and update procedures are what most often come up before moving a cluster into production. Below are answers to questions asked before the start, so that the boundaries of the work are clear in advance.
How are nodes updated if the service runs around the clock?
The rollout proceeds one node at a time: it is removed from load balancing, the load goes to the remaining nodes, and pods move according to the rules for the allowed number of unavailable replicas.
We set maintenance windows during hours of minimum traffic, and for services that cannot survive a restart, we configure several replicas and correct readiness probes in advance.
How are environments isolated — test, pre-production, and production?
Each lives in a separate namespace with its own secrets, quotas, and limits. Network policies by default deny traffic between environments, and we grant permissions selectively. Test environments cannot see production databases and cannot write to them by mistake.
What is included in backup?
Scheduled snapshots of persistent volumes and manifests, with copies in a separate storage. We verify recovery in drills: we deploy an environment from a backup and compare the result, so that "there is a backup" means a confirmed recovery time.
Where is the boundary of responsibility between you and our team?
Our side is the platform: cluster configuration, updates, network policies, monitoring, and infrastructure alerts. Your side is application code, releases, secret contents, and user rights.
We fix the boundaries with a design and policy at the handover stage, so that during an incident it is clear who looks at what.
Let's discuss the task: we will come back with an estimate and work plan
Tell us how the cluster is currently set up: how many node pools, what network model, where Terraform state is stored, and who makes changes bypassing the configuration. Such a description already reveals most bottlenecks — from manual edits to the absence of checks before apply.
In the first conversation, we clarify:
- what loads and node profiles are needed for the services, whether there are peak periods and autoscaling requirements;
- how environments are separated and whether they need to be split across subscriptions and namespaces;
- what requirements exist for fault tolerance and the acceptable recovery window after a node or zone failure;
- what checks changes must pass before rollout to production;
- who on the customer side makes infrastructure decisions and how changes are approved.
We will attach to the response a staged work plan with acceptance points, the team composition with roles and responsibility areas, and the start procedure: what we do in the first week and what will be needed from your side.
Describe the task in free form — the current configuration, what is unsatisfactory, and what result is needed. We will come back with an estimate, a work plan, and clear timelines.