Why managed Kubernetes in Azure fails under load
Managed Kubernetes in Azure removes control plane maintenance from the team, but not load distribution: the scheduler, limits and scaling rules remain on the team's side.
Cluster degradation rarely looks like an outage — it is a slow increase in latency that is noticed only through complaints.
The first scenario is distribution skew. Pods are scheduled on two or three nodes while the rest sit idle: resource requests are set below actual consumption, the scheduler considers nodes free, and zone placement rules are not defined. At peak, hot nodes run out of capacity while half of the group is underutilized.
The second cause is resource limits set "by eye." An underestimated limit causes memory-related container restarts, while an overestimated one prevents the node from packing and holds a reserve that is never used.
Until resource requests are calibrated to the real profile, scaling decisions are made on distorted data.
The third is the lack of node autoscaling. Pod replicas are added, but the node group does not grow, new pods remain pending, and all traffic goes to overloaded instances.
The opposite error is thresholds firing after the peak, and nodes appearing when the cluster has already degraded.
Finally, "silent" failures: container restarts and growing latency do not surface in notifications because metrics are collected but there are no thresholds for them.
We break down how this is addressed: from load distribution and limit calibration to signals that warn of degradation in advance.
What changes in service operation after cluster configuration
After configuring managed Kubernetes in Azure, the changes are visible not in diagrams but in four metrics collected from the cluster itself: response time under peak, successful deployment rate, service recovery time after node failure, and the number of incidents due to manual edits.
Response time under load stops depending on who is on duty. Capacity is added based on consumption metrics, not a phone call, so the peak is handled by standard scaling and the request queue does not grow.
The resource reserve is defined by policy — the service maintains its usual response time both on a weekday baseline and during a promotion.
Zero-downtime updates rely on rolling deployment with readiness checks for each new copy: traffic goes only to instances that have responded, and unready ones are removed from load balancing.
If checks fail, the rollout stops itself, and the version rolls back to the previous one. The successful deployment rate grows not because releases became simpler, but because a faulty release no longer reaches everyone at once.
Cluster fault tolerance works at another level: when a node is lost, its load is rescheduled to free nodes rather than waiting for an engineer.
Service recovery time is measured in minutes and does not depend on the time of day — the scenario is described in advance and tested in drills.
A whole class of failures due to manual edits also goes away: configuration is described in files, changes pass review, and discrepancies between actual and described state roll back automatically.
Cluster organization options and their technical limitations
The choice comes down to two questions: who owns the control plane and whose machines run the worker nodes.
A fully managed cluster gives the control plane to the provider — updates, fault tolerance and certificates are in the provider's zone, while you retain node pools, manifests and applications.
A hybrid keeps the control plane managed and places nodes on your own hardware: in a private perimeter, another cloud, or a rack.
| Option | When it fits | Limitations |
|---|---|---|
| Fully managed cluster | Need a quick start and minimal operations, load fits in the provider's cloud | No access to control plane nodes, some network add-ons and privileged modes are unavailable |
| Hybrid: managed control plane and own worker nodes | Data must remain in your own perimeter, but there are no resources to maintain the control plane | Need secure channels to the control plane, your own metric and log collection agents, a separate node update procedure |
| Cluster on your own machines | Strict isolation or hardware requirements, and an operations team is available | Control plane, state storage, certificates and updates are entirely your responsibility |
The limitations of the managed option are important to know in advance: you cannot change API server parameters, install arbitrary add-ons on the control plane, or rely on access to its logs.
Anything requiring node privileges or a special network scheme moves to your own pools or is addressed by a hybrid.
In practice, the hybrid scheme often wins: a managed control plane removes most operations, while your own nodes cover data and hardware requirements.
We select the option for the constraints of a specific perimeter and fix what exactly remains in your team's area of responsibility.
Implementation stages: from audit to handover to operations
Cluster implementation is not a one-time installation but a sequence where each step relies on the previous one. An error in network or access rights design surfaces months later, when fixing it is most expensive. Below are seven stages that the work follows.
- Infrastructure audit. We analyze current services, load profile, peaks, dependencies and data isolation requirements. The output is a map: what we move to the cluster and what makes sense to leave as is.
- Architecture design. We distribute workloads across node pools, define autoscaling rules, network boundaries, the certificate issuance scheme and access separation.
- Environment preparation. We create separate environments for testing and production, connect the image registry, secret store, metric and log collection.
- Cluster configuration. We deploy managed Kubernetes, enable automatic updates and state backup, and verify that failure of a single node is invisible to services.
- Load testing. We run the real traffic profile, observe autoscaling behavior at peak and deliberately take a node down — failover should occur without lost requests.
- Policies and access. We fix access policies, secret rotation, resource limits, the update procedure and the incident response plan with owners on each side.
- Handover to operations. We train your team to release and roll back updates, read metrics and alerts, and troubleshoot typical failures. Configurations, access and documentation remain with you.
What you retain: configurations, diagrams, access policies
The cluster is handed over not as "running" but as described: each element is reproducible from the repository, and the solution can be justified during security approval.
Below are the artifacts that remain in your perimeter and allow you to operate the platform without us.
- Declarative configuration descriptions: cluster manifests, node pools and network add-ons in version control — with change history and rollback to the previous state.
- Network diagram: address spaces and subnets, routing to the corporate network, service publication points, private access to nodes.
- Access policies: roles and their bindings, integration with the corporate account directory, separation of rights between development and operations, rules for access to secrets.
- Update policy: procedure for updating platform and node versions, windows, preliminary workload compatibility check, rollback plan.
- Recovery policy: backup composition, frequency, allowed downtime and recovery point, drill recovery protocol — a copy is considered working only after verification.
- Observability perimeter: list of metrics and logs, alert thresholds, notification routes, on-call shift composition.
- Instructions for typical operations: service connection, granting access, node pool scaling, diagnostics of common failures — with commands and expected result.
We verify the diagrams against the actual state at acceptance: we eliminate discrepancies before signing so that documentation does not diverge from the cluster from day one.
Case study: reduced recovery time and eliminated degradation
Before
The customer is a service with a pronounced load peak: for several hours a day, traffic increased several times, while the rest of the time the system was idle.
The cluster ran on the managed service Azure Kubernetes Service, but operations relied on manual steps: rollout was launched from an engineer's terminal, the number of nodes was adjusted for the peak in advance and at random, and rollback to the previous build existed only on paper and had never been tested on a live perimeter.
When a node failed, recovery stretched to tens of minutes while the on-call engineer figured out what exactly had failed.
What we changed
The infrastructure description was moved to a repository — manifests and charts became a single source of truth. Rollout became a pipeline step with pod state checks after each step and automatic rollback if the build fails readiness probes.
We added node autoscaling based on load, resource limits for each service and network policies so that one service does not consume cluster resources. We configured state backups and run recovery on a copy of the perimeter quarterly.
What changed
Recovery time after node failure decreased from forty minutes to five — pods migrate automatically. Manual operations during rollout went from twelve to one, and that one is reduced to release confirmation.
At peak, requests no longer time out: nodes are added within a minute, and capacity is not held with a "by eye" reserve, so resource consumption stopped growing with the peak.
What to do if a Kubernetes cluster is already deployed manually
A manually deployed cluster does not need to be demolished — first we describe its actual state: where workloads run, which objects are edited manually, what lives only in scripts and engineers' memory.
Without such inventory, cluster migration becomes transferring the same problems to a new site. Legacy configuration usually contains real value — a couple of custom improvements for a real task — and a dozen deviations from the standard scheme that drag on for years.
Each deviation gets an explicit decision: migrate as is, bring to standard, or discard as outdated. Some homegrown add-ons are replaced by native tools of the managed perimeter, some remain as a separate layer on top of it with a clear owner.
Thus the transition to a managed cluster does not lose accumulated work and at the same time removes manual maintenance from the team.
The new perimeter is deployed in parallel and does not interfere with the old one. Workloads migrate in groups — by services or teams, not all at once: first stateless ones, then those working with disks.
Switching occurs at the entry point: until the new perimeter confirms stability by metrics, weight returns to the old one with a single edit.
Separately about state — volumes are transferred by replication with checksum verification, queues and background jobs wait to be processed, and the low-load window is selected by schedule, not calendar.
Zero-downtime load migration relies on reversibility: until the final step, the old cluster remains operational and can take the load back.
Common technical concerns before starting work
Before the start, customer engineering teams ask almost the same set of questions: where the boundary is between our work and their operations, when the cluster updates and what happens to it under sharp load growth.
We answer before the first commit — on Azure Kubernetes Service this removes most surprises during production rollout.
We fix the boundaries in a responsibility matrix. The control plane, updates, monitoring, access policies and backup are our area; business logic, repository access and data remain with the customer. This closes the "who is to blame" dispute when an on-call alert fires at night.
How are update windows coordinated?
We assemble the schedule from your version rollout schedule: the control plane updates in an agreed window, worker pools in waves, with node expansion before old ones are decommissioned. During a release freeze, the window is moved, and manual mode remains available.
Who is responsible for managed cluster security?
We cover the infrastructure perimeter: role separation, network policies, namespace isolation, secret rotation, access audit and image scanning before rollout. Secrets and access inside applications remain with the customer's team.
What happens under load growth?
Node and pod autoscaling are configured in advance, together with subscription quotas and a safety pool for peaks. At the start we run the load profile and see where the cluster hits limits first — resource limits or the application itself.
Let's discuss the task and come back with an estimate and implementation plan
To start, no ready documentation is needed: read-only access to the subscription, a list of services migrating to managed Kubernetes Azure, and availability requirements are enough — allowed downtime, maintenance window, peak load.
What is not stated in words, we will reconstruct from the configurations and logs of the current perimeter.
The review takes several days. Based on its results, we come back with a project estimate and an implementation plan where each work item is tied to a stage and a verifiable result.
- Target architecture diagram: node pools for different load profiles, namespace separation, network policies and access roles.
- Implementation plan by stages: subscription and network preparation, cluster deployment, service migration, update and backup configuration.
- Estimate of scope and timeline — by stages, with an explicit boundary: what we do and what remains on the customer's side.
- List of risks and rollback points: what happens if a node update fails and how we return to the previous state.
- Access requirements: what level of rights in the cluster and subscription each of your engineers receives.
- Acceptance procedure: which checks confirm that the cluster is ready to accept production load.
As a first step, we usually take the cluster configuration and workload into descriptions that can be versioned and reproduced — after that, any changes go through them, not manually in the console.
We fix the scope of work and the start after plan approval: describe the task — we will come back with an estimate, timeline and a clear first step.