Implementing automatic resource scaling based on load metrics

Why Kubernetes autoscaling cannot keep up with load

A classic failure looks like this: trigger thresholds are set for an average day, and the metric is averaged over a long window. Load grows in minutes, the decision to add copies is made with a delay, and the first requests are already queued. Then the cumulative effect kicks in: the queue lengthens faster than service copies are added.

The second layer of the problem is a resource bottleneck at the cluster level. Even when Kubernetes autoscaling correctly increases the number of copies, there is no free capacity on the nodes: new copies hang waiting because machines were not prepared for them in time.

Cluster node overload occurs — only tasks that already consume memory and CPU time remain on them, and the scheduler has nowhere to place the rest. The application waits for resources, timeouts expire, and this ends in service failure at peak.

Most often this is treated with manual scaling: the on-call engineer sees an alert, adds copies and nodes, and an hour later removes the excess. The scheme works exactly until a peak arrives at night or when there are several peaks in a row.

Manual scaling also does not save you from the opposite error — after a decline, capacity remains enabled because no one is in a hurry to remove it.

Let's break down how this is addressed: which metrics to use as a basis instead of averages, how to link service scaling and node pool scaling, where to keep a pre-warmed reserve, and how to configure a return to the original size without latency drops.

What changes after implementation: time, resilience, failures

Before automation is configured, load growth was noticed by the on-call engineer: the graph went up, complaints followed, and only then did someone manually add replicas or expand the node pool.

Between the traffic surge and infrastructure readiness, ten minutes to half an hour passed, and all that time some requests experienced delays or errors. Downtime due to lack of resources was not an accident but a standard consequence of manual mode.

After implementation, the decision to expand is made by automation based on metrics: CPU load, request queue length, number of simultaneous connections.

Reaction time to sustained growth fits within one to two minutes, and thresholds and steps are selected so as not to disturb the cluster on every short peak.

Kubernetes provides standard mechanisms for this — the work is not in enabling them, but in calibrating them to the profile of a specific load.

The class of failures also changes.

The scenario of "hitting the ceiling and starting to return errors" goes away: when the threshold is exceeded, replicas are added, and the share of requests served without delays stays at the same level both at peak and during quiet hours.

Behavior during load decline is configured separately — excess replicas are removed gradually so as not to catch a rollback immediately after the decline.

The burden on the on-call shift drops noticeably: night manual resource increases are replaced by anomaly analysis, not firefighting. Downtime losses stop being measured in hours — incidents related to lack of capacity move into a separate class and are handled pointwise. Describe the load profile — we will come back with an estimate and a setup plan.

Kubernetes autoscaling modes for different load patterns

Modes do not replace each other: the choice is determined by how the load behaves — smooth growth, sharp peaks, or predictable windows. We compare options along three axes: what signal the mode reacts to, which profile it suits, and where it hits limitations.

Comparison of scaling modes

All modes share the same general settings: thresholds for adding and removing instances (different thresholds remove jitter), stabilization windows, and minimum-maximum limits.

A combination of modes is normal: the schedule covers known peaks, metrics pick up the rest, and the queue provides backup for deferred processing. Next — how we select thresholds for the load profile.

How implementation proceeds: from load audit to handover

Kubernetes autoscaling implementation follows a fixed sequence: first we measure, then we design, and only then do we enable automation in the production contour.

The reverse order — rules set by feel and copy-pasted from documentation — results in either idle replicas or overload at peak. Below is how the work looks from start to handover.

1. Load audit. We capture the profile over a full cycle: daily and weekly fluctuations, peak hours, spikes from external events and batch jobs. We rely on actual measurements — CPU load, memory consumption, request queue length, response time — not on declared capacity.

2. Scheme design. We choose which indicators control scaling, set thresholds, change step, and replica count limits. We separately define warm-up of new instances and stabilization time so automation does not "jitter" on every short spike.

3. Automation setup. The scheme is transferred into declarative configurations: they live in the repository, go through approval, and are applied automatically.

   Manual edits in the production contour are excluded — otherwise the state diverges from the description within a couple of weeks.

4. Load testing. We simulate a peak, smooth growth, and a sharp traffic drop, and check behavior when an individual node fails. We measure time to reach operating mode and the share of requests served without errors at peak.

5. Phased rollout to production. We enable the new rules first on part of the services, with the ability to quickly roll back to previous thresholds. This way, an automation failure does not affect the entire contour on an individual calculation basis Transfer and training.

   We hand over configurations, the scheme, a description of thresholds, and the procedure for anomalies; we train the on-call team. After handover, changes to the rules are made by your engineers.

Configurations, schemes, and regulations that remain with you

Autoscaling lives not in a presentation, but in a set of files and regulations by which the cluster runs without us.

We hand over everything we wrote and verified under load: configurations, schemes, thresholds, and instructions — with versioning in your repository and an explanation of why exactly this value was chosen.

• Autoscaling manifests: target CPU and memory load, minimum and maximum replica count, stabilization windows — so the system does not react to short spikes and does not shut down the service during a one-off peak.
• Resource allocation rules: container requests and limits, priorities and service classes, placement across node pools and zones — so critical services are not evicted by background jobs.
• Cluster scheme: pool composition, node roles, availability zones, external dependencies, and failure points with a note on what happens if each element fails.
• Metric sources and collection rules: which indicators signal growth, which are ignored, where they are stored, and how their reliability is verified.
• Rollback instructions: how to disable autoscaling, return the previous configuration revision, safely drain a node, and bring the service up in a predictable time.
• On-call regulations: who receives alerts, which thresholds are considered an incident, what the on-call engineer does themselves and what they hand off, and by what criterion they close the case.
• Documentation: service map, change log, threshold rationale, and acceptance procedure — so a new engineer can figure it out without messaging the author.

Case study: seasonal peak without manual interventions in the cluster

Before: manual scaling on alert

The cluster served an online store with a pronounced seasonal peak: the start of a night promotion caused traffic to grow 4–6 times in 10–15 minutes.

Replicas were scaled manually — the on-call engineer received an alert, added capacity, waited for cache warm-up, and only then opened traffic. From alert to readiness took 25–40 minutes.

During this window, the frontend layer was overwhelmed: the share of 5xx responses at peak reached 7–9%, and some orders were lost irrevocably. On-call shifts during the season turned into night wake-ups 4–6 times a week.

This is a class of failures determined not by hardware reserve but by the speed of reaction to load growth.

After: automation by load metrics

We collected signals that reveal overload in advance: CPU load, request queue length, and the number of active sessions per node.

On these, we built Kubernetes autoscaling with separate rules for business hours, the night window, and promotion starts, added stabilization windows against flapping, and a warm-up step — a new replica accepts traffic only after cache readiness.

Reaction to growth decreased from 25 minutes to 90 seconds, and the share of 5xx at peak — from 7–9% to 0.3%.

Manual interventions during the season — zero instead of 4–6 per week, and during quiet hours the number of running replicas drops on its own: allocated resources in the off-season decreased by about three times. We returned to edits only when we needed to raise the triggering threshold itself.

How does vertical scaling differ from horizontal?

Vertical scaling increases resources for a single replica; horizontal scaling adds copies of the service and distributes load between them. The difference is not which approach is more modern, but which ceiling is hit first: node capabilities or application architecture.

The vertical path does not require code changes: state remains in place, and data does not need to be sharded. There are two limitations — node reserve and a restart when changing the resource class, i.e., a short service interruption.

That is why stateful services scale this way: databases, message brokers, monolithic applications where splitting into copies requires reworking the logic.

Stateless services scale horizontally: APIs, background job handlers, web frontend. Copies are interchangeable, replicas are added and removed on the fly, updates go one by one — without downtime.

The price is externalized state, correct readiness checks, and balancing; otherwise some requests go to a not-yet-warmed copy.

In practice, modes are combined: the data layer is scaled vertically, the application horizontally, and automatic scaling by load metrics controls the number of replicas within specified limits.

Technical doubts before starting work

Doubts before implementing Kubernetes autoscaling are usually the same: will it break the existing configuration, who will get the right to change resource limits, and what will happen to the manual settings accumulated in the cluster. Below are answers to the questions that most often come up during the assessment stage.

Who will be able to change resource limits?

Rights are distributed by roles: a developer manages the limits of their own workload, while common policies change only through approval. All changes go through version control — you can see who moved a threshold and why, and you have something to roll back.

What will happen to manual settings?

We inventory them: some become baseline values, some become exceptions with an explicitly described reason. Manual intervention is not disabled; it simply stops being the only way to relieve a peak.

How are changes rolled back if something goes wrong?

Before enabling in the production contour, we run the behavior on a test one and observe the reaction to peak and decline. Rollback is returning the previous description and thresholds; automation does not leave workloads stuck at minimum after a decline.

Let's discuss the task and come back with an estimate and plan

For the estimate to be a calculation rather than a guess, we need source data on load, not just the current autoscaling settings in Kubernetes. The key is the profile: how load rises and falls over a day and a week, where the peaks are, and how predictable they are.

From this data, it is clear whether reaction by application metrics will be fast enough or whether node reserve is needed, and which triggering thresholds will be safe for response.

What to send for an initial review:

• load profile for a typical week: graphs of requests, queue lengths, CPU and memory consumption;
• cluster scheme: how many nodes, what types of workloads live on them, what runs one-off;
• current scaling rules and limits — even those set by eye, they provide a starting point;
• list of services with strict response time requirements, where scaling up may not keep up with a spike;
• external dependencies: databases, caches, queues — they do not scale themselves and often become a bottleneck;
• incident history: when resources were insufficient, and when downtime was incurred for nothing.

Then comes the work order: we analyze the data, reproduce the peak profile on a stand, select metrics and thresholds, and check behavior during node failure and when load returns to normal.

After that, we come back with an estimate and plan — stages, responsibility boundaries, and acceptance criteria by which you can verify the result. The absence of some data does not block the review: we will capture what is missing ourselves on your contour and show what we got.