Why HPA Kubernetes does not scale pods to real load
Evening peak, queues grow, response time goes beyond seconds — and the number of replicas stays put. Pod autoscaling does not work not because the controller is broken: it faithfully does what it was told.
The discrepancy is almost always in the input data — thresholds, metrics, and application behavior have diverged from the load profile.
First source — thresholds set "by eye".
HPA calculates percentage not from limits but from requests, so an underestimated CPU request keeps the service in overload mode and pins the counter at maximum, while an overestimated one masks a shortage of replicas under load until first failures. Target level is taken from measurements, not from feelings.
Second — readiness probes. If the probe reports readiness before the application has warmed cache and opened DB connections, traffic goes to a not-ready pod and it crashes.
Too strict a probe has the opposite effect: pods do not come into service, and peak load on the service falls on fewer instances.
Third — metric lag and long startup. The metric arrives after the aggregation window, the decision is made even later, and the pod starts longer than the peak lasts: new instances appear when load has already subsided.
If resource headroom on nodes is not provisioned, node overload in the cluster blocks scaling — replicas are described, but there is nowhere to place them.
Therefore, setting up HPA Kubernetes begins not with editing thresholds, but with consumption measurements, bringing requests in line with reality, correct probes, and explicit scaling behavior. We analyze how this is addressed.
What managed application autoscaling provides
Manual scaling cannot compete with traffic burst: while the engineer notices growth, deploys copies and checks their readiness, some requests already receive errors.
HPA closes this gap — replica count is recalculated by load metrics and application signals such as queue length or number of active connections.
Reaction speed to peak is set not "by eye" but from the service profile: for synchronous handlers target utilization is usually 60–70%, for queue processing — by number of messages per replica.
Then with a twofold traffic increase, new copies start working within tens of seconds, peak passes without degradation, and does not turn into an incident with root-cause analysis.
The share of eliminated failures is a direct consequence: some failures are removed not by permanent capacity reserve that idles during quiet hours, but by timely expansion.
Behavior during decline is configured separately — stabilization window prevents killing and raising replicas again on every short dip, otherwise the service starts "breathing" and loses stability.
Recovery time after burst depends on how quickly metrics reflect reality and how readiness checks are designed: until the new copy is ready to accept traffic, it must not enter request distribution.
In combination with node scheduling, this provides resilience under load even if a separate machine fails — traffic moves to remaining ones, and autoscaling adds missing copies.
Autoscaling modes: comparison of approaches
Autoscaling in the cluster is not one mode, but a set of mechanisms enabled according to load character.
Scaling by metrics provides basic response to growth in resource consumption, by external metrics — reaction to real request flow or queue length, by schedule — anticipation of predictable peaks.
Choice depends on how far in advance the traffic profile is known and whether the service has a metric that honestly reflects the amount of unfinished work.
| Option | When suitable | Where it breaks |
|---|---|---|
| By resource metrics | Even load, service is constrained by CPU or memory | Reaction with delay: metrics arrive averaged, burst manages to degrade response |
| By external and custom metrics | Queues, request flow, backlog length — load is visible before degradation | Requires reliable metrics source; if unavailable, autoscaling goes blind |
| By schedule | Known peak hours, reports, nightly batch jobs | Does not react to unplanned growth; overestimated minimum replicas idle |
| Manual replica count | Rare releases, stable traffic | Forgotten value change: either downtime or overpayment for idle copies |
Usually it is not the mode itself that breaks, but scaling thresholds.
If the scale-up threshold and scale-down threshold are too close, replicas start being added and removed in a loop — the service flaps, the database gets connection spikes, response time drops more than under static load.
This is treated with a stabilization window for scale-down and threshold separation, so the decision to reduce copy count is made by stable trend, not by one dipped point.
On a running system, modes are almost always combined: basic scaling by metrics covers everyday fluctuations, schedule — known peaks, external metrics — what resource metrics do not see, and manual boundaries prevent extremes. Next we analyze how to set these boundaries and avoid flapping when load changes.
Implementing HPA Kubernetes: stages of engineering work
The order of work is the same for one service and for dozens: first we capture facts about load, then change configuration. A skipped audit almost always results in thresholds set at random — autoscaling either lags at peak or flaps replicas on every short burst.
- Load profile audit. Collect metrics for a representative period — day, week, month, — separately mark peaks of sales, reporting windows, and nightly batch jobs. Record by which signal each workload should scale: more often it is not CPU, but response latency or queue length.
- Scheme design. Determine the list of workloads for autoscaling, replica count boundaries, linkage with node scaling, and behavior on load removal. Agree on what is considered emergency state and what is normal peak.
- Configuring scaling thresholds. Set target signal values, averaging windows, reaction delays, and behavior policies for scale-up and scale-down. Step limiters prevent the cluster from runaway when the metric jumps.
- Load testing. Run scenarios on the test environment, reproducing real traffic profile, and compare time to reach required capacity with expected. Separately test failure of one replica and loss of a node.
- Handover to operations. Provide configurations, scheme, alerts for hitting the upper boundary, and regulations for changing thresholds, so the support team can edit them without us.
Acceptance criterion is not "objects created" but reproducible behavior under load: reaction time, absence of unnecessary restarts, and clear failure point.
What remains in project: configurations and regulations
Autoscaling is not one enabled mechanism, but a set of related settings that someone must change and maintain. Therefore the project leaves artifacts by which your team reproduces scaling behavior and changes it without us.
- Autoscaler configurations — descriptions for each service: target metrics, trigger thresholds, minimum and maximum replica count, delays before scale-down.
- Service scaling scheme — what grows under load, and what is constrained by database, cache, or external service and limits growth before others.
- Access rules — who in your team changes replica boundaries and metrics, how edits pass review, and do not go directly to live environment.
- On-call regulations — by which thresholds notifications arrive, who reacts and within what time, when the second line is engaged.
- Operations documentation — how to view current replica count and decision history, how to temporarily disable autoscaling on a service, what to do if metrics fail.
- Verification scenarios — load runs with which you reproduce burst yourself and make sure policy worked after edits.
- Rollback plan — procedure for returning to fixed replica count if behavior gets in the way, and points after which rollback is still safe.
We hand this over not as an archive of files, but as a walkthrough: we go through each item with your team so configurations do not remain a formality.
Case: eliminated failures and downtime during load peaks
How it was and what we changed
Project — service with sharp daily peaks: nightly batch processing and morning user influx tripled load in half an hour. Autoscaling worked formally — one CPU load threshold for all services.
Application instances did not have time to come up before peak, and after decline hung for hours and occupied cluster resources.
During peak hours, share of unavailable requests reached several percent. On-call shift reacted manually: on average 8–10 interventions per week of 15–20 minutes each, because automation triggered with delay and did not see the request queue.
We rewrote rules: added custom metrics by queue length and response time, separated thresholds by service types, set stabilization periods against fluctuations on short bursts, and correct requests and limits for each container in Kubernetes.
Separately checked behavior on node loss — instances redistribute themselves, without manual startup.
After changes, share of unavailability during peaks fell from several percent to fractions of a percent, on-call interventions decreased to single cases per month, and reaction to load growth accelerated from 15–20 minutes to 1–2. Resource consumption did not grow: the cluster stopped holding extra replicas during decline hours.
How metric-based calculation differs from schedule
Metrics react to actual load: HPA reads utilization, queue length, or latency and adds replicas when they go beyond target value.
Unplanned growth is caught only by this method, but with lag: metric is collected at interval and averaged, controller recalculates desired replica count, pod still passes readiness checks. Under peak, some requests land on already overloaded pods.
Schedule is more reliable where load repeats by clock: morning login, nightly batch processing, evening report. It does not depend on availability of the metrics system and pre-warms pods, bypassing cold start. The price — outage outside schedule and extra replicas during quiet hours.
Combined scheme brings both approaches together: schedule raises lower boundary before known peak, metrics cover the difference on warmed pods.
But only if schedule does not overwrite replica count directly — otherwise conflict of scaling rules arises: scheduler sets its value, HPA returns its own after recalculation cycle, replicas swing, and pods are killed right under traffic.
Therefore schedule is given only boundaries: minimum and maximum replicas, target thresholds, set of active metrics. The owner of replica field remains one controller.
Behavior is smoothed by policies: separate rules for scale-up and scale-down, stabilization window against short bursts, averaging period against metric noise. With multiple rules, controller takes largest change over period.
Choice of scaling strategy comes down to profile predictability: repeating — schedule as base, metrics as safeguard; bursty — the opposite.
Technical doubts before launch
Before putting autoscaling into production, engineers raise the same questions: what to scale on, will service "flap", will database hold up, and how to verify all this without risk. Below are short answers that we go through before enabling HPA policy.
There are no metrics — what to scale on then?
By default, Horizontal Pod Autoscaler reads CPU and memory utilization from Metrics Server. If CPU lags and does not reflect real picture, we connect external metrics: queue length, requests per second, handler lag. We take as reference the one that grows earlier than service degrades.
Will replicas "tremble" — either added or disappearing?
Trembling is removed by behavior settings: stabilization window during load decline, replica count change step, gap between trigger threshold and operating value. We scale up quickly, down with delay, so short burst does not drag rollback with it.
Will autoscaling kill the database?
Application pods are scaled, not connections to database, so growth in replica count multiplies connection pool. We limit upper replica boundary, move connections to pool proxy, and verify database holds query peak, not only traffic peak.
How to verify configuration before production operation?
We run step load on test environment and look not at traffic graph, but at replica graph: do pods manage to be added before response time grows. Separately verify pod readiness, disruption budget, and coordination with cluster node scaling.
Let's discuss the task: where implementation begins
To make work estimate based on real picture, not assumptions, at start we need a short set of inputs.
Requirements for fault tolerance are gathered in parallel: the earlier target indicators are clear, the more accurate the scaling implementation plan is — without rework after first peak.
From customer, enough is:
- list of services that must scale, and their priority relative to each other;
- current deployment manifests — as is, without preliminary cleanup;
- load metrics for several weeks: CPU and memory utilization, request count, queue lengths;
- target indicators: response time under peak, allowable downtime, recovery time after failure;
- schedule of peaks — campaigns, hours of highest activity, batch jobs;
- cluster boundaries: how many nodes are available, what quotas, whether automatic node addition exists;
- change windows and accepted release rollback procedure.
Our side at this stage: load profile analysis, formulation of fault-tolerance requirements, draft policies and metrics for HPA Kubernetes, implementation plan with verification on load test environment and list of failure points. We return with estimate, timelines, and list of risks.
Describe your task — we will return with estimate and plan.