Capacity Planning Implementation Based on Load Metrics

Why capacity planning breaks under real load

Capacity headroom is rarely consumed evenly.

In a cluster, some nodes carry a high share of CPU and memory requests, while others sit idle because of strict placement and anti-affinity rules: requests are reserved for a service as a whole, even though actual consumption is several times lower.

Therefore capacity planning based on averages shows "half is free," while cluster overload occurs on a specific group of nodes.

Add to this the delay of horizontal scaling — from the moment a metric rises to the readiness of a new replica, minutes pass, and all that time peak load is handled by the same set of nodes.

A peak almost never grows evenly across all metrics. One link hits the limit first: the connection pool to the database, the concurrent request limit on the external service side, disk writes, or the processing queue.

Then a cascade kicks in — request retries, increased timeouts, filled buffers — and a failure due to insufficient capacity looks like a failure of the entire contour, even though exactly one resource has been exhausted.

You can find it in advance only from the consumption profile of each service individually, not from an aggregate chart. In Kubernetes, such things are visible through limits and requests on objects, but they need to be interpreted together with real traffic.

Manual estimates lag for two reasons. They rely on average values from the previous period and miss a change in profile: a new customer scenario, a batch export, a seasonal spike.

And second, capacity is expanded after degradation has already occurred, in crisis mode, without headroom for the next peak.

That is why we start the analysis with measurements: we collect a profile by service, find the infrastructure bottleneck, and calculate what reserve is needed for peak scenarios. Next — how this is addressed.

What capacity planning based on load metrics provides

Capacity planning without metrics is a bet on the peak: power headroom is taken "by eye," and in one of two scenarios it either sits idle or runs out at the most inconvenient moment.

The danger is not load growth as such, but a failure of the "resource shortage" class: the queue overflows, response time grows in steps, and consumers get errors instead of predictable slowdown.

The foundation is regular load capture: requests per second by operation type, queue depth, CPU and disk saturation, database connection utilization, response time at the 95th and 99th percentiles, storage growth.

We store the series in Prometheus with a long retention period — this makes weeks and seasons visible, not just the last hour. From the history, two numbers are derived that are usually missing: the saturation point for each metric and the rate of approach to it.

We calculate power headroom from the saturation point, not from average consumption: for example, one third up to the threshold beyond which response time grows nonlinearly, and a forecast horizon of two to three months ahead.

The growth model is built on historical windows taking into account daily and weekly seasonality — a load growth forecast gives a specific date for reaching the threshold instead of an abstract "it will soon get tight." We plan expansion in advance, with time for procurement and testing in a test environment.

The second effect is predictable degradation.

When limits and quotas are set according to measured thresholds, overload looks like growing latency for non-priority operations rather than failure of the entire service: critical requests go through, background reports and recalculations slow down first.

Reduced downtime and resource planning come not from load having stopped growing, but from the failure point being known in advance.

Approaches to capacity planning: comparing modes

Choosing a capacity planning mode is a trade-off between forecast accuracy and the cost of error.

Manual calculation relies on peak values from past periods and built-in headroom; metric-based forecasting relies on load dynamics over several weeks; automatic scaling reacts to the current state without human involvement. Below — where each mode works predictably and where it starts to get in the way.

Pure modes are almost never found in practice: the baseline is fixed based on historical metrics, while the upper bound is handed to automation under a load forecast.

What suits a specific case depends on the traffic profile and acceptable degradation time — so we start the comparison with measurements and incident analysis, not with tool selection.

How we implement capacity planning: stages of work

Capacity planning starts not with purchasing hardware, but with an inventory of what the system is already telling about itself. When metric collection is only formally configured, forecasting relies on noise: averages hide degradation, and background tasks masquerade as peaks. That is why we start with data reliability and only then move to calculations.

1. Metric audit. We verify which counters are collected, at what frequency and retention depth, and where there are gaps between services. Labels with uncontrolled cardinality are not included in calculations.
2. Load profile. We break traffic down into daily and weekly cycles, separating user peaks from background tasks — reindexing, backups, heavy reports. Otherwise headroom is calculated for load that is not related to the peak.
3. Hypothesis validation. We compare peaks with degradation history and keep metrics that actually predict the problem. Unconfirmed hypotheses are removed from the model.
4. Threshold values. We calculate by percentiles, not averages, separately for reads and writes, with headroom for loss of a node or zone. Each threshold is tied to an action: scale up, limit, disable something secondary.
5. Run on a live profile. We compare the forecast with actuals on a historical interval, then repeat the peak scenario on a test bench. A deviation above tolerance sends the model back for recalculation.
6. Embedding and handover. We turn thresholds into alerts and autoscaling rules, fix the metric owner and review triggers. We hand over the regulation — the team runs planning on its own.

What we hand over: artifacts and working regulations

The project ends not with a report, but with a state in which your team changes capacity without us.

Therefore the handover includes not only configuration, but also the rules by which it operates: what to do when load grows, who reviews thresholds, where to look in the first minute of a failure. Below is the set that remains with the customer.

• Capacity model — load profiles by service, identified bottlenecks, headroom for each resource, and the point after which scaling is mandatory.
• Infrastructure diagrams — service topology, traffic flows, fault-tolerance boundaries, and zone separation points.
• Alert configuration — rules with thresholds for latency, saturation, and error rate, split into "wake up the on-call engineer at night" and "review in the morning."
• Capacity review regulation — frequency, owner of the numbers, metrics that trigger a review, and the form for recording decisions.
• Infrastructure documentation — descriptions of services, configuration parameters, and standard operations, sufficient for a new engineer to understand without verbal retellings.
• Instructions for the on-call shift — scenarios for typical failures: what to check, what to restart, when to escalate.
• Knowledge transfer to the team — joint reviews, meeting recordings, issued access, and validation on real tasks, not just words.

Two to three weeks after handover, we conduct a follow-up review: which alerts fired without cause, which thresholds turned out to be low, what in the instructions did not match reality — and we fix this together with your engineers.

Case: removed cluster overload and service downtime

Before: degradation during peak hours

The service ran in a Kubernetes cluster, and limits and the number of replicas were set by eye during the first launch.

During evening peak hours and mailings, node utilization reached 95%, the request queue grew faster than autoscaling kicked in, and the gateway began returning errors. Over the quarter — four incidents of 40–90 minutes each and one unplanned outage lasting several hours.

After: metrics and managed headroom

We collected metric history for several months: requests per second, response time by percentiles, CPU and memory utilization, queue depth.

From it, we built a forecast taking into account the daily profile and seasonal spikes, and from the forecast — scaling thresholds and rules for proactively adding replicas.

Headroom was made manageable: a fixed share of capacity is reserved for the forecast peak, seasonal expansion is started on schedule, before the load arrives, and load tests verify that the new ceiling is sustained.

There was never any capacity-related downtime, peak-hour incidents stopped, and node utilization at peak remains below the degradation threshold. Expansion stopped being emergency work — it is a planned procedure with a known timeline.

How does metric-based planning differ from peak-based calculation?

Calculation by peak load starts from a single point: take the maximum recorded spike, multiply it by a headroom coefficient, and get a configuration for the entire period. The logic works as long as the peak remains representative.

In practice, it is often an anomaly — a one-time mailing, a retry storm, a scan — and it freezes resources that sit idle the rest of the time.

Metric-based planning (capacity planning) starts with the request distribution: how load changes by hour, day of week, and season, where the average is, where the percentiles are, how long and frequent the spikes are.

The model answers not the question "what if the worst happens," but the question "what share of time do we maintain the target response time under this profile." Average load sets the baseline size, variance sets the amount of headroom, and peak frequency sets the required reaction speed.

Next, the distribution is calculated by component. There is usually one bottleneck — the database, queue, or external service — and it hits its limit before the others.

Therefore the headroom coefficient is set separately for each element, based on its variance and time to scale up: with automatic scaling in minutes, only a small headroom is needed; with manual addition in hours, it is noticeably higher.

This leads to different decisions. Peak-based calculation answers "can we withstand the worst day" and almost always produces an oversized configuration that no one revisits later.

Metric-based planning answers how many resources are needed for the expected profile and at what point it is time to scale — so there is less excess, and the decision is based on data.

Technical doubts around capacity planning

Capacity planning relies on measured data, not guesses: the forecast is built from historical load series with headroom for seasonality and peaks. Hence the main doubts — how much the forecast can be trusted and where to get metrics if there are still few of them.

We have almost no metrics — where should we start?

A basic set is enough: compute resource load, memory consumption, response time, request queue depth, and the number of concurrent connections. This is sufficient to see the bottleneck and estimate headroom.

If there is no telemetry at all, the work starts with collection — a model on empty data is meaningless.

Is this compatible with our monitoring?

We do not replace the existing collection system; instead, we connect to it as a consumer of metrics in standard formats.

Dashboards and alerting rules remain in the familiar environment, access is granted according to the current role model — there is no need to duplicate agents.

How long does implementation take?

The first forecast for key services comes two to three weeks after access to metrics. The full loop with automatic recalculation and an alert threshold for approaching the limit usually takes one and a half to two months. The threshold triggers before degradation, so headroom for overload remains.

Let's discuss your task and come back with an estimate and a plan

To keep the analysis based on facts rather than guesses, send us the initial load data.

A metrics export for several weeks, a service diagram, and a description of peak windows are enough — this is sufficient to see where headroom is exhausted and which node will hit the ceiling first.

• load metrics over time: requests per second, latency percentiles, CPU and memory consumption for each service;
• description of peak windows — campaigns, releases, seasonal spikes — with dates and duration;
• service and dependency diagram: queues, external calls, shared databases, and caches;
• target indicators: what response you consider normal and what share of failures you allow at peak;
• incident history: what failed, for what reason, and how long recovery took;
• growth plans: expected traffic increase and the deadlines by which you need to be ready;
• read-only access to metrics and configurations, if you are ready to provide it during the analysis phase.

Next, we build a load model, find bottlenecks, and calculate headroom up to the limit.

The work plan includes failure points, target values for each service, the order of changes, and validation on a controlled peak in the production environment — this way capacity planning is confirmed by measurements before the real surge.

Describe your task — we will come back with an estimate and a plan, and outline the scope of work and the project launch sequence.