Right-sizing virtual machines: where resources sit idle
On almost any infrastructure older than a year, the same picture is visible: resource profiles were set “with headroom for growth” and have not been revisited since.
Peak requests were closed by adding cores and memory, and no one removed them afterward — that is how idle resources appear, reserved for a scenario that never materialized.
The second cause is load profile skew. The virtual machine template is one, but the roles behind it differ: a service that keeps state in memory and a service that computes get the same set.
As a result, one machine consumes allocated memory and barely touches compute, the neighboring one hits the ceiling, while average utilization across both nodes looks completely normal.
The cost of skew is double. First, we pay for reserved volume, not consumed volume: idle machines keep utilization around a few percent, and this is not a one-time overpayment but an ongoing one.
Then resource shortage arrives at peak — where the profile does not match the real nature of the load, the machine degrades: request queues and response time grow, and failures appear on the peak wave. The response to failure is usually the same — add more — and the skew becomes permanent.
Right-sizing virtual machines begins not with downsizing, but with capturing the actual profile: how the machine consumes memory, compute, and disk operations over a day and a week, where it has peaks and where it has a flat plateau. Next, we examine how this is addressed without risk to peak scenarios.
What aligning the load and resource profile delivers
We capture the load profile over a window that covers both peak and trough: memory and CPU utilization, queue depth, number of resource-shortage failures.
We look not at the average but at the distribution — it immediately shows that the machine is sized for a rare spike and sits idle most of the time.
Prometheus provides these series in a form suitable for review, so right-sizing virtual machines here is based on facts, not feelings.
After alignment, a whole class of failures goes away: incidents due to memory shortage at peak, response degradation under load, crashes during parallel queue processing.
The peak no longer hits the ceiling of the allocated resource — the ceiling is selected specifically for it, not based on last season’s data.
The volume of resources kept in reserve decreases: some machines are downsized, some move to another class. The reserve does not disappear — it moves into autoscaling rules that add capacity at a threshold and remove it when the load subsides.
Downtime and recovery time fall for two reasons: typical resource incidents simply do not occur, and if a failure does happen, the on-call engineer sees a clear picture — which parameter hit the limit and what to do about it. Recovery from a template takes minutes instead of dozens of minutes of manual diagnosis.
Options for bringing virtual machine profiles back to normal
Reducing the profile to the actual load can be done in different ways, and they are not interchangeable: each has its own cost of error and reaction horizon.
First, we fix the boundaries — resource reduction is limited by peak windows, headroom for spikes, guest system overhead, and the minimum profile specified by the application vendor in the documentation.
| Option | When it fits | Limitations |
|---|---|---|
| Manual profile review based on metrics | Stable load, fleet of up to dozens of machines, infrequent changes | Requires analyzing history over a full cycle; the result becomes outdated between revisions and depends on who reviews it |
| Automatic recommendations based on telemetry | Large heterogeneous fleet, no resources for ongoing manual analysis | Advises but does not act; quality depends on the depth of history and the presence of an agent inside the guest system |
| Flexible scaling by rules | Variable load with predictable daily and weekly peaks | Does not save from slow growth of memory consumption; requires correct node drain from service |
In practice, approaches are combined: recommendations provide candidates for reduction, manual review confirms them, and flexible scaling covers peaks.
The key rule is to reduce in steps, no more than one grade at a time, with an observation window no shorter than a full business cycle and retained ability to return to the previous profile.
Work sequence: from metric collection to policy handover
The work sequence is built so that every change relies on a measured profile, not on the feeling “this machine clearly has too many resources.” The implementation stages proceed sequentially: first data, then decision, then validation — and only after that, policy handover.
- Fleet inventory. We consolidate machines, their purpose, owners, and tags into a single registry. We immediately mark duplicates and environments where load is effectively absent.
- Load profile capture. We accumulate metrics for 2–4 weeks: median, peak percentiles, daily and monthly spikes, memory consumption, disk I/O. We record exactly what hits the ceiling.
- Target configuration scheme. A “current → target” table with headroom for peak and justification for each class. We coordinate with service owners rather than changing silently.
- Changes. We proceed in groups: first non-production environments, then production — with a maintenance window, rollback point, and stop criterion.
- Validation under load. Synthetic test plus observation of real traffic after switchover. We compare the new profile with the captured one: if peak again hits the ceiling, we adjust the configuration.
- Policy handover. We provide a document: how to read metrics, how often to review configurations, and by what rules new machines are created.
The outcome is not a one-time cost-saving campaign but a working rule: the fleet is reviewed based on metrics, so configurations do not “drift” back after a few months.
Policies, schemes, and procedures: what we hand over to the team
Engineering work ends not with a report but with a package that the team can use to act independently. Below is what remains with you after profile review: from the fleet map to the rollback procedure if the new size turned out too tight.
- Fleet map — for each machine: group, role, current profile, actual utilization over the observation window, and headroom to the ceiling.
- Resource profiles by group — standard sets for the web tier, message queues, databases, and build nodes, with justification for the selected headroom.
- Environment schemes — what is hosted where, how environments are connected, and where fault-tolerance boundaries run.
- Review policies — how often and by what triggers the profile is checked: load growth, major release, seasonal peak.
- Rollback procedure — how to roll back to the previous size, what to check afterward, and who confirms the decision.
- Change policy — who approves the new value, how the reason is recorded, and where the decision history is stored.
- On-call documentation — machine profiles, alert thresholds, short instructions for degradation cases.
We hand over the package in your format: in a repository, knowledge base, or internal portal. It is updated together with the infrastructure, so a new engineer enters the loop by document, not by colleagues’ stories.
Case study: profile review in a cluster with peak load
A cluster for a high-load service lived with profiles chosen by eye: each service was given memory and CPU headroom for the worst-case scenario.
In reality, utilization stayed at 10–15%, and excess reserves took more than half of the allocated volume — while during peak hours some services still hit limits and added latency. Unavailability for the quarter reached 0.4%, and almost all those minutes fell on peak.
What we changed and what we got
First, we captured the consumption profile over a full cycle — a work week and peak days, using median and upper percentiles of CPU, memory, and disk operations.
Then we divided services into three classes: steady baseline, peak services with an upper bound and vertical autoscaling, and a separate pool for batch jobs, which we moved out of the shared loop into a nightly window.
We separated requests and limits so that the Kubernetes scheduler placed pods according to actual demand, not maximum appetite. Changes were validated with a load test using traffic from the heaviest day — before production rollout.
After the review, excess reserves shrank from 60% to 15% of allocated volume, average utilization rose to 40–45%, and peak metrics did not degrade. The 95th percentile of response time during maximum hours even decreased: services stopped competing for memory.
Unavailability in the next quarter fell to 0.05%, and memory-shortage incidents stopped. Rollback remained cheap: old profiles were retained as the previous release configuration, and returning took minutes.
What to do if the application is already deployed under load
Right-sizing on a running system is done in parts. We do not touch the entire loop at once: we choose one component, move it to the new profile, observe for several days, and only then take on the next.
Then a calculation error costs not downtime but one restart of an instance removed from load balancing.
We calculate the target profile from consumption metrics over a long period, not from a one-time measurement. We account for daily and weekly peaks and separately for rare spikes: reporting period close, nightly recalculations, seasonal campaigns.
The profile is selected for peak with headroom, not for average: it is average that turns into memory shortage at the most inconvenient moment.
The main risk is not the review itself but loss of headroom. A machine that had twice as much memory as needed does not reveal this until a new data stream appears. I/O can also become a bottleneck: we cut CPU but hit disk operations.
Therefore, before each change we record a checkpoint: disk snapshot, previous profile, set of “before” metrics. Then an observation window — if response or queue length goes beyond the corridor, we return to the previous size. Rollback takes minutes and does not require rebuilding the environment.
The order of traversal is from less critical services to the core. We start with components that have a duplicate instance, and only after confirming stability do we touch what carries the main load.
Technical objections: risks of resource reduction and how they are addressed
Objections in reviews are usually the same: if we cut resources, production will fall over at peak.
They are addressed not with promises but with data: the calculation relies on long-term consumption metrics — CPU load, memory, disk operations, network — taking into account daily and weekly peaks, not average values.
The second safety loop is reversibility and boundaries. The change is applied to one machine or a small group, the observation window is fixed, and rollback is a return to the previous profile from the template, not manual editing on a live system.
How to make sure resource reduction is safe?
We look at upper values and duration of load retention, not the average. If consumption spikes briefly, we keep headroom: we reduce what is idle, not what runs at the edge.
What happens if load grows after the change?
Autoscaling rules and a threshold at which resource is added remain on the group. Exceeding the threshold alerts the on-call engineer — load growth does not go unnoticed.
How often should configurations be reviewed?
On a schedule — once a quarter, plus unscheduled after a major release, architecture change, or noticeable shift in the request profile. We fix the review schedule so profiles do not become outdated silently.
Where does automation end?
We automatically apply only changes that have passed metric validation and have a ready rollback. For critical nodes and databases, we coordinate the decision with the service owner — there the cost of error is higher than the benefit of reduction.
Let’s discuss the task: we will collect the profile and return with a plan
So that right-sizing does not remain a one-time snapshot, we start with inventory.
We need a minimum from you: consumption exports for each machine over a full load cycle, not a quiet week; a list of services with a note on what is critical and what can tolerate a restart; windows in which changes are allowed; and the contact of the engineer who confirms edits for their group of machines.
If there is access to the monitoring system or hypervisor, we will collect the exports ourselves; there is no need to prepare them in advance.
The first step is ours: we compare peak and median load, filter out one-time spikes and keep recurring ones, mark machines with headroom for seasonality, and calculate the target size. Then we return with an estimate and work plan:
- a table for each machine: current size, actual consumption, target size, and the rationale for it;
- a list of machines that should not be touched, with the reason — there headroom maintains production stability;
- the order of changes by groups, with windows and rollback points to the previous size;
- forecast of the effect: how many resources are freed and how much the load spread between peak and idle is reduced;
- fleet review timeline — how often we return to the profile and what events trigger an unscheduled pass: load growth, release change, seasonal peak;
- role distribution: what your side confirms and where we act ourselves.
Describe the task in two or three paragraphs — based on this data, we will return with an estimate, work plan, and implementation procedure.