CPU, memory, and I/O limits: configuring cgroups v2 on Linux

cgroups v2 and resource limits: where production breaks

In a container environment without resource limits, a node survives until the first spike. A service with no memory ceiling grows to the kernel limit during a leak, pushes neighbors into swap, and the scheduler starts spending time reclaiming pages instead of doing useful work.

Degradation under load spreads across the entire node — including processes that have nothing to do with the culprit. This is the classic noisy neighbor: formally one service is at fault, but in practice the whole node stops responding.

CPU overload looks different but leads to the same place. One thread with a greedy loop consumes its allocated quota, scheduler queues grow, and API gateway response time drifts beyond reasonable bounds.

After timeouts, external systems begin to consider the node unavailable, even though the hardware is healthy and the network is normal. Diagnostics lag behind: node-level metrics show overall load growth but do not point to a specific culprit.

cgroups v2 closes this with a single hierarchy tree.

cpu.max sets the upper bound on CPU time, cpu.weight the share under contention, memory.max a hard limit with process termination, memory.high a soft threshold with throttling instead of failure, io.max bandwidth on a disk device, memory.swap.max a swap ban for services for which it is contraindicated.

PSI completes the picture: it shows how much time tasks spent idle waiting for a resource, meaning the signal arrives before the failure.

Next we look at how to configure resource limits on a node so that production does not break: which metrics to start with, what limits to set, and how to distinguish a memory leak in a service from honest load growth.

Measurable effect: latency, downtime, service isolation

While services share a common resource pool, one process with a memory leak or I/O spike takes the entire node: neighbors lose responsiveness, the kernel triggers emergency memory reclaim, and everything is already degrading.

With cgroups v2, each module lives in its own branch of the hierarchy and cannot exceed its allotted share — cascading failure as a class disappears.

Response time under load stops being a lottery. Scheduler weight (cpu.weight) and a CPU time ceiling give a service a guaranteed share even when a neighbor is loaded — it does not need to wait for a core to free up.

This is visible in the distribution tail: the slowest requests stop lagging behind the median by multiples rather than by orders of magnitude.

Memory and disk operations are limited softly. The memory.high threshold enables page compression instead of hard process killing, while io.max prevents backup or reindexing from strangling transactional requests.

Consumption becomes predictable — it can be used to calculate service placement density on a node without a "by eye" margin.

Service isolation turns a node from one common pot into a set of independent circuits: one degrades, the rest hold their response level.

Background tasks and evening peaks stop being the cause of nighttime investigations — node stability no longer depends on the noisiest neighbor.

Limit modes: hard limits, reservations, weights

A hard ceiling and a weight solve different problems, and confusing them is a source of outages.

A hard limit (cpu.max, memory.max) sets a boundary that a service will not exceed even when host resources are free: exceeding memory triggers the reclaim mechanism rather than degradation of neighbors.

A guaranteed reservation (memory.min, memory.low) works the opposite way — it protects processes from reclaim when memory pressure begins on the node.

Weight-based priority (cpu.weight, io.weight) does not limit anything: it engages only under contention and merely redistributes freed resources. Therefore weights are the right choice for batch tasks that should yield to online load but can use idle time.

A separate group is process count limiting (pids.max) and I/O ceiling (io.max): they are often skipped, yet they are exactly what holds a node together when a service exhausts its file descriptor limit or unleashes a process avalanche.

In practice, modes are combined: a critical service is given both a reservation and a ceiling, while a background one gets only weights so that it does not compete for the resource during load hours.

How we implement limits: from audit to handover

Limits set from documentation rather than from reality either strangle a service or fail to apply: the working memory set turns out to be three times larger than declared, and startup goes into throttling. Therefore we do not change settings until we have captured real consumption profiles.

1. Consumption audit. We look at where processes live in the cgroups v2 tree, what falls into the root slice, and which services on the same machine actually consume resources.
2. Load profiles. We capture a daily and weekly profile: working memory set, peak windows, number of threads and disk operations, pressure metrics during neighbor degradation.
3. Policy design. We separate soft and hard memory thresholds, set CPU weight and ceiling, I/O limits with headroom for spikes, and build the slice hierarchy.
4. Configuration. We move values into slice configuration, handle system and application services separately, and verify that startup and background tasks do not hit the ceiling.
5. Load verification. We repeat the peak scenario: throttling does not cut responsiveness, memory protection triggers predictably, neighboring services do not degrade.
6. Rollback plan. We record which parameters are returned and in what order, where the point of no return is, and how quickly to remove a limit on a specific service.
7. Handover to operations. We provide the regulation: values, metrics and alerts for approaching the ceiling, the procedure for changing limits, and those responsible for revising profiles.

What remains with you: configurations, diagrams, regulations

The result of the work is not a launched server, but a package by which your operations team runs the system without us and without verbal explanations.

Everything configured in cgroups v2 is in the repository: limits are described declaratively, changes pass validation, edit history is preserved and rolled back with one command.

• Configuration files with limits for each service: cpu.max, memory.max, memory.high, io.max — with the value and a comment stating which measurements it came from.
• Resource allocation diagram: group hierarchy, what is nested in what, where resources are reserved for system processes, and where headroom is left for peak load.
• Value sets for environments: separate profiles for production, pre-production, and local development so that testing does not repeat production limits.
• Maintenance regulation: check frequency, memory.high trigger thresholds, response to sustained consumption growth, and procedure for collecting per-group statistics.
• On-call instructions: step-by-step scenarios for typical events — a group hits the limit, a process was killed due to memory shortage, the disk subsystem slows down due to contention.
• Limit change procedure: who approves a change, how real demand is measured, how a value reaches the nodes, and within what time it is rolled back.
• Monitoring map: which group indicators are displayed on dashboards and which alerts early signs of approaching the boundary are tied to.

The package is designed for an engineer seeing the system for the first time: from it, they can investigate an incident without pulling up a chat history and hand over on-call duty shift by shift.

Case: the node stopped falling under peak load

Before: failure of one service stopped the node

One service with a slow memory leak gradually ate memory from its neighbors on the node. When free memory ran out, the kernel started emergency process termination, and it was not the problematic process that fell, but the entire node together with working services.

Load peaks came several times a week, and each time recovery took hours: manual restart, data integrity check, rewarming.

What changed

We moved the node to cgroups v2 with a single hierarchy instead of disparate controllers. We set hard limits on memory and process count, relative CPU time weights, and disk I/O quotas for each service.

The key is group termination: on memory overspend, one problematic process is killed, not the entire node. The behavior holds even when neighbors peak simultaneously.

Result

Node failure frequency dropped from several times a week to zero over the observed period, and recovery time from hours to minutes: the problematic service restarts according to the restart policy, while the others do not notice.

Metrics were collected in Prometheus — during peaks, consumption hits the set limit rather than growing until exhaustion.

The failure class "a leak in one service takes down its neighbors" is closed, and the limits live in configuration and are revised without rebuilding images.

What to do if limits are already configured in a previous cgroups version?

Moving policies "as is" will not work: in the previous branch, each controller lives in its own hierarchy, while in the new one there is a single tree and its parameters are combined.

cpu.shares gives way to cpu.weight with a different value range, quota and period merge into cpu.max, disk limits into io.max.

This means migration is a recalculation of values and explicitly enabling the required controllers in cgroup.subtree_control, not replacing names by a template.

Backward compatibility is provided by mixed mode: some controllers remain in the old hierarchy, some move to the new one.

This lets a node survive a reboot in a convenient window, but it is not suitable as a permanent scheme — the process begins to be accounted for in two trees, and a controller handed over to the old hierarchy is not returned back.

Migration without downtime is built on gradual enablement: first one node, then the rest one by one. While a node is removed from balancing, load moves to its neighbors, and after a restart, services under systemd come up in the new tree. At each step, only one node is changed, and rollback costs one restart.

Post-migration verification is not just "the service started." Check that controllers are active and limits have applied to processes, collect pressure metrics memory.pressure, cpu.pressure, io.pressure — in the previous branch they simply do not exist. Then a load profile close to production confirms that the restrictions do not cut useful work.

Common technical doubts and objections

Before starting cgroups v2 limit work, engineers and managers ask the same questions: will we throttle load, how reversible are the settings, and who is responsible for system behavior after handover. Below are direct answers without generalities.

Are the settings reversible if something goes wrong?

The parameters are hierarchy files, and we record their state before changes. Rollback is returning several values; no reboot is needed.

We store changes declaratively in the configuration management system, so every step is reproducible and repeats on another machine without manual work.

How can we tell that the bottleneck is not in the limits?

Before changes, we collect metrics: forced throttling time, memory events, I/O statistics, resource pressure indicators. If throttling is not growing but latency exists, the cause is disk, network, or the code itself, and the limits did not create it.

Is this compatible with containers?

Container environments and orchestrators work with cgroups v2 through a system driver, so our rules and limits from manifests apply consistently. There are no conflicts: the stricter limit at the service level applies.

What happens after the work is handed over?

We stay in touch according to the agreed regulation: we investigate incidents related to limits, recalculate parameters as load grows, and update documentation. We hand over monitoring dashboards, alerts, and a step-by-step rollback instruction.

Let's discuss the task: where to start and what we need from you

To come back with an estimate and a plan rather than clarifying questions, a few inputs are enough for us. The more precisely you describe the current state, the less time reconnaissance takes and the faster work begins.

• What is placed on the node: how many services, which are latency-critical, and which can tolerate queuing.
• How limits are currently set: hard limits, weight coefficients, or processes living without boundaries.
• Symptoms: latency spikes, emergency process termination, growth of the disk operations queue.
• Platform: kernel branch, distribution, whether there is unified group management via systemd or an orchestrator.
• Constraints: maintenance windows, continuity requirements, who grants access to nodes.
• Who on your side makes decisions and verifies the result after changes.

Then the procedure is simple: we discuss the task, fix target indicators for CPU, memory, and disk operations, and agree on a phased work plan with checkpoints.

The first next step is to collect group profiles on a test node; then we set limits and run the configuration under load. If the metrics are not met, we return the previous set — rollback remains in your hands.

You choose the engagement format: one-off work on a narrow area or ongoing support with profile updates as load grows.