Setting Up NUMA Awareness for Multiprocessor Servers

NUMA awareness: why a server loses performance

A modern multiprocessor server is not one large machine, but several compute nodes, each with its own local processor memory. As long as an application runs within a single node, memory access takes a short path, and latency is predictable.

As soon as a thread starts reading data from another node, the request goes over the inter-processor bus: latency increases severalfold, and bandwidth is shared among everyone who accesses the same place.

The catch is that the operating system by default distributes memory “fairly”: the node of a page is determined by the first access, and the scheduler may move the process to another core.

The result is degradation under load that is almost invisible on a test bench — with a dozen connections everything is smooth, but at peak response time grows in steps.

The cause is sought in the application code, disk, or network, although the culprit is memory topology, which no one agreed on in advance.

NUMA awareness is the coordinated configuration of the scheduler, virtualization tools, and the application itself, in which computations stay close to their data.

Threads are pinned to nodes, memory is allocated on the required node, and inter-processor traffic becomes a deliberate exception rather than a constant background.

A separate part of the work is page migration: automatic balancing sometimes helps, and sometimes endlessly moves the same pages between nodes, consuming exactly the resource that was meant to be saved.

Next, we discuss how we capture the actual topology of a specific server and what we change in the configuration so that response no longer depends on which node a thread lands on.

The effect of configuring memory placement on a multiprocessor server

The effect of tuning is visible not in average response time, but in its spread and in how the service behaves at peak.

While processes and their memory pages are distributed across all nodes, some requests go to a remote memory controller — the average remains acceptable, so the problem goes unnoticed for a long time, while the tail of the distribution reveals it immediately. Below are three slices by which we measure the result.

Response time spread

As long as the application gets memory from any node, request latency depends on where the scheduler placed it — hence the jagged tail of the distribution and a 95th percentile that is several times higher than the median.

Pinning processes to nodes and a local page allocation policy compress the spread: requests that were served 3–5 times longer than the rest fit into a narrow corridor, and adjacent measurements stop diverging.

Stability and even distribution of threads under load

Threads are distributed across cores within a node, and each works with its own local memory — contention for the inter-socket bus stops being a bottleneck.

Throughput scales more linearly: when the number of concurrent tasks doubles, response time increases noticeably less, not proportionally. For the service owner, this looks like “the service stopped sagging during peak hours,” even though the application code did not change.

Eliminated class of failures

The most expensive effect is that a class of failures disappears in which a process does not get local memory: the node spends resources on page eviction, and requests wait not for computation but for memory access.

The share of such failures drops to zero, and we lock the topology and policies in the configuration — the platform vendor’s recommendations (Intel) for memory placement across nodes are transferred into templates so that response predictability is maintained as load grows and new services are added.

Approaches to NUMA awareness: comparing modes and limitations

On a multiprocessor server, accessing memory of its own socket is cheaper than accessing memory of the neighboring one, and the difference shows up in tail latencies, not in average response.

Who divides pages between nodes — the kernel with automatic placement or you manually — is determined by policy. There are four realistic options, from completely skipping tuning to hard pinning.

We compare by the criteria that usually decide it: response predictability, resilience to growth in the number of threads, maintenance cost, and behavior when hardware changes.

The order of work is from the weak mode to the strict one: first a soft policy and measurements, then process pinning, and only then hard pinning. If you start with the last one, you get the opposite problem: nodes are underloaded, free memory exists, but it is unavailable.

Implementation order: from topology diagram to handover of the operating procedure

Implementation proceeds in stages: each change is confirmed by measurement and remains reversible. We start with a topology audit — without a map of nodes, cores, and I/O device affinity, any edit turns into guesswork.

Then we transfer the findings into a placement scheme, configure, verify under load, and hand over the operating procedure.

1. Topology audit. We collect a map of nodes and cores, the links between them, and the affinity of network cards and disk controllers. We record baseline measurements: latency of access to local and remote memory, and the distribution of threads across nodes.

2. Placement scheme. We determine which services we pin hard to nodes, which get only preference, and which we leave free. Separately, we spell out where isolation is needed — otherwise soft pinning will imperceptibly become hard.

3. Configuration. We transfer the scheme into configuration: rules for process and interrupt affinity, memory distribution, scheduler parameters. Each change is a separate step with a rollback point.

4. Load testing. We run a profile close to production: mixed requests, peak bursts, long-running background. We look not only at average throughput, but also at tail latencies.

5. Deviation review. We compare against baseline measurements and explain every discrepancy: where locality helped, where pinning narrowed resources. We remove disputed rules rather than keeping them “just in case.”

6. Operating procedure. We hand over the node map, current affinity rules, the list of things that must not be changed without re-measurement, and the verification procedure after a kernel update or hardware replacement.

What the customer receives: diagrams, profiles, procedures

After NUMA awareness tuning, the work ends not with a report, but with a set of artifacts that your team lives with afterward — without us.

Everything is stored in a configuration repository and in the server documentation; the structure and names are agreed with your engineers at the handover stage.

• Topology map. A diagram of memory node and core placement: you can see which cores go to the neighboring node for memory and where throughput is lost. It is used to plan the placement of new services without repeated measurements.
• Affinity profiles. Ready-made sets for the database, cache, and balancer: which processes are pinned to which cores and at what thread count. The profile is applied as a template rather than assembled anew.
• Access policies. Who is allowed to change affinity, which edits require a test bench run, and what must not be touched on the production server. This closes failures of the “someone adjusted the cores by hand” kind.
• Server documentation. Node passport: state before and after tuning, reference measurements for memory and latency, and the relationship of configuration to hardware composition and system modes.
• Maintenance procedure. The order of planned work: how to save and restore profiles, what to check after hardware replacement, and the procedure for accepting a new server into the same profile.
• Verification scripts. An automated test that affinity has been applied and holds after service restarts, with a one-page report for the on-duty engineer.

Case study: how response degradation was eliminated on servers with two processors

The client is a service where each server has two processors, and the database runs under PostgreSQL.

At peak load, some requests began responding several times longer than others: processes went for data into the neighboring processor’s memory, and the service queue grew faster than the load itself.

What the measurements showed

The share of remote memory accesses reached 34%, and latency was concentrated specifically in those requests: the spread in response time between the fastest and slowest operations differed by almost an order of magnitude.

When reaching peak, the service did not slow down evenly but began dropping some requests due to timeout — that is, response degradation looked like a cliff rather than a smooth decline.

What was changed and what was achieved

We pinned the database processes and the connection pool to local memory nodes, set a page placement policy, and disabled automatic migration, because of which pages moved between processors at the most inopportune moment. Additionally, we separated services competing for the same memory onto different nodes.

The share of remote accesses dropped to 3–4%, response time spread decreased by about six times, and peak throughput almost doubled — without replacing hardware.

The peak response degradation that the project was launched for is no longer reproduced either on the test bench or in the production environment.

What to do if the server is already configured manually and without documentation?

An inherited configuration is almost always multilayered: some affinity settings are defined by kernel boot parameters, some in systemd units, some by commands that were once executed manually and never recorded.

The first step is to change nothing and capture the current placement scheme as it is: which processes are pinned to nodes, how interrupts are distributed, and what the kernel actually sees, not what is written in old notes.

The snapshot is taken before any edits and saved as a reference for comparison: information about topology and current node weights, a list of per-process affinities, processor group settings, and kernel boot parameters.

Alongside it are locality metrics: the share of accesses to a “foreign” node and the execution time of tasks that do not fit into local memory. Without this reference, it is later impossible to prove that things got better rather than merely different.

Separately, we analyze what exactly the manual configuration was compensating for. Often it treated a symptom — interrupt skew toward one node or a poor node choice by the scheduler — and when affinities are removed, the defect returns.

Therefore, verification before changes includes checking the intended against the working: a rule may not apply at all if the process is restarted from a different unit or the container ignores the inherited mask.

Backward compatibility rests on the order of work: we change one layer at a time, starting with the one least related to load, and keep a ready rollback to the captured snapshot.

The pilot node comes first, where the same locality and latency metrics are compared under a control load. Only after a stable result is the rule transferred to the remaining machines.

As a result, the configuration turns from a set of manual guesses into a described and reproducible one: the placement scheme is documented, every change is compared against the reference, and a return to the previous state is possible at any moment.

Technical concerns: compatibility, risks, limitations

Pinning processes to local memory nodes works both on a physical server and inside a virtual machine: the hypervisor publishes the topology to the guest, and the guest kernel passes it further — to resource managers and containers.

The boundary of applicability is simple: if VMware or another hypervisor masks the topology or itself spreads the load across all cores, the gain inside the guest will be noticeably smaller than on “bare metal.” Therefore, we start by checking: does the system see several nodes, and does their numbering match the real sockets — if not, any pinning sends processes not where intended.

The impact on already running services depends on the configuration method.

Changing the memory distribution policy at the kernel level applies to new processes and does not touch current ones, while pinning through a resource manager requires a careful restart — we allocate a window for it and test on one cluster node, comparing latencies before and after.

Every change is made reversible: a profile is disabled with a single configuration edit, and the service returns to its previous mode without rebuilding the environment.

Will running services have to be stopped?

Not always. Policy changes at the kernel level are picked up by new processes, and the service continues running while gradually transitioning to the new scheme.

A restart is needed where memory management goes through a resource manager — such work is scheduled in a maintenance window.

How are changes rolled back if things get worse?

The configuration is stored in a version control system; rollback means restoring the previous profile and restarting the affected services.

Before rolling out to the entire cluster, we keep reference measurements so we can compare behavior after rollback and confirm that we have returned exactly to the original state.

Let’s discuss the task: send the server diagram and load profile

To understand what NUMA awareness will give specifically on your servers, access to the production environment is not needed — source data about the hardware and load profile is enough.

The more complete the picture, the fewer assumptions in the estimate and the more realistic the timeline. Here is what is useful to attach to the task description.

• output of lscpu and numactl --hardware from each server — node topology, memory layout across channels, distances between nodes;
• service placement diagram: what runs in containers, what runs in virtual machines, where migration is enabled, and how limits are configured;
• load profile — which processes are latency-sensitive, how load changes by hour, where the peaks are, and how long they last;
• current affinities and kernel parameters: unit files with CPUAffinity, NUMA balancing policies, scheduler settings;
• metrics under load: local memory misses, share of remote node accesses, service response time at peak;
• incident descriptions: at what point response grows, what has already been tried, and what had an effect;
• maintenance windows and rollback requirements — whether nodes can be rebooted, whether there is hot standby.

Send what you have: the full set is not required; we will list what is missing separately and suggest how to capture it.

After reviewing the diagram and load profile, we build a model of process and memory distribution, note where pinning will provide a gain and where it will hit limits, and come back with an estimate and plan — stages, expected effect on latency and utilization, and timeline. Describe the task, and we will discuss where to start.