Configuring File Descriptor Limits for High-Load Applications

Why an application hits file descriptor limits

Every connection, open file, socket, or channel is a file descriptor taken from the process table. The table is not infinite: its size is set by soft and hard limits, which are inherited from the parent at startup.

systemd sets its own value for the service, the shell sets its own via ulimit, and worker processes often get a limit different from the one configured “on paper.”

While a service runs in a calm mode, free descriptors do not run out — the limit becomes visible only under load. Every incoming connection, cache access, log write, or queue write takes one more.

If connections are not closed in time — there is no limit on hold time, error branches are lost — a descriptor leak occurs: the counter grows monotonically and does not return to the baseline.

In logs it looks the same: the error “too many open files,” refusal to accept a new connection, while the process is alive and health metrics respond. Outwardly — service failure under peak load: the load balancer sees an open port, but requests are not served. Clients go into retries and finish off the remaining free descriptors.

The peak is dangerous because unclosed connections and retries add up: each new connection requires a descriptor, and they are released more slowly than they arrive. We break down how this is addressed — from limit audit to leak control.

What configured limits provide: fewer failures and less downtime

An unconfigured limit manifests not at startup but at peak: the worker process hits the soft limit, new connections are rejected, the share of errors in logs grows — while the service responds to availability checks and looks alive.

Diagnostics drift toward the application and network, although the cause is one parameter that no one set for the real load profile. We break down what changes when it is set.

Limits are set explicitly — soft and hard separately, tied to the service itself, not to an administrator’s interactive session. In systemd, this is LimitNOFILE in the unit file; for processes outside the service manager, rules in limits.conf.

Otherwise, after a restart the node returns to the default value, and the incident repeats in the same place.

The effect is measurable along three axes. Reduced failures are visible by the share of unsuccessful connections and the return code on accept — it goes to zero within the tested headroom.

Service recovery time after peak is reduced: there is no need to wait for descriptors that processes hold in waiting to be released. Resilience under load stops depending on luck — the connection queue does not grow avalanche-like, degradation is smooth and clears on its own.

Predictability comes from the limit being described in the repository and rolled out identically to every node: a new instance in the cluster behaves like the others.

We confirm headroom with a load test run using a profile close to peak, and keep the threshold above the calculated maximum by a clear factor. Downtime prevention rests on this headroom, not on round-the-clock on-call.

Limit configuration options: what suits the load

The descriptor limit is not set in one place: a process has a soft and hard limit, the system has a global ceiling, and services and user sessions get values through different paths.

The order of application depends on who starts the process: the shell on login, the service manager, or the parent that passed values by inheritance.

Therefore, tuning for load is a choice of level, not a search for the “right file.” Editing at one level does not affect processes started differently, and the limit silently remains the same: the application hits a failure when opening connections, although everything looks correct in the settings.

The result is checked against the actual value in the running process, not the config file: restarting the service and a new session give a different picture. For a loaded profile, the system ceiling is also raised — otherwise the service hits the global limit before its own.

How implementation proceeds: from audit to policy handover

File descriptor limits are inherited by a process at startup, so editing configuration “live” changes nothing, and a value set with a large headroom does not solve the problem — it only delays failure and masks a leak. We work in a fixed order: first measurement and scheme, then changes.

1. Configuration audit. We collect actual values across all layers: kernel limits, system thresholds, parameters of the service that starts the application, and user session limits. Discrepancy between layers is a typical reason why the process sees something different from what was written in the file.
2. Current state measurement. We look at how many descriptors the process holds at rest and at peak, how the number of open connections and files grows, and what remains in logs after previous failures. This gives a starting point, not a guess.
3. Configuration scheme. We determine exactly where we edit, which processes get exceptions, and how the value applies to both the service and manual launch. We separate soft and hard thresholds and fix headroom relative to peak consumption.
4. Configuration. We make edits to the configuration and service definition, restart according to plan rather than during an incident, and verify that the new value is actually picked up by the process.
5. Load verification. We reproduce the production profile: peak traffic, connection spike, failure of an adjacent node. We separately look at application behavior as it approaches the threshold.
6. Observability. We set up metrics for the number of open descriptors and connections and an alert on growth rate — then a leak is visible long before the process hits the ceiling.
7. Policy handover. We provide a document with target values, the application order on a new node, a post-change verification checklist, and a “symptom → what to check” table.

After handover, the customer’s team can roll out limits on new servers itself and understand when descriptor growth is normal and when it signals a problem.

Configurations, schemes, and documentation based on the work results

Limit configuration is not considered complete while it lives only in the command history on a specific server.

We leave your team a set of artifacts with which an engineer can reproduce the same configuration on another environment, explain the chosen values six months later, and understand where to look for the cause if the counters hit the ceiling again.

• Configuration files with separate soft and hard values for system services and user sessions, with comments: where the headroom came from and what happens if it is reduced.
• Application startup profiles — exactly where the limit is raised: in the service definition, launch wrapper, or container environment, so the value is not lost on restart and deployment.
• Infrastructure diagram with node roles and a note on which machines have raised limits and which are left at baseline and why.
• Access policies: which roles can change kernel parameters and restart services, and how this relates to configuration rights.
• Maintenance policy: how to check the actual value of a running process, which counters to watch as load grows, and at what frequency.
• Target value calculation: peak concurrent connections per process, number of workers, service connections, and recovery headroom.
• Acceptance checklist after application and rollback procedure to previous values if the new configuration has an unexpected effect.

Plus a memo for the on-call shift: typical symptoms of hitting the limit and first-line actions before escalation. All of this is handed over in your configuration repository, not in separate emails.

Case: eliminated service failures under peak load

How it was

At peak, the service stopped accepting new connections: in logs — failure to open a file or socket; outwardly — a surge of errors and connection breaks.

The on-call engineer saw the symptom, not the cause: metrics tracked response codes but not the number of open descriptors, so every investigation began with manual log review. From first signal to recovery, tens of minutes passed, and all that time some requests were lost.

What changed and what we got

First we took a baseline: how many descriptors the service holds at peak and how many it opens per request.

It turned out that the soft limit ceiling was fully consumed even during normal operation — there was no headroom, and since the limit is set per process, with several worker processes the ceiling is consumed multiple times faster than expected.

We raised the soft and hard limits for the process, separately the system-wide ceiling, adding headroom for growth, and eliminated the leak: connections return to reuse instead of remaining open until restart.

We shifted observability to the cause. The number of open descriptors and its share of the limit are plotted next to traffic in Prometheus and on shared dashboards, and the alert triggers as the ceiling is approached, not after failure.

After the changes, peak passes without this class of failures: limit usage stays within two-thirds instead of hitting the limit, diagnosis time dropped from hours to minutes, and manual service restart under load is no longer required.

How the system limit differs from the service limit

The file descriptor limit is set at two different levels, and confusing them is the source of most “mystical” failures. The system limit is the total ceiling for the entire machine, maintained by the kernel.

The process and service limit is how many descriptors a specific process can open; it is always lower than the system one and is inherited at startup.

Priority is determined by who spawned the process: a child process does not reread the configuration but receives values from the parent, so the result depends on the launch chain.

Key nuance: services started by the service manager do not go through PAM, so edits for user sessions do not affect them — hence the classic discrepancy “one limit in the terminal, another under the service.” Within one level, the soft limit is the working value and the hard limit is the upper bound: the soft one can be raised, but not above the hard one, and the hard one not above the system one.

We check the actual value via /proc/<pid>/limits for the specific process, not via configuration files. Application order: system ceiling → unit or parent parameters → inheritance by child processes → the value the process actually works with.

Common technical concerns when configuring limits

A discussion of limits almost always comes down not to the question “to raise or not to raise,” but to the engineering team’s caveats: what risk this poses to a running service and how to prove afterward that things got better. Below are the points that most often come up during approval.

Behavior under load deserves separate mention: at average traffic, the headroom looks excessive, but at peak the same process hits the limit because some descriptors are held by connections and temporary files. Therefore, we calculate from the peak profile, not the daily average.

Won’t the new limit mask a real leak?

There is a risk, and it is the main one. If descriptors are not closed in handlers, a high limit will simply delay failure.

Therefore, before the change we take a snapshot of open descriptors across several processes: growth tied to request count is a leak, a flat profile is lack of headroom. We do diagnostics before edits, not after.

How can we be sure the headroom is actually needed?

Through monitoring: we collect the number of open descriptors per process, the usage share of the limit, and the open failure counter.

We set an alert at 75–80% of the limit so there is headroom before degradation, and compare it with request history — this shows whether consumption grows over time or hits seasonal peaks.

Does this conflict with the current configuration?

Limits can be set in several places — system-wide, in the service definition, in the container platform. What applies is not the largest, but the one set later in the launch chain, so we consolidate values to a single source and document them. Otherwise, after restart the behavior will differ from what was tested on staging.

Let’s discuss the task and come back with an estimate and work plan

Estimating file descriptor limit configuration rarely requires a long investigation — but the scope of work and implementation plan directly depend on how fully the environment is described. Below is input data sufficient to come back with timelines and change order.

• Service composition: which processes hold open connections and files — web servers, queue workers, proxies, databases — and how many instances run on one node.
• Load profile: peak number of concurrent connections, daily profile, how the system behaves at peak.
• Current settings: soft and hard limit for user and process, restrictions in the service unit, system-wide open files limit.
• Failure symptoms: error texts from logs, when they appear, what exactly crashes and restarts.
• Launch method: who initiates processes — service, supervisor, or container — and whether limits are passed inside.
• What has already been changed: which values were edited and why the result was unsatisfactory.
• Environment constraints: containers, orchestrator, whether there are rights to change system parameters.

With this set, we discuss the task and fix agreements: which values to set, how to test under load, and in what order to roll back.