Load testing: where infrastructure breaks first
Load testing of infrastructure answers not the question “how many requests per second the service can handle,” but the question of what exactly fails first.
The database connection pool is exhausted before CPU time runs out: the application starts waiting for a free slot, response time grows nonlinearly, and outwardly it looks like “the service is working, but slowly.” Failure points under load almost never coincide with what monitoring shows in idle mode.
The second typical layer is disk operations and queues. Log writes, synchronous disk flush, task accumulation in background worker queues: while headroom is large, everything looks steady, but as it approaches the throughput limit, the queue grows faster than it can be drained.
Degradation under peak load proceeds as a cascade—a slow disk slows down the worker, the worker holds a connection, the connection exhausts the pool.
The third layer is limits of external services: call quotas, concurrent request limits, timeouts on the partner side.
The bottleneck in the environment is often exactly there, beyond our control, and without measurement it is invisible: the service responds quickly until the limit is reached, and after that, some requests are rejected.
Therefore measurement is structured as a search for throughput limit with gradual load increase, not as a single run at maximum. We explain how to set up such measurement and what it shows about your infrastructure headroom.
Measurable effect: failure rate, capacity headroom, recovery time
We record the effect of load testing not by average CPU utilization, but by the behavior of the environment at the limit. We raise the load profile in steps with a constant request rate: steady flow on wrk2 shows where response time starts to grow, not just where throughput ends.
We capture each position before and after changes on the same profile—otherwise the comparison proves nothing.
- Failure rate under load — timeout errors, dependency failures, and responses marked as overloaded are counted separately. The class “the service went down entirely under peak” is no longer reproduced after configuring limiters and queues.
- Capacity headroom of the environment — it is known at what multiple of production traffic response time starts to grow and where the emergency threshold lies.
- Service recovery time — failure run: how long it takes to return traffic, what steps the on-call shift takes and what it checks afterward.
- Predictable scaling — the point for adding instances is tied to a specific metric, not to the on-call engineer's intuition.
- Eliminated failure class — we record exactly which failure scenario has been closed and which checks will prevent it from returning.
For each item, the report retains before-and-after graphs, thresholds, and run conditions. The on-call shift relies on them during the next traffic increase, not on guesswork.
Load testing formats: modes, test benches, limitations
The load profile defines not only the scenario—the read/write ratio, peak duration, behavior under failures—but also where a run can even be launched.
The format is chosen along three axes: isolation from the production environment, measurement reliability, and the cost of error if the test affects live users. Below are the options that are actually chosen between, and the limitations of each.
| Variant | When it fits | Limitations |
|---|---|---|
| Isolated test bench | Hypothesis testing and regression after code or configuration changes | Does not reproduce network latencies, load balancer behavior, and the profile of neighboring services |
| Production environment replica | Realistic measurements are needed before rolling out to production | Diverges from production in data volume and traffic pattern, requires ongoing maintenance |
| Run on production environment | Once, in a low-load window, for capacity validation | Risk to users, no repeatability, measurement mixed with real traffic |
| Continuous load monitoring | Low-intensity background runs, canary measurements after release | Weak signal at extreme modes, requires baseline values and automatic comparison |
The measurement method has its own limitations worth remembering when reading the report. The load generator itself consumes resources and at high profiles becomes a bottleneck earlier than the service under test: numbers drop even though the system is still holding.
The first seconds of a run are distorted by cache and connection pool warm-up, so the ramp-up segment is excluded from calculations, and average response time hides the distribution tail—percentiles are examined.
The chosen format is recorded in the procedure: profile, duration, warm-up, acceptance criterion, and test bench. Then results are comparable across runs and releases, and latency growth is visible before users notice it.
Work stages: from environment audit to procedure handover
Load testing begins not with starting the generator but with an inventory of the environment: where load balancers are located, what connection pool and timeout limits are set, how external dependencies behave.
Without this map, the run measures the limit of the generator itself or one randomly guessed parameter, not the system.
Then the work proceeds in stages—each closes a separate hypothesis and leaves an artifact that can be rechecked.
- Environment audit. We capture the connection diagram, connection limits, state storage locations; we separate nodes for isolated measurements from those checked only jointly.
- Load scenarios. We agree on profiles: steady load, stepped increase to failure, short peaks. We fix the target intensity and allowable latency at p95 and p99.
- Test data preparation. We populate the bench with a comparable volume of records and key distribution—otherwise cache and indexes behave differently than on real data.
- Bench assembly and generator calibration. We verify that the load tool does not hit its own resource limits, and add warm-up before measurements.
- Runs and observation. We capture application and host metrics simultaneously and change one variable at a time—so the bottleneck is identified unambiguously.
- Result verification. We repeat the profile and compare variance between runs: reproducibility is more important than a single nice value.
- Procedure handover. We hand over scenarios, configurations, dashboards, and the order of repeat checks after changes in the environment.
At the output, the customer's team is left with a procedure, not a one-off report: before every environment change, the run is repeated using the same scenarios, and regression is visible before going into production mode.
What you keep: configurations, diagrams, policies, reports
After project delivery, the environment remains in your repository and your infrastructure—we do not keep it “in-house” or tie it to our servers.
Below is what developers and the on-call shift use without our involvement: from ready-made load profiles to thresholds by which a run is considered failed.
- Load configuration set — parameterized scenarios for different traffic profiles: short bursts, long plateau, stepped increase. Launch with one command, without manually editing scripts for each run.
- Environment and measurement point diagram — what is measured and where: generators, load balancer, application, database, cache. It immediately shows at which link time is lost, not just the final “it got slower.”
- Load thresholds — intensity, error rate, and latency at the 95th and 99th percentiles at which a run is considered failed, and by which automatic stop is configured.
- Test data access policies — anonymized database copy, rules for updating it, and a strict prohibition on writing to the production environment during a run.
- Instructions for the on-call shift — launch order, allowed windows, what to look at during degradation, how to interrupt a run and roll back introduced changes.
- Run report template — latency graphs, error distribution, bottleneck, comparison with the previous build under identical conditions.
- Regression procedure — which scenarios are mandatory before rollout and at what result the release is blocked automatically.
All of this is described so that a new engineer on the team can reproduce the run from the documentation without asking us questions.
Case study: how degradation under peak load was eliminated
A service with daily traffic unevenness: under the evening peak, the error rate reached 6–7%, and response time grew several times above the production level.
Failures were intermittent—the environment held up due to cache and power headroom, so the on-call shift learned about degradation from support requests, not from alerts.
We assembled a reproducible load profile on wrk2—with constant intensity instead of “bursts,” so repeated runs would be comparable to each other.
The bottleneck turned out not to be in the application: under peak, the database connection pool was exhausted, and aggressive request retries finished it off and turned local latency into a cascading failure.
We increased the pool, limited retries with a delay, and moved heavy background tasks outside the peak window.
| Metric under peak | Before work | After |
|---|---|---|
| Error rate | 6–7% | up to 0.2% |
| Response time | 5–8× increase | within 1.5× of normal |
| Degradation detection | support request | alert on latency and pool saturation |
| Run repeatability | variance up to 40% | within 10% |
After eliminating the degradation, we ran the load three more times with intensity steps—the picture was reproduced, and no cascade occurred.
Alert thresholds were tied to latency and pool utilization, so the on-call engineer sees the problem before clients do, and the load profile itself remained in the regression environment and is launched before major rollouts.
What to do if the environment is running and you cannot take it down
On a live environment, a load run is no different from regular traffic—the only question is who will notice first that the system is at its limit.
Therefore the test is built around the budget above which degradation becomes irreversible: queues grow, response time crosses the threshold, and stopping takes minutes instead of seconds.
The gentle run mode is defined by three limiters: request rate, number of concurrent connections, and total duration. We keep intensity constant, without stepped ramp-up—so it is visible exactly where degradation begins, not where it has already happened.
An automatic stop is mandatory: as soon as the 95th or 99th percentile of response time exceeds the threshold, the load is removed on its own, before users notice it.
Some runs are more appropriate during a maintenance window—there the question of impact on people is removed, intensity can be increased, and behavior during node failure can be checked.
On live traffic, only a small share remains: one node from the group temporarily removed from balancing, or a small share of requests through a separate route.
Before that, the environment is run on a copy with the same data profile and the same limits as in the production environment.
A generator like wrk2 maintains a steady request flow and correctly measures latencies under load—averages are useless here; the distribution tail matters.
The system limit is captured not with one large run but with a series of short ones, with a known stopping point in advance. This way the test remains a safe experiment, not an incident.
Technical concerns: isolation, data, measurement reliability
Before starting load testing, the customer's team has the same questions: where to run the load, where to get data, and whether the numbers can be trusted. We address them in advance—at the start this is cheaper than in the report.
How is the run isolated from the production environment?
For the test, we deploy a separate bench—a copy of the production topology with its own database, cache, and queues.
The load generator lives in its own subnet with resource quotas, so it does not take capacity from production services, and traffic does not go through external load balancers: users receive neither extra connections nor failures.
Where do test data and personal information come from?
We take an anonymized sample: names, addresses, and contacts are replaced with synthetic data, but we preserve structure and proportions—table volume, share of active profiles, read/write ratio. If export is impossible, we generate a dataset according to the agreed profile. Personal information does not leave the customer's environment and does not appear in reports.
How reliable are the measurements, and is the result reproducible?
Before measurements, we warm up the system and capture background metrics to separate load from the services' own activity. The report includes the run mode: number of connections, request profile, duration, bench composition.
A repeated run under the same conditions yields variance within a few percent; otherwise we look for the cause rather than averaging.
What we need from you to start and how we will come back with an estimate
For the work plan to be realistic and not an “industry average,” a minimum of input data is needed.
We collect them in the first conversation—usually 40–60 minutes with your engineer or service owner—after which we put the agreements in writing and come back with an estimate and plan.
- Business-critical scenario: which request routes and in what proportion generate real traffic, what we consider a successful response.
- Load profile: peak and average intensity, number of concurrent clients, run duration, and ramp-up speed.
- Target metrics: allowable response time at high percentiles, error rate at which a run is considered failed, degradation threshold for stopping.
- Test environment comparable to production: addresses, credentials, restrictions on external services and ways to substitute them.
- Limitations and windows: when load can be applied, with whom we coordinate, who on the operations side can press “stop.”
- Point of contact: the responsible person on your side and the channel where we promptly resolve issues during runs.
Then the order is: we review the data, come back with a work plan—scenarios, choice of load generator (for steady intensity and precise percentiles we use wrk2, for short bursts—wrk), metrics, roles, and run windows. We agree on the plan with your engineers and only after that begin.