HAProxy reverse proxy: where the scheme breaks under load
The reverse proxy is first in the scheme and is the first to bear the consequences: while traffic is moderate, a configuration of a dozen lines handles the load unnoticed.
It breaks where a burst of requests or failure of an application node turns the proxy into a single point of failure for the entire service — ingress disappears, not a separate function.
Connection overflow manifests earlier than the rest. The frontend queue is designed for a calm flow: during a burst, free slots run out, some clients receive a connection drop instead of a response, and the backend capable of handling them sits idle.
Degradation under load is nonlinear — response time jumps, because new requests join the same queue as old ones and wait alongside them.
The second bottleneck in the scheme is the absence of node health checks. Without periodic polling, the balancer continues sending traffic to where the application is already returning errors, and the failure of an application node drags down the entire service.
Nearby is another typical cause: manual configuration edits without rollback, when reload happens without syntax validation and without a saved previous version — and at the moment of an incident, the engineer is left with no way back.
All three scenarios are closed at the level of scheme and rules, not heroic actions during a failure.
Below we break down exactly how: limits and timeouts for the real traffic profile, health checks with node removal from rotation, safe application of changes with preservation of the previous version and quick return to it.
What a managed balancer provides: resilience and predictability
Node failure ceases to be an event for the user. The balancer on HAProxy polls application servers at a specified interval and, with rise/fall thresholds, removes a failed one from the pool before the client gets an error, while the standby node takes the load itself.
Recovery time is measured in seconds and no longer depends on who noticed the problem first — the on-call engineer or the monitoring system.
Throughput grows on the same hardware. Connection reuse to servers and balancing by the number of active requests eliminate skew when one node holds more clients than the others; reuse of secure sessions reduces the overhead of establishing a connection.
An entire class of failures related to manual intervention goes away: rolling out a new version goes through graceful removal of a node from rotation rather than stopping the service, and failed requests for repeatable operations go to a live node.
Session stickiness rules are set explicitly, so "floating" authorization errors, which otherwise take weeks to find, do not occur at all.
A transparent event log appears: a unified record format captures response time for each server, response codes, and which node served the request.
From it you can immediately see where degradation is — in the application or on the path to it — and response predictability stops being a matter of guesswork: the 95th percentile stays in a narrow corridor even under peak load.
Proxy and balancing modes: what to choose for the task
The proxy mode determines what is available at the ingress point: whether only the address and port are visible, or also headers, path, domain. This determines content-based routing, encryption termination, and the level at which health checks are counted.
Four options that are actually chosen between:
| Option | When it fits | Limitations |
|---|---|---|
| Pass-through at connection level | Non-standard protocols, databases, end-to-end encryption to the server | Request contents are closed: no path-based routing, no header editing; check is only port availability |
| Encryption termination at ingress | Web services with a shared domain, a single certificate, connection reuse | Key resides on the proxy; the internal segment without re-encryption requires a trusted network |
| Content-based routing | Multiple services behind one address, canary releases, version separation | Full request parsing is required: higher latency at ingress, a single server failure is more noticeable |
| External managed service | No own fault-tolerant ingress, ready-made certificates needed | Less control over algorithms and timeouts, provider lock-in |
After choosing the mode, request distribution and health checks remain: for long-lived connections, an active session counter is more beneficial; for services with session affinity, holding the client on a node; active probes remove a server from rotation and return it only after several successful responses. This removes the main cause of "floating" 500s under load.
Implementation order: from scheme audit to handover to operations
Work proceeds sequentially: until the current topology and traffic profile are determined, any configuration change is guesswork. Therefore we start with a scheme audit, and switch production traffic to the new configuration only after it has run on a test environment. Below are the implementation stages.
- Scheme audit. We gather the actual picture: where the reverse proxy sits, which frontends and backends are declared, how routing rules are laid out, and how server availability is checked. We compare this with error logs and response code statistics.
- Target scheme design. We decide what we balance — HTTP or TCP, where TLS is terminated, how user sessions are held, and by what signs a node is considered unhealthy and removed from rotation.
- Configuration setup. We describe frontends and backends, routing rules, timeouts, connection limits, and request retries. We keep the configuration in version control — every change goes through review.
- Testing on a test environment. We run failures: node crash, graceful restart, load growth, certificate expiration. The readiness criterion is that failure of one node is unnoticeable to the user.
- Managed traffic transfer. We switch in portions: first part of the requests, then the whole flow. Rollback to the previous scheme remains available with one command until the end of the transfer.
- Handover to operations. We give the on-call shift a service map, change regulations, a set of metrics and alerts, the procedure for scheduled certificate renewal, and actions during degradation.
What the team keeps: configurations, schemes, and regulations
Everything we configure in HAProxy remains in your repository with edit history, not in correspondence with an engineer.
Below is the list of artifacts the customer receives at handover: by them, operation and development of the service do not depend on who exactly configured the configuration.
- Working configurations — files with comments for each section: routing rules by domain and path, timeouts, concurrent connection limits, balancing mode, and parameters for server availability checks.
- Connectivity scheme — request flow map: encryption termination points, server pools behind the proxy, failover order, behavior when a node drops out, and traffic return logic after recovery.
- Access policies — rules for restricting by addresses and request rate, separation of service and public routes, handling of client headers, and maintenance mode during release.
- Rollback regulation — step-by-step procedure for returning to the previous configuration version: where backups are stored, how to check the configuration before applying, and how long rollback takes.
- Certificate renewal procedure — who is responsible for issuance, how automatic renewal works, what to do on a validation error, and how to ensure the new certificate is picked up without connection interruption.
- Instructions for the on-call shift — typical failures and signs of their appearance in logs, what can be fixed on the spot, and in which cases to escalate the task, indicating monitoring trigger thresholds.
- Change verification — description of the stand where edits are tested before production, and the set of checks that must pass before applying the configuration on production nodes.
Case: how the ingress point survived peak load without degradation
The client is a personal account service with a daily audience of tens of thousands of people. There was a single ingress point: a single node received all traffic, and it also held certificates and terminated secure connections. While load grew smoothly, the scheme coped, but at a promo peak it hit the limit.
What it was like at peak
During sale hours, concurrent connections became twice the usual, and the node began hitting the connection limit. The share of requests ending in error reached 7%, and response time for some users exceeded 4 seconds — carts did not survive until payment.
A separate pain: any restart meant up to 9 minutes of complete unavailability, because there was no one to switch to.
What was changed
We assembled a fault-tolerant pair of HAProxy nodes in two different availability zones, combining them with a shared virtual address.
The active node receives traffic; the standby constantly checks its state and takes over the address on failure — in fractions of a second, without breaking already established connections.
Separately, we recalculated queues and timeouts for the real request profile, so that a slow client would not occupy slots of healthy ones.
What changed
Downtime on node failure decreased from 9 minutes to less than a second — the user does not have time to notice the switchover. The error share at peak fell from 7% to 0.2%, and response time on the same traffic volume — from 4.5 to 0.8 seconds.
Updates and maintenance now go node by node: the second holds traffic, and a separate downtime window is no longer needed.
Connection balancing or content-based routing?
If traffic is opaque — a connection to a database, message broker, encrypted stream that the proxy does not decrypt — connection balancing is enough: the decision is made once when establishing the connection, and then all packets go to one node.
As soon as HTTP requests with different paths, hosts, and headers appear, distribution alone is not enough — content-based routing is needed.
Connection balancing is cheaper: the proxy does not parse the request but counts connections per node — round-robin, by number of active, by client address, when a session must be held on one server.
There is one limitation: within a connection you cannot see where /api is and where static content is, so you cannot distribute load by request types.
Content-based routing works at the HTTP level: the request is parsed in full, rules spread it across pools by host, path, or header. Health checks appear by response code rather than open port, session label in a cookie, and forwarding of the original client address. The price is computation for every request and a set of headers shared with the application.
Combining approaches is a working scheme: a node receives HTTP traffic and distributes it across pools, while leaving some pools' connections opaque — for databases, queues, or an encrypted stream whose keys remain with the service owner.
The choice of mode also determines which health checks the pools will have and how you will diagnose failures.
| Option | When it fits | Limitations |
|---|---|---|
| Connection balancing | Opaque traffic: databases, queues, encryption without decryption at the proxy | Paths and headers are not visible, request types cannot be separated |
| Content-based routing | HTTP services, multiple applications, static content separate from API | More expensive in computation, requires agreed headers and rules |
| Combination | Mixed environment: HTTP routes outside, opaque streams inside | Scheme is more complex, a map of pools and checks is needed |
Common technical doubts before changing the scheme
When migrating the scheme, the main question is not "will it work" but what will happen to already open connections at the moment of switchover. Below are four doubts that most often come up before we touch anything in the production environment.
Are current connections broken when switching nodes?
For short requests, the transition is unnoticeable: the client repeats the request, and the reverse proxy sends it to a live node.
Long connections — streaming, WebSockets — are drained via graceful shutdown: the server is removed from rotation but waits for a response on already open sessions; new requests go to the remaining ones.
Do we need to change anything in the application?
In most cases no — the application receives requests as before, only the ingress point changes. Edits are needed where the code itself computes the client address from headers: if the original address is not forwarded, the balancer address appears in logs and IP restrictions. This is fixed by configuration, without releasing a new code version.
How quickly is a failed node disabled?
The polling interval and check threshold are set explicitly. With well-tuned values, a non-working node drops out of rotation within a few seconds — the same amount of time it takes to return after recovery.
Too frequent checks cause false positives under peak load; too rare ones stretch out degradation, so thresholds are selected based on the actual traffic profile, not "by eye".
How to update configuration without downtime?
First a syntax check of the file, then a graceful reload: the new process starts and accepts traffic, the old one finishes active sessions. The downtime window is zero, sockets are not recreated.
Rollback is returning the previous configuration and repeating the same procedure, so we make changes in small steps.
Let's discuss the task and come back with an implementation plan
To come back with an implementation plan rather than general agreements, we need a set of inputs from you. From them we estimate timelines, choose the balancing algorithm and switchover order — so work proceeds without stopping the service.
- List of services that will go through the reverse proxy: addresses, ports, protocols, and where certificates are terminated — on the balancer or on the application side.
- Current scheme: what is in place now (HAProxy or another balancer), how traffic is distributed, what node health checks are configured, and how rollback is performed.
- Expected load: peak requests per second, number of concurrent connections, share of long sessions — the balancing algorithm and limits depend on this.
- Constraints on work windows: allowable switchover duration, whether a standby environment exists, whether DNS records can be changed and at what time.
- Access: servers, configuration repository, certificate management panel, rights to change network rules.
- Certificate requirements: who issues them, whether automatic replacement and chain validation during renewal are needed.
- Acceptance criteria: node failure, traffic burst, scheduled certificate replacement — and who on the customer side confirms the result.
That is enough to prepare an implementation plan with stages, rollback points, and timeline estimates. Describe the task — we will come back with a plan and scope of work.