Configuring Envoy proxy: where traffic breaks at peaks
When a reverse proxy under load behaves unpredictably, looking for the cause in the application is almost pointless: the bottleneck at ingress manifests before requests reach services.
Long-lived connections — streaming exports, heavy reports, real-time sessions — break during a planned process restart because it finishes faster than already accepted requests can complete. In logs, this looks like a surge of error responses out of nowhere, with no code changes.
The second source is configuration desync between nodes. Routing rules are updated on one server by rereading the file, while on the others they remain unchanged.
While traffic is distributed across all nodes, the same address falls sometimes under the new route, sometimes under the old one: some clients receive a correct response, others receive a routing error.
Reproducing this on staging does not work — there is only one node, and the discrepancy simply has nowhere to come from.
The third class of failures is manual edits under load. Changing rules during peak hour requires rereading the configuration, and doing this without breaking already established connections is not always possible.
Each such operation opens a window in which part of the traffic is lost, and the more often routes change, the wider this window.
That is why Envoy proxy configuration starts not with the configuration file, but with the answer to the question of how rules reach all nodes simultaneously and what happens to connections at the moment they are applied.
We break down how this is closed: from dynamic configuration delivery without restart to controlled connection termination.
What a managed reverse proxy delivers: effect in numbers
A single entry point removes ingress-layer work from applications: client certificate verification, rate limiting, retries to unavailable nodes, scheme negotiation.
The service handles its own logic, not transport — and stops crashing from a malformed request or from a neighbor not responding in time.
We assign this role to the proxy Envoy, where routes, rules, and mutual certificate authentication are described declaratively rather than spread across application code.
What changes in numbers
Configuration apply time. Previously, a route edit meant image rebuild and restart — tens of minutes and broken connections. Dynamic configuration is picked up on the fly: a change takes effect in seconds, active connections are not broken, restart is not needed.
Failure share under load. Balancing based on actual node state, automatic retries on another instance, and circuit breaking to an unhealthy service remove cascading errors during spikes. Degradation of one instance stops being noticeable to the user.
Manual deployment operations. Routes, certificates, and rules are described as a versioned set of files and applied with one change — instead of a series of manual edits on each node.
Separately, the class of incidents “updated ingress layer — some services fell off” goes away: fault tolerance of the ingress layer is ensured at the proxy level, not by heroic actions of the on-call engineer.
Comparison of configuration modes: static, files, dynamic
In Envoy, configuration reaches the process via one of three paths: a static file at startup, configuration as code from a repository with auto-apply, or a stream from a control server.
The difference shows in the cost of an edit — restart with broken connections versus route update on live traffic.
The fourth path is not maintaining a proxy layer at all, but taking a ready-made cloud load balancer. Below are the applicability boundaries of each option, including those where the approach hits its own limits.
| Option | When suitable | Approach limitations |
|---|---|---|
| Static configuration | Staging, single services, rare route edits | Any change — restart with broken connections; does not scale to dozens of nodes |
| Configuration as code in a repository with auto-apply | One or two environments, scheduled changes, need audit and rollback via history | Apply takes minutes; no reaction to node failure or load growth |
| Dynamic configuration delivery from a control server | Many services, frequent releases, canary schemes, hot route changes | Requires a control server and its observability; an error propagates to all nodes at once |
| Ready-made cloud load balancer | Don't want to maintain a proxy layer, traffic already goes through the cloud | Less control over timeouts, retries, and mTLS policies; harder to change provider |
In practice, options are combined: static brings up the proxy itself, dynamic updates routes and certificates. The common limitation is the same — without checking configuration in a test environment and a phased rollout, an error hits all traffic at once.
Implementation order: from traffic audit to handover
We start with inventory: which services receive external traffic, where TLS terminates, and where mutual authentication is needed. Without a traffic map, editing routes turns into guessing — first record facts, then touch configuration.
- Traffic audit. Capture the picture from logs and metrics: protocol shares, peak values, long connections, request sources. The output is a list of routes with owners and encryption requirements.
- Routing scheme. Design listeners, clusters, and rules: by domain, path, or header, with mTLS policies, certificate verification, and timeouts. Agree on the scheme before writing configuration.
- Node setup. Deploy the Envoy proxy and connect dynamic configuration so edits apply without restart and without breaking connections. Separately test behavior when the configuration source is unavailable.
- Load testing. Run a profile close to production: peak requests, connection drop halfway through a response, failure of one node. Review response time and error share, adjust limits.
- Phased traffic migration. Switch flows in parts, from less critical to core, with the ability to return a route back with one edit. At each step compare metrics with the baseline.
- Handover of regulations. Hand over the scheme, configuration in version control, change release procedure, and rollback procedure; train on-call staff.
After that, your team changes routes and certificates — according to documented rules, without downtime and without us.
Configuration artifacts: schemes, policies, regulations
Several proxy configurations edited manually in an editor live exactly until the first engineer leaves. A working Envoy setup is a set of consistent artifacts that record routes, access rules, and the procedure for changing them. Below is the list of what remains with the customer along with access to staging environments.
- Routing schemes: a map of virtual hosts, path prefixes, and target services with response timeout, number of retries, and load distribution rules.
- Node topology: where proxy instances are located, how configuration is distributed, and what happens when connection to its source is lost.
- Access policies: mutual certificate verification rules, trusted client lists, restrictions by methods and paths, request rate limits.
- Roles and permissions: who changes routes, who only reads metrics, who issues certificates, and the approval procedure for such actions.
- Change release regulation: syntax check, run on an intermediate staging, rollout across nodes, result confirmation.
- Rollback regulation: storage of previous versions, trigger for starting rollback, and the time in which configuration becomes operational again.
- On-call shift instructions: typical symptoms, diagnostic commands, escalation procedure, and actions the on-call engineer does not perform without approval.
Documentation lives in the same repository as configuration and is updated together with it — a new engineer gets up to speed without retellings, and every change remains reproducible.
Case study: numbers before and after change
The client came with a platform of nine services: entry to the production environment was held on one node with configuration edited manually.
Deploying a new version meant restart — traffic went into error for 1–3 minutes, and with a typo in a route the window stretched to 20 minutes.
Certificates were renewed on schedule and twice per quarter broke mutual authentication between services — externally it looked like failures out of nowhere.
What we changed
Moved ingress to Envoy with dynamic configuration: routes and the list of services are delivered by the control layer via subscription, so the rule applies without restart and connection break.
mTLS was configured with automatic issuance and rotation — short certificate lifetime and reissue without on-call involvement. Routes and health checks were moved into declarative files, changes pass validation before application.
What we achieved
Deployment downtime disappeared: configuration changes on live traffic, and the share of successful responses at release moment does not differ from a normal day. The path from edit to effect on all nodes shrank from 25–40 minutes to one minute.
Incidents with expired certificates and node desync stopped recurring: over a quarter — not a single unplanned downtime due to an ingress change.
Rollback is now a return to the previous configuration version, not an emergency restart with warm-up, so recovery time dropped from tens of minutes to a couple of minutes.
How does mutual authentication at the node differ from verification in the service?
Mutual authentication at the node and verification of service authenticity are two different levels, and confusing them is expensive.
At ingress, Envoy answers the question “who came”: whether a certificate issued by a trusted certificate authority is presented, whether the chain converges to the root, whether the validity period has expired, and whether the owner name matches the expected one.
The service answers another question — “what is this caller allowed to do”: a specific operation, role, data access scope. The first is about authenticity, the second is about authority, and substituting one for the other breaks the whole model.
The trust zone ends where the secure connection terminates. If verification is left only in service code, each of them carries its own logic for chain parsing, revocation checking, and root certificate updates — and sooner or later one lags behind the others.
Divergence in rules means part of internal calls pass with weakened requirements, and no one notices until an incident investigation begins.
Internal is a call that passed the ingress layer and carries a confirmed zone label.
In practice it looks like this: outbound calls are intercepted at the node level, the certificate is inserted automatically, and the service does not store private keys at all and cannot read them.
This approach removes a whole class of errors — a forgotten revocation check timeout, a root certificate baked into the image, a self-signed key left over from debugging.
Everything that can be verified without knowing business logic is moved to the ingress layer: chain, validity periods, revocation, the set of allowed protocols and versions, address restrictions.
Authorization remains in the service — it depends on the domain and does not work without it. Separation enables certificate rotation without service rebuilds and a single point where security perimeter policy changes. Therefore, the two levels do not replace but complement each other.
Technical concerns: rollback, debugging, fault tolerance
In Envoy, dynamic configuration comes from a control server, and its application does not require process restart: listeners and clusters update on the fly, and established connections live until active requests complete.
If the new set turned out broken, the node returns to the last valid version without manual intervention.
What happens to established connections when new configuration is applied?
The listener switches to the new set of rules, but active streams are not broken: the connection lives until closed by the client or until the timeout from the policy expires. For long sessions, the time is set separately so an update does not tear current work.
How to roll back a failed change?
Configuration is versioned, so rollback is a return to the previous set, not a manual edit from memory. The scenario is rehearsed in advance: it is known in what time the node returns to an operational state and which connections survive the switch.
How to read logs during incident investigation?
Access logs, proxy logs, and tracing are linked by request identifier: by it you can see the node, rule, and failure reason. This is enough to separate a routing error from an upstream service failure.
What happens if one node fails?
Nodes operate as a group: if one falls, traffic goes to live ones, and availability checks remove unhealthy instances from rotation. Each node stores configuration locally, so control server unavailability does not bring down an already running perimeter.
Let's discuss the task: what is needed from your side and what we do
The start should not turn into a week of correspondence: collect inputs immediately so the estimate is based on a real environment, not assumptions. Below is what is needed from your side and what we cover.
- Accesses. Read-only — to nodes where the proxy runs, to the repository with current configuration, to the secret management system, and to the DNS zone. We share the list of accounts and roles in advance, we do not request anything unnecessary.
- Environment inventory. Which services are published externally, where entry points are, how many environments and how they are isolated. The number of configurations and the routing scheme depend on this.
- Flows and certificates. Domains and paths, where clients come from, whether mutual certificate verification at ingress is needed and who issues them — your certificate authority or an external one.
- Configuration delivery. Is it currently edited manually or delivered automatically, is there a way to return to a previous version, and who has the right to edit in the production environment.
- Observability. Where access logs, metrics, and traces go, what alerts are already configured and by what thresholds they trigger.
- Constraints. Availability requirements, windows when changes are prohibited, and the approval procedure for edits.
Our scope — designing routing and access policies, layering Envoy configuration, updating without node restarts, and the described rollback procedure. Based on the review, we come back with an estimate and work plan: stages, rollout sequence, risks, and checks at each step.