Dark launch: how to test under load without risk
Classic load testing answers the question "will the system handle N requests," but not the question "what will happen with real data, real scenarios, and real dependencies." A synthetic load profile is always poorer than production: a test environment does not reproduce rare combinations of parameters, cache behavior under warm-up, or cascading calls to external services.
As a result, the bottleneck surfaces at the moment of release — when the cost of an error is at its highest.
Dark launch removes this limitation. A new version of the service is deployed alongside the working one and receives a copy of live traffic — a shadow stream. Requests are duplicated to the new version, but its responses are discarded and do not affect what the user sees.
This way, load testing runs on a real profile: the same peaks, the same anomalies, the same rare branches of logic as in production.
The key difference from a canary release is the absence of risk for the client. A canary release serves the new version to a portion of live users, and any error is directly visible to them.
Shadow traffic is isolated: a failure in the shadow contour does not lead to degradation of the main one, and response divergence is recorded for analysis.
This changes the very economics of a release — the team gets data on behavior under load before the decision to roll out is made.
Let's break down how this works: how to separate the shadow stream from the live one, what to use to compare the responses of the old and new versions, and where the boundary lies beyond which traffic duplication starts to do harm.
What traffic mirroring delivers: measurable benefits
The shadow circuit receives a copy of a real request, and the response is discarded — the production flow remains unchanged.
This makes it possible to run a new version of the service against a live load profile rather than synthetic traffic, which almost always diverges from reality in the composition of requests and their distribution. Below is what exactly changes in the metrics once mirroring is already in place.
Response time and headroom under load
The comparison of the old and new branch runs on the same traffic, so the difference in latency is visible without laboratory conditions. Typically, it is possible to catch a 10-20% degradation before rollout and eliminate it during refinement rather than in an incident.
Honest headroom emerges: it becomes clear at which profile the new version starts hitting the limit.
Risk reduction and the number of incidents
The main value is the class of failures that never reaches users. Logic errors, data schema incompatibilities, unhandled branches — all of this surfaces in the shadow circuit, where there are no live clients.
In practice, the share of release rollbacks drops severalfold, and recovery time shrinks because the cause is already known from comparing responses.
Reliability under a real profile
Mirroring reveals behavior under peaks, rare scenarios, and "dirty" requests that cannot be invented in a test suite. Shadow circuit metrics — the share of response discrepancies, latency distribution, error rate — become part of acceptance criteria. The rollout decision relies on numbers, not on a feeling of readiness.
Comparison of Shadow Testing Approaches
Shadow traffic can be captured in different ways, and the choice of mode determines what you will actually see. Full mirroring gives an honest picture across all scenarios, but it doubles the load on the new version and requires it to handle the same request profile.
Selective traffic reduces part of the load, but leaves blind spots: whatever did not make it into the sample will only be checked in production.
| Option | When it fits | Limitations |
|---|---|---|
| Full mirroring | Before a major release, when the request profile is known and stable | The new version receives 100% of production traffic — resource headroom and write isolation are required |
| Selective traffic | Targeted verification of a changed area, expensive processing | Some paths remain unchecked; the sample needs to be revisited when the profile changes |
| Asynchronous mirroring | Load on the primary system is critical, response latency does not matter | The copy lags behind the original; time discrepancies need to be accounted for during comparison |
| Synchronous mirroring | Event ordering and cause-and-effect relationships matter | Any delay in the copy drags down the primary flow |
The main limitation of any mode is side effects. As long as the copy writes to the same database, sends emails, or calls external services, comparing results is meaningless: you are measuring not the behavior of the new version, but the mutual influence of the two flows.
Therefore, write isolation and replacing external calls with stubs are mandatory conditions without which the approach does not work.
The second factor is how you compare responses. Byte-for-byte matching is rarely achievable: timestamps, request identifiers, and field ordering produce false discrepancies.
You need normalization rules and a threshold for acceptable differences, otherwise the report will drown in noise and people will stop using it.
In practice, modes are combined: full mirroring at the acceptance stage, selective — on an ongoing basis for changed areas. Such a set covers both one-off checks and regular monitoring.
Implementing dark launch: a step-by-step process
Implementing dark launch begins with an audit: we map the current traffic flow, find the points where requests can be mirrored without affecting production, and check whether there are enough resources for double the load.
At this same stage, we determine which scenarios are even worth testing — payments, search, authorization — and agree on success criteria: what share of requests we copy and what we consider a discrepancy.
After that, the work proceeds in stages, each of which ends with a verifiable result, rather than "generally done."
- Infrastructure and traffic audit. Inventory of services, load balancers, and queues, measuring peak load, identifying source services for copying requests.
- Designing the mirroring scheme. Choosing the traffic capture point, filtering rules (excluding personal data and irreversible operations), routing copies to an isolated environment.
- Setting up the shadow environment. Deploying a copy of the target service with separate databases and stubs for external integrations so that test requests are not sent to real partners.
- Enabling and configuring mirroring. Gradually raising the share of copied traffic, monitoring latency and resource consumption at the source — production must not degrade.
- Comparing responses and testing hypotheses. Automatically matching responses from the shadow and production environments, analyzing discrepancies, separating real defects from expected differences.
- Documentation and handover. Flow diagrams, configurations, filtering rules, enable and disable procedures — we document everything so the team can repeat the launch without us.
- Rollout to production. We move the verified logic into the production environment, turn off mirroring, and keep monitoring during the observation period.
The key at every step is reversibility: mirroring is turned on and off with a single setting, and the shadow environment has no access to production data. This allows the test to be stopped at any moment without consequences for users.
Artifacts you receive after implementation
After implementing traffic mirroring, what you are left with is not a "set of settings" but a working loop with a reproducible configuration.
Below is what exactly gets documented in files and repositories so that the scheme does not depend on a single engineer and survives a team change.
- Mirroring configurations — the rules by which a share of production traffic is copied to the shadow loop: which routes and methods are subject to copying, what percentage of requests is duplicated, how payments and operations with side effects are excluded.
- Traffic flow diagrams — from the copy capture point to the shadow environment and back, with explicit labeling of where responses are discarded and why the production path remains untouched.
- Access and isolation policies — separation of rights between the production and shadow loops, rules for handling copies of personal data, masking of sensitive fields before they reach the shadow.
- Operational regulations — the procedure for enabling and disabling mirroring, the on-call engineer's actions when responses diverge, conditions for automatic rollback to normal mode.
- Divergence reporting — regular summaries: the share of requests where the new version responded differently, analysis of causes, dynamics by release.
- Instructions for connecting new services — a step-by-step checklist so that the next version can be brought into the shadow loop without rebuilding the scheme from scratch.
- Decommissioning plan — how to correctly wind down mirroring once the new version is confirmed on production traffic.
Each artifact is handed over in your repository and in your documentation system — not in a closed form at our end. Describe the task: we will come back with an estimate and a plan for what from this list we close at the start and what we build up as services are connected.
Case study: how traffic mirroring accelerated releases 3x
Before traffic mirroring was introduced, the release cycle was bottlenecked by manual verification: every new version of the service was run on a synthetic test bench where the load and request profile were far from real.
Errors related to concurrent access and rare scenarios surfaced only in production — incident analysis and rollback took more time than the development itself.
What was changed
A traffic mirroring loop was deployed: production requests were copied and asynchronously sent to the new version of the service, whose responses were discarded and did not affect users.
Comparison of responses from the main and shadow versions ran automatically, and discrepancies were compiled into a report by request type.
Isolation was configured separately: the shadow loop did not write to production storage and did not call external payment and notification integrations — only reading and logging.
Metrics before and after
- The time to ship a feature to production dropped 3x: from two weeks of manual runs to a few days, where most of the time is spent analyzing discrepancies rather than waiting for the test bench.
- The share of rollbacks after release fell by roughly 70% — most defects were caught on shadow traffic before rollout.
- The number of incidents requiring an urgent rollback decreased from several per month to isolated cases per quarter.
Why it worked
The key turned out to be not the act of copying traffic itself, but the discipline of comparison: discrepancies were handled as tasks rather than as noise, and a knowledge base of known differences accumulated.
This produced the effect that teams implementing a similar approach note — Dark Launch stops being a one-off experiment and becomes part of the release process.
The result for the customer is predictable rollouts and fewer emergency nights for the team.
How does dark launch differ from canary releases?
A canary release switches a portion of real users to the new version: traffic goes to the new code, and it also generates the response.
Dark launch does the opposite — production traffic is copied and sent to the new version in parallel, but its response is not returned to anyone. The user still receives the result from the stable version, while the new one processes the same load idly.
This is the key difference: canary tests how the new version serves people, dark launch — how it behaves under real request volume without risking service quality.
Hence the different selection criteria. Canary releases are needed when it's important to evaluate the behavior of the system as a whole: latency, errors, correctness of responses in live scenarios.
Mirroring is used earlier — when the new version isn't yet ready to respond, but you need to understand whether it can handle the load, whether the results match the current logic, and whether resource consumption is growing.
The responses of the two versions are compared line by line: discrepancies in results reveal defects before they reach users.
The mechanics are also different. In a canary release, the traffic share is set on the load balancer and gradually increased — from single-digit percentages to a full switch, with automatic rollback based on error metrics.
In dark launch, a copy of the traffic is tapped from the stream and directed to a shadow contour isolated from production data: writes to the database are disabled or go to a separate store, external calls are replaced with stubs.
Otherwise, the shadow version would start changing real state — for example, double-charging funds or sending emails.
The methods also diverge in terms of load. Canary loads the new version only with the share that was switched to it, so it may not see peak scenarios.
Mirroring reproduces the full profile of production traffic, including spikes — but requires resources for the shadow contour and careful filtering of personal data before copying.
In practice, they are not opposed but arranged in a chain: first dark launch confirms that the version handles the load and computes the same thing, then a canary release tests it on live users with a gradual expansion of the share.
The choice of method is a question of which risk you are willing to accept now: showing the new version to people or first making sure of its behavior on a copy of the traffic.
Common technical questions about shadow traffic
Shadow traffic raises three natural concerns: whether production will be affected, whether data will leak, and whether services will have to be rewritten. Let's examine each one concretely — from write isolation to compatibility with the current stack.
Isolation is built at the transport level: shadow requests are tagged with a marker header, and a routing rule blocks them at the entry point — they physically cannot reach production handlers.
A copy of the response goes to a separate stream, while the main client receives the result from production. If the shadow branch fails or slows down, a timeout triggers and the request is silently discarded — production never knows about it.
On load: the share of mirrored traffic is set explicitly (usually 1-10%), and shadow calls run at reduced priority with a separate limit on concurrent connections. This eliminates the risk of the replica consuming resources from the production environment.
Shadow requests write to the production database — won't that corrupt the data?
No, as long as writes are isolated. The shadow branch works either with a read-only replica, or with a separate schema, or with a transaction that is rolled back upon completion.
Mutations (creating an order, debiting a balance) do not reach production — they are intercepted by a stub layer. We verify this during setup: we run a scenario with writes and confirm that not a single row appeared in the main database.
How does this integrate with services that are already running?
At the level of the ingress proxy or load balancer — the application does not need to be changed. A copy of the request goes to a new address, while the old version continues to respond to the client.
If the service is behind a closed perimeter, we spin up a mirror in the same network segment. For asynchronous queues, a parallel twin topic is used.
What if the shadow version responds slower than production?
This is expected and does not affect the client: the shadow response is not awaited by anyone. Latencies are recorded in metrics separately from production ones, and discrepancies in responses are compiled into a report. It is precisely these discrepancies that catch regressions before rollout.
Can mirroring be disabled quickly?
Yes — it is a flag in the routing configuration. Removing the flag instantly returns the scheme to normal operation, and the shadow environment simply stops receiving copies. Rollback does not require restarting services.
Let's discuss your dark launch implementation task
Tell us how your environment is set up now: where traffic is received, how releases are published, whether there is a load balancer with manageable rules, and where logs are written.
That is enough to understand whether mirroring can realistically be set up without rebuilding production and what volume of copy your infrastructure can handle. After that, we will come back with an implementation plan and an estimate of timelines and scope of work.
To make the conversation specific, it is useful to immediately clarify several things:
- what traffic needs to be mirrored — all incoming traffic or specific routes and methods;
- whether there is a mechanism that can duplicate requests at the load balancer or proxy level;
- how the new version of the service will read data — read-only or write to a separate storage;
- which dependencies are critical: database, queues, external services, cache;
- where metrics and logs are currently collected and whether production and test load streams can be separated;
- what checks must pass before the copy goes to real data;
- who on your team is responsible for access and change windows.
Based on the discussion, you will receive a mirroring scheme for your environment, a phased implementation plan with rollback points, and an estimate of labor costs.
We will analyze the risks — increased load on storage, duplication of side effects, data divergence — and build in ways to mitigate them in advance.
Implementation proceeds in iterations: first a copy on a small share of traffic, then expansion, and only after stable metrics — transition to a permanent scheme.
Describe the task — we will come back with a plan and an estimate.