Why without chaos engineering failures break the service suddenly
Typical scenario: one node fails in the cluster — disk, network interface, or process. The failover layer should have redistributed load in seconds, but instead after a few minutes a neighboring service goes down.
The mechanics of cascading failure are predictable: clients retry failed requests, the number of calls to surviving nodes grows severalfold, the task queue overflows, and timeouts begin to trigger faster than the service can respond.
Service degradation spreads across dependencies — storage, cache, message queue — and the external layer sees not a node failure but a failure of the entire chain.
The worst part is that the failure point is usually not where checks are configured. Monitoring collects metrics for each component separately and shows green: nodes are alive, resources are normal.
Connections between services, behavior under partial unavailability, and recovery order remain untested, because in normal mode they do not manifest.
The result: the team learns about the incident from users, not from the system. The recovery plan is written on the fly, the on-call engineer determines dependencies from event logs, and every minute of such debugging is paid for with downtime and lost requests.
Chaos engineering changes the order: failure injection is started on a schedule in a controlled window, rather than arriving as an accident.
We’ll break down how this is addressed: which failures to test first, how to limit the blast radius of the experiment, and what should appear in the recovery plan after each run.
What controlled failure testing in production and staging provides
Regular failure experiments turn service resilience from a matter of faith into a measurable value.
This is not a one-time check before launch, but a mode: failure is introduced in a controlled manner, with blast radius limits, an observation window, and automatic configuration rollback.
The approach grew out of Netflix practice, where node loss was first tested manually and then made a standard procedure.
Testing brings benefit only if its result is visible in numbers. Here is what we record before and after each cycle, and what shows that experiments really work:
- Recovery time. The service returns to standard indicators in minutes: the limit is known in advance because it has been tested on a live environment, not on paper.
- Early defect detection. Errors in handling timeouts, retries, and unavailable dependencies surface in staging, not in a nighttime incident.
- Failure readiness. The on-call engineer knows what breaks first and acts according to a practiced procedure instead of finding the cause from scratch.
- A class of incidents eliminated. Degradation of an external dependency, cache desynchronization, and session loss stop recurring from release to release.
- Limited blast radius. Failure control is limited by zone and time — the experiment does not go beyond the dedicated environment.
Each cycle leaves behind not a report but a change: a new limit, an automated failure test, or a correction to the retry scheme. Therefore, what grows is not the number of experiments conducted, but the share of failures that the system survives without manual intervention.
Experiment modes and comparison of approaches
Modes differ not by force of impact, but by what class of failures we test and how much risk we are willing to accept in the production environment.
Scheduled drills provide a controlled scenario and post-mortem analysis, while automated impact tests resilience continuously, but over a narrow area.
In automated mode, the schedule is set in advance: Chaos Monkey selects a share of targets in a window, so the scale of the effect is predictable.
| Option | When it fits | Limitations |
|---|---|---|
| Scheduled drills | We test hypotheses on critical paths; post-mortem analysis and fixes based on results are needed | Require a window and manual control; do not cover rare combinations of failures |
| Background impact | Continuous verification is needed that services survive failure without team intervention | Force and area must be strictly limited, otherwise metrics become noisy and alerts become false |
| Node isolation | We confirm cluster resilience: failover, timeouts, request retries | Only horizontal resilience is visible; external dependency failures do not manifest |
| Testing in a test environment | We trial new and aggressive scenarios before the production environment | The copy differs in load and data; some failures cannot be reproduced on it |
How to combine the modes
The modes do not compete: on an environment copy we trial hypotheses, with node isolation we confirm resilience, and in the production environment we maintain background impact with limits on force.
The transition from manual drills to automation happens as data accumulates — first a narrow area and a short window, then expansion.
How we implement chaos engineering in infrastructure
Implementation begins not with launching a failure, but with describing the system: without a dependency map, any experiment turns into an unplanned accident.
We go from inventory to operations regulations, recording at each step what exactly we are testing and at what signal we stop.
Sequence of work
- Inventory and dependency map. We collect services, queues, storage, external integration points, and connections between them — including implicit calls and shared storage. We get a diagram that shows which failure spreads where.
- Defining failure boundaries. We mark which nodes may be stopped in production mode, which only in an agreed window, and what we do not touch at all. For each node, we record the expected system behavior and acceptable degradation.
- Designing the experiment scenario. We formulate a hypothesis: what we break, with what share of load, and what should happen. We set the success metric and target zone in advance — one instance, a separate environment, or a share of traffic.
- Preparing experiment shutdown. We configure safeguards: time limit, rollback based on a threshold metric, automatic failure removal. The experiment must stop itself, without manual intervention by the on-call engineer.
- Running in a controlled window. We start with minimal intensity and expand it gradually, monitoring alerts and dependency state. The on-call shift knows the scenario and the stopping point.
- Analyzing results. We reconstruct the timeline: where resilience worked, where an alert came later than user reaction, and which workarounds turned out to be fictitious. For each discrepancy, there is a specific change in configuration or code.
- Handover of operations regulations. We provide the list of scenarios, rules for launching them, roles, shutdown procedure, and instructions for on-call staff. After that, the team runs experiments itself, without our participation.
This order gives a predictable result: found weak points are closed before a real failure, and experiments remain a controlled procedure, not a one-time test left to chance.
What remains with your team after implementation
We hand over not “access to a tool,” but a working kit: using it, an on-call engineer who has never run a chaos experiment conducts a drill in an isolated environment, sees deviations, and understands at what point to stop the impact.
- Experiment configurations — templates with impact area, load share, execution window, and automatic stop condition when a threshold is exceeded.
- Resilience diagram — a map of services with failure points, backup zones, and boundaries beyond which we do not let the experiment go.
- Access policies — roles and approvals: who has the right to launch drills in the production environment, and who works only in staging.
- Drill regulations — frequency, window, procedure for notifying related teams, and rules for stopping when deviations occur.
- Responsibility matrix — service owner, on-call engineer, the person who decides on rollback, and the nighttime escalation channel.
- Scenario and report library — accumulated experiments, found bottlenecks, and records of how they were closed.
- Pre-release checks — running scenarios in staging as a pipeline step, rather than a manual “let’s try it on Friday evening.”
After two or three cycles, drills run on schedule and without us: the team adds scenarios itself, analyzes incidents, and expands the resilience diagram. We stay in touch for analyzing complex cases, but the bottleneck is no longer knowledge of the tool.
Case: how controlled failures reduced service downtime
Before: failures were learned about from requests
The service handled planned load, but the failure of one node sent part of requests into errors: the load balancer did not manage to remove the instance in time, users received failures, and the on-call engineer learned about the problem from complaints.
Service downtime was counted in tens of minutes, recovery was manual, and every repeated failure followed the same scenario — architectural fixes were postponed until a planned window.
What we tested and what we changed
Controlled failures (Chaos Engineering) were launched in the pre-production environment via Chaos Monkey (Netflix): it shuts down service instances on a schedule, without warning the team.
Additionally, we tested dependency unavailability and network delays — resilience stopped being a one-time action and entered the normal development cycle.
Metrics before and after were taken on the same set of scenarios: share of successful requests, average recovery time, number of repeated failures during the observation window.
Before implementation, availability dropped with every node failure, and recovery took tens of minutes; now an instance failure is invisible to the user, and repeated failures are isolated.
In the architecture, we moved state from the local disk to shared storage, limited and made request retries idempotent, added a readiness check for a node before including it in balancing, and added autoscaling thresholds.
Alerts were tied to user metrics rather than hardware load — the on-call engineer sees the failure before the client.
How a controlled experiment differs from an emergency drill
Chaos engineering is a controlled experiment with a pre-formulated failure hypothesis, not a simulation of a complete system crash.
The practice that Netflix formalized as an engineering approach rests on three constraints: a testable statement, a narrow impact area, and an automatic stop when metrics exceed a threshold.
The hypothesis is formulated before launch and must be falsifiable: “when one service instance fails, response time remains within the target value, and the client sees no errors.” A statement that cannot be verified by a metric is no longer an experiment, but an observation of “what will happen”: risk is created, no conclusion is gained.
Limited area is the second feature. The impact is set by a share of traffic, number of instances, one availability zone, and a short time window.
The emergency brake triggers without human involvement: as soon as latency or error share exceeds the threshold, the experiment is rolled back.
An emergency drill solves a different problem — it trains not the system, but people.
A major failure is simulated, and the response of the on-call shift is tested: how quickly the signal arrived, along what chain escalation went, whether the on-call engineer had enough authority, and whether the instructions matched reality.
Mature practice separates these modes: an experiment is launched frequently and looks for a weak point in the architecture; a drill runs according to plan and evaluates team readiness.
Mixing them is dangerous — in a drill, no one tests a hypothesis, and in an experiment, panic among on-call staff consumes its entire meaning.
Technical doubts before launching experiments
Doubts before the first experiment are usually the same: will it bring down production, will there be enough metrics to distinguish an expected failure from an accident, and who is responsible for the consequences. We address them point by point — we close these questions before the start, not after.
Can an experiment bring down production?
Production is protected by impact boundaries: we start in a test environment, then take one service replica behind the load balancer, and only after a stable result — a share of real traffic.
Each scenario has a stop condition: as soon as metrics go beyond the normal boundary, the impact is removed automatically, without waiting for a decision from the on-call engineer.
What if there are not enough metrics to understand the result?
Then we do not start the experiment. First we close observability: end-to-end request tracing across dependencies, saturation metrics for queues and connection pools, logs with a single request identifier.
Without this, a failure is indistinguishable from a normal load spike — there will be no conclusions, only fright.
Will data suffer and who is responsible for the consequences?
We test behavior during read failures, delays, and dependency unavailability — but we do not initiate destruction of storage or delete volumes; recovery from copies is practiced in a separate environment deployed from a snapshot, where production data does not get in.
We fix responsibility in the regulations before the start: the engineer launches an agreed scenario, the on-call engineer holds the right to stop, and criteria are prescribed in advance. Stopping becomes part of the procedure, not a personal decision at the moment of an accident.
Let’s discuss the task and get back with an estimate and plan
We start the analysis with inputs that reveal the real picture: diagrams of services and connections between them, a list of critical areas, balancing and retry settings, and the window in which checks are allowed.
This is enough to show where the environment already withstands failure and where it falls from the loss of one node, and to turn resilience testing (chaos engineering) into a work plan with a clear order.
- What to send: a diagram of services and dependencies, current load and timeout limits, and the history of recent failures with a description of how they were resolved.
- Task analysis: we reconstruct the failure map, mark single points of failure and places where there is no limit on request retries.
- Team composition: an infrastructure engineer, an observability specialist, and the person who will be responsible for the environment after handover.
- Work plan: the order of checks from least risky to most destructive, with windows and stop conditions at each step.
- Acceptance criteria: a list of checks with expected behavior — recovery time, share of successful requests, failover correctness.
- Operations handover: instructions for launching checks, scenario templates, approval regulations, and joint analysis of results.
Describe the task in free form — we will get back with an estimate and a plan.