Setting Up Chaos Engineering with Chaos Mesh

Chaos Engineering: why failure surfaces in prod

A service goes into production after passing load tests and looks healthy: nodes are alive, metrics are steady, alerts are silent.

But links between components are almost never tested for failure — a client to a queue with no retry limit, a shared connection pool across multiple dependencies, a synchronous call in the middle of a critical scenario. Each such link remains an untested dependency and until the first failure behaves as if reliable.

A failure in production rarely looks like "one node went down." More often it is a chain reaction: one zone becomes unavailable, retries multiply the request flow, the queue overflows, and what falls is no longer related to the original cause.

Service resilience is determined not by average response time, but by behavior at the boundary — when a dependency responds slower than usual or responds with an error.

The team learns about this from users, not from monitoring: alerts are set for explicit failure, and averaged metrics smooth over degradation while it is still local.

Hypotheses about the cause begin to be tested directly on the live contour — that is, on the same users, and analysis goes by logs after the fact.

We analyze how this is closed: failure experiments are launched in a controlled contour, repeated on every release, and turn guesses into testable assertions.

For this we use Chaos Mesh — node shutdown, dependency latency and error scenarios are described declaratively and rolled back by timer, leaving no traces in prod.

Controlled failures: what changes in operations

While failures are not introduced intentionally, the system is tested only by users. Growing latency, timeouts to external services, and degradation of a database replica are discovered through support requests, not from a graph.

Scenarios in Chaos Mesh reverse this order: network delays, loss of connection to a dependency, service instance crash, and disk pressure are reproduced on a schedule with a limited blast radius — by label or namespace — and the experiment stops automatically as soon as the observed metric crosses the threshold.

The first thing that changes is the time to detect degradation. Alert thresholds are calibrated on actually injected failures, not on guesses: if a latency increase does not raise an alarm, it is visible in the same run.

Timeouts and retry policies are checked there too — usually that is exactly where configuration diverges from the actual system behavior under load.

Then a whole class of failures goes away: cascading retries, a missing circuit breaker on a degrading dependency, a timeout shorter than the actual response time. This stops making it to production because it breaks under controlled conditions.

Failure readiness becomes measurable — the share of experiments that pass without exceeding the set values.

Downtime reduction rests on recovery time.

Failover when a node or zone fails, queue behavior when a handler is unavailable, and the duty engineer's order of actions are practiced in advance — during an incident, the first minutes are not spent figuring out how the system behaves on failure.

Chaos Engineering: experiment modes and limitations

Chaos Mesh describes a check as a separate object: selectors determine which instances are hit, the mode sets the failure scenario itself, and the experiment window sets duration and schedule. Hence three levels of impact: one replica, the whole service, segment isolation.

The wider the blast area, the higher the value of the check and the stricter the rollback requirements — the experiment must be removed by timer, not by the duty engineer's memory. Below — how the modes differ and where each has its limit.

There are also general limitations of the method. Checks reproduce failures inside the cluster, but not physical hardware failure and not loss of an entire availability zone.

For stateful services and quorum, we coordinate a broad scenario separately and launch it through stop and resume: the experiment can be interrupted with one command.

Implementation stages: from audit to handover to the shift

Implementation stages are built so that a controlled failure is first tested where its consequences are reversible, and only then allowed into the production contour.

The task of the first pass is not "break and see", but to get a dependency map and a pre-described expected system behavior for each impact. Below — the sequence we go through with the customer's team.

1. Infrastructure audit and dependency inventory. We collect a map of services, external calls and queues, mark single points of failure and areas without redundancy — without this, scenarios miss the target.
2. Failure scheme and hypothesis matrix. For each node we formulate a hypothesis: what happens when an instance is unavailable, response latency grows, or a zone or database fails. We record the expected behavior and acceptable degradation level.
3. Setting up scenarios with blast radius limitation. Impact is addressed by labels, coverage is limited to a share of instances, and we add automatic stop by time limit and by deviation of key metrics.
4. Run in a safe contour. First launches — on a test environment and on a copy of production traffic, to calibrate thresholds and make sure the scenario does not go beyond the sample boundaries.
5. Work in production in agreed check windows. One scenario at a time, duty engineer on call, ready rollback plan; we observe by metrics, traces and logs, and stop criteria are written in advance.
6. Review of results. We aggregate detection and recovery time, see which hypotheses were not confirmed, and fix root causes: request retries, liveness checks, limits, routing.
7. Handover to the duty shift. We hand over the scenario catalog, start and stop checklist, alert thresholds and owners — then the shift runs checks independently according to the regulation.

What remains in your repository after the work

The work ends not with a report in email, but with a set of files that lives alongside your manifests and is updated by the same commits. Otherwise, in six months no one will remember which scenario brought down the payment contour and why it was disabled. What remains in hand:

• Scenario documentation — for each experiment: hypothesis, target objects, duration, expected system behavior and the sign by which the failure is considered reproduced. Descriptions are stored as code, not in a wiki.
• Dependency diagrams — a map of services indicating what degrades when a node fails, where there is no reserve, and which areas hold load under partial capacity loss.
• Check configurations — metric thresholds at which the experiment is interrupted automatically, and rules for restoring the initial state after the run.
• Access separation — roles for launching the experiment, changing the scenario and viewing results, so that launch is not available to everyone with deploy rights.
• Stop regulation — who is authorized to interrupt the run, how it is done and in what time the system returns to normal.
• Instructions for duty engineers — a short memo: what to do when a check triggers, where to look, which alerts to consider expected during the experiment.
• Review regulation — how the result is recorded: whether the hypothesis was confirmed or not, what changes in architecture and monitoring follow.

Each item is versioned, goes through review and is deleted together with an outdated scenario. Thus the set does not turn into a dump of instructions, but remains a working tool of the duty shift.

Case: how a zone failure was found before an accident

The service lived in three availability zones but could not survive the loss of one: the load balancer continued sending requests to the isolated zone, clients waited for a response until timeout, and retries finished off the remaining ones.

The incident stretched almost 40 minutes, although the failure affected only a third of capacity. The cause — no one checked the system behavior on zone failure: configurations were verified on paper, not under load.

What changed

Failure scenarios were described as code and run first in the test contour, then in production — in a low-load window, via Chaos Mesh. The set is simple: zone isolation, response latency, partial packet loss, node failure.

Each experiment has a stop condition when metrics leave the boundaries and automatic rollback on completion.

Metrics before and after

Before implementation, with zone isolation the share of successful responses fell to 60%, failover took minutes, and recovery required manual intervention.

After a series of experiments and fixes to readiness checks, timeouts and retries, the failure stopped going beyond one zone: failover fits into tens of seconds, the share of successful responses stays above 99%, and recovery takes 3–5 minutes. Resilience under load is now measured before release, not discovered during an incident.

What to do if the cluster already runs under load

A running cluster under load is not an obstacle for checks: they are launched without stopping the service, but not "all at once." <cite>Chaos Mesh</cite> outlines the blast area beyond which the experiment will not go under any development of events: a specific contour, a group of replicas by label, failure type and duration. The critical request processing path is not included in the check.

Then — gradual expansion. The first run goes on one replica in the test environment, then on a share of production replicas in single-digit percent, and only after confirmed resilience does the share grow.

Separately recorded are which scenarios are checked at each step: first individual pod failure, then node failure, then degradation of the link between services.

Between steps they look not at "is everything alive", but at specific metrics: error share, response time at the tail of the distribution, queue depth.

Stop conditions are set before launch, not during. Auto-pause by timer, disabling by flag, removing the experiment when metrics leave the threshold — all this works without a duty engineer at the console.

To this is added a pre-written rollback: in what way settings are returned and in what time the contour comes to the initial state.

Exactly from these three things — boundaries, steps and stop conditions — a safe experiment is composed. It gives what everything is started for: hypotheses about failures are checked on real traffic during working hours.

Describe your contour and the failures that raise questions — we will return with a check plan and assessment.

Limitations of the method and technical doubts

The method works where we control the infrastructure, and stops working at the boundary. Pods, the network inside the cluster, volumes and configurations we break in a controlled way; an external queue, payment gateway or someone else's DBMS cannot be checked this way.

For them the scenario is rebuilt: unavailability is emulated on our side — timeout, connection drop, error response — and we look at how the application survives it.

Here it is easy to get a false picture. If latency is set on all outgoing calls, the service will start falling because of our own stand, and the conclusion "the architecture does not hold" will be wrong.

Injection is therefore limited to one dependency, a window is set, and the experiment is marked, to separate its effect from background degradation.

False positives are a separate topic. Liveness checks see the same picture as a real failure, and launch restarts on top of the experiment: the signal drowns in noise. Pre-recorded normal indicators, contour isolation and a pause of auto-recovery during the window help.

Rights to launch are limited strictly: the ability to inject a failure equals the ability to bring down the production contour. It is granted by roles for specific namespaces, and for scenarios with a wide radius, confirmation by a second engineer is connected.

Let's discuss the task: where the check begins

The first step is not a ready list of experiments, but your story about the task. Enough in your own words: which services are critical, where the boundary between contours runs, what has already broken and which failure is the most damaging.

The state of observability is also important — without metrics and tracing, the check turns into guesswork.

Then we return with an assessment and a plan. They include:

• dependency map: what exactly holds the service and how it will behave when each link fails
• experiment boundaries — separate contour, isolation and limitation of the impact area
• pilot scenario with stop criteria and quick rollback rules
• work plan by stages with timelines, control points and team composition
• list of observability improvements without which measurements are meaningless
• report format after the pilot: found defects and their priority

We set experiments in your contour — on Chaos Mesh, if it is already used, or on a tool for your stack.

Describe the task in free form — we will return with an assessment, plan and list of the first scenarios that will give maximum effect.