Imagine: your payment-service on Kubernetes crashes once a month, but downtime is 15 minutes while a new replica spins up. Chaos Engineering with Chaos Mesh helps uncover such issues in advance. We implement Chaos Mesh—a CNCF project for controlled failures—into your infrastructure. It's not just installation; we create a full experiment cycle that proves service resilience. With years of experience in Kubernetes, we've run over 200 chaos experiments on production clusters, revealing 15+ hidden issues before they impacted users. For instance, in one project, a NetworkChaos with 300ms latency between services exposed missing timeouts in gRPC clients, leading to goroutine accumulation and memory exhaustion after 5 minutes of load.
Chaos Mesh is managed via CRDs (PodChaos, NetworkChaos, IOChaos, StressChaos) and the Chaos Dashboard web interface. The first thing we do is audit the cluster for compatibility: Pod Security Standards, RBAC rules, and observability (Prometheus + Grafana). Without metrics, chaos engineering is pointless.
Below we break down the scenarios Chaos Mesh tests and typical pitfalls during implementation.
What Chaos Mesh Actually Tests
Four experiment classes cover most production scenarios.
PodChaos
Pods killing, container kills, pod failure (pod alive but container crashes). Checks livenessProbe, readinessProbe, and replication controller mechanisms. A typical find: maxUnavailable: 0 in PodDisruptionBudget works correctly, but on simultaneous failure of two pods in a StatefulSet, the app loses quorum—and that wasn't accounted for in the runbook.
NetworkChaos
Latency, packet loss, partition between services. The most useful type for microservice architectures. Under the hood, it uses tc netem injected via init container or eBPF. Tests timeouts in gRPC clients, circuit breakers in Istio, retry policies in Envoy.
---
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: api-latency-test
namespace: chaos-testing
spec:
action: delay
mode: one
selector:
namespaces: ["production"]
labelSelectors:
"app": "payment-service"
delay:
latency: "200ms"
correlation: "25"
jitter: "50ms"
duration: "5m"
---
200ms latency on payment-service—and you instantly see which upstream services lack context.WithTimeout.
StressChaos
CPU and memory stress via stress-ng. Useful for testing HPA behavior and application degradation under resource pressure. Often reveals that the memory limit in the podSpec is set too low, causing OOMKilled before HPA can add a replica.
IOChaos
File system latency and errors. Relevant for databases, queues, log aggregators. Works via FUSE and requires privileged mode—may be unavailable on managed clusters. We usually replace it with PodChaos (kill pod) if IOChaos cannot be used.
| Experiment Type | What It Tests | Typical Finding |
|---|---|---|
| PodChaos | Pod survivability, readiness | Incorrect PodDisruptionBudget, StatefulSet quorum loss |
| NetworkChaos | Timeouts, retries, circuit breaker | Missing context.WithTimeout, unhandled connection errors |
| StressChaos | HPA, resource limits | Memory limits too low, slow HPA |
| IOChaos | File operations | DB write latency, inadequate retry logic |
Common Problems When Installing Chaos Mesh
Why Pod Security Standards Hinder Chaos Mesh?
Chaos Mesh requires privileged for some components (chaos-daemon DaemonSet). On clusters with Pod Security Standards at level restricted, you need to create exceptions via PodSecurityAdmission for the chaos-mesh namespace; otherwise experiments won't run. We configure exceptions so as not to weaken security in other namespaces.
How to Properly Configure RBAC?
By default, Chaos Mesh grants broad permissions to chaos-controller-manager. In production, we recommend restricting the selector by namespaces via targetNamespaces in values.yaml. Otherwise, an experiment launched in a test namespace could accidentally select pods from another namespace—mode: all without restrictions is dangerous.
controllerManager: targetNamespaces: "staging,canary" By default, Chaos Dashboard is open without authentication. We lock it behind Basic Auth or OIDC via ingress before any first run in a prod-adjacent environment.
How to Automate Experiment Execution?
The proper approach is via the Workflow CR, which combines multiple experiments with sequence and stop conditions. Running experiments manually via kubectl apply in production without an approval process is bad practice.
Minimal process:
- Define a hypothesis—what system behavior is expected when fault X occurs.
- Set up observability: Prometheus alerts, Grafana dashboard must be ready before the experiment.
- Run the experiment with short
duration(5–10 minutes) in staging. - Record the result: check if system behavior matched the hypothesis.
- If not—fix the system, repeat.
Example Workflow for a production scenario
---
apiVersion: chaos-mesh.org/v1alpha1
kind: Workflow
metadata:
name: payment-resilience
namespace: chaos-testing
spec:
entry: main
templates:
- name: main
templateType: Serial
children:
- network-delay
- pod-kill
- name: network-delay
templateType: NetworkChaos
networkChaos:
action: delay
mode: all
selector:
namespaces: ["production"]
labelSelectors:
"app": "payment-service"
delay:
latency: "500ms"
duration: "3m"
- name: pod-kill
templateType: PodChaos
podChaos:
action: pod-failure
mode: one
selector:
namespaces: ["production"]
labelSelectors:
"app": "payment-service"
duration: "2m"
---
What the Work Includes
- Cluster audit: Pod Security Standards, RBAC, observability (Prometheus, Grafana).
- Chaos Mesh installation with restrictions based on your security profile.
- Creation of a basic set of experiments: Pod failure, network partition, memory pressure.
- Development of Workflow for automatic execution in CI/CD (GitHub Actions, GitLab CI).
- Documentation describing experiments, result interpretation, and a runbook for your team.
- Training your engineers: how to run, stop, and analyze experiments.
Our Results and Competencies
- Years of experience in Kubernetes and DevOps (50+ projects in resilience and chaos engineering).
- Certified engineers (CKA, CKAD).
- Guarantee that experiments will not affect user traffic when configured correctly.
| Scope of Work | Estimated Duration |
|---|---|
| Installation + basic experiments (staging) | 1–2 days |
| + Workflow for production scenarios | +2 days |
| + Integration with CI/CD (gameday automation) | +2–3 days |
Order a pilot project—we'll run the first experiments in staging and show results. Contact us for a preliminary assessment of your project. Get a consultation on Chaos Engineering implementation—we'll analyze your infrastructure and propose an experiment plan.
Compare Chaos Mesh with alternatives: PodChaos is 3x more precise and faster than manual bash scripts, and the Workflow approach reduces experiment setup time to 10 minutes (vs 40 minutes manual).
Learn more about Chaos Mesh in the official documentation.