Setting Up Chaos Engineering with Gremlin in Kubernetes

Chaos Engineering Gremlin: failures that tests don't cover

All checks passed, the staging environment is stable, the release goes to prod — and a day later the service goes down when a node is taken out for maintenance.

The cause is not in the code: the application has never run under conditions where one of its dependencies is unavailable, and standard tests do not model this. Mocks and integration runs check logic, load tests check throughput, but not behavior during failure.

The practice of Gremlin in Kubernetes closes exactly this gap. Controlled failures are injected into the cluster: node failure, network partition between pods, degradation of an external service, disk I/O saturation.

The agent executes the scenario on a schedule or manually, capturing metrics before, during, and after.

First runs expose hidden failures that in normal life surface at the worst moment. Retries without a limit on number or time turn a delay into a cascading failure.

The readiness probe responds successfully when the container can no longer serve traffic, while the load balancer keeps sending requests. The queue grows, but the alert is configured only for failure, not degradation.

A separate category is bottlenecks in dependencies. A shared connection pool to the database means that one slow query blocks the other handlers. An external provider responds slower than usual, no timeout is set, and workers are busy waiting. In isolation, such a relationship is invisible — only a simultaneous failure reveals it.

Next — how to build fault-tolerance testing safely for prod and get conclusions, not a list of failed dashboards.

What a controlled fault-tolerance test gives

Controlled fault-tolerance testing stops being a one-off event and becomes a regular procedure: scenarios run in an isolated environment, the scope is limited, and the result is recorded in a report and in monitoring.

We work with Gremlin, which provides a controlled blast radius — the experiment can be interrupted at any moment without affecting production traffic.

What the customer gets in measurable form:

• Eliminated class of failures. Pod failure, loss of an availability zone, degradation of an external dependency, cache unavailability — each scenario is worked through and closed by a configuration fix, not postponed to the backlog.
• Reduced service recovery time. The path from detection to recovery is reduced severalfold: the response of the on-call engineer and automation is rehearsed in advance, rather than figured out during an incident.
• The share of automatically handled failures grows. Restart, failover to standby, shedding excess load happen without human involvement.
• Confirmed system resilience. The hypothesis "it should withstand" is replaced by a protocol: what was broken, what was expected, what was obtained, and how it was fixed.
• Failure readiness before deployment. Critical changes undergo resilience testing before the version is released, not after an incident.
• Refined monitoring thresholds. Each experiment adjusts alerts and autoscaling rules, so false positives go away.

Next — how experiments fit into a regular cycle: schedule, scenario selection, and stop criteria.

Experiment modes: comparison of approaches and limitations

The experiment mode is determined not by the tool, but by the cost of error and the amount of statistics you will get.

A one-off check gives a snapshot on a specific version; continuous mode gives data across all changes over a period; an isolated environment gives the freedom to reproduce destructive scenarios without risk to users.

In practice, modes are combined: platform failures are tested in an isolated environment, while in production a limited set of checks is left with automatic interruption based on an observability signal.

A platform like Gremlin allows narrowing the experiment's blast radius to specific services and labels, but does not remove the limitations of the mode itself.

The task dictates the choice: the wider the experiment's blast radius, the stricter the requirements for automatic interruption and the more important a pre-defined stop criterion.

Implementation stages: from audit to handover of runbooks

Without a dependency map, a chaos experiment turns into a real outage: it is impossible to predict what will fall off together with the pod we intentionally remove.

That is why the work proceeds sequentially — from inventory to handover of operations documents, and each stage provides input for the next.

1. Infrastructure audit and dependency inventory. We compile a registry of services, connections, and external failure points — without it, it is impossible to choose a safe blast area and assess the consequences.
2. Experiment design. We formulate a testable hypothesis: what exactly we break, at what share of load, in what time window, and what we consider confirmation of resilience.
3. Stop criteria. We define guardrail metrics and automatic interruption thresholds so that the scenario shuts itself down before degradation reaches the user.
4. Setup and launch. We deploy scenarios via Gremlin first in the test environment, then on a limited share of production load, with logging of every step.
5. Safety review. Review together with operations: access rights, isolation of the blast area, exclusions for critical nodes, manual rollback procedure.
6. Shift training. On-call engineers learn to distinguish a planned experiment from a real failure, read its log, and launch scenarios according to a checklist.
7. Handover of runbooks. We hand over scenario descriptions, launch policy, schedule, and experiment history — the team continues the work without our involvement.

Implementation artifacts: manifests, policies, runbooks

Implementation does not end with one successful experiment. After the work, your repositories retain a set that lets the on-call shift and developers reproduce any scenario without the contractor's involvement — from launch to post-mortem of the consequences.

• Configuration manifests — experiment descriptions are stored in version control next to application manifests. Any scenario can be repeated, rolled back, or moved to another cluster, rather than rebuilt from memory.
• Access policies — roles and bindings that separate the experiment environment from the production environment: launch is allowed only for designated accounts, emergency stop is allowed for the on-call engineer without the right to change parameters.
• Service dependency diagrams — a map of connections indicating failure points, cascades, and critical paths. It shows what to test first and where the experiment affects an adjacent system.
• Scenario catalog — a set of checks with expected system behavior, trigger threshold, and success criteria. The catalog is expanded by your team.
• On-call shift instructions — the procedure for before, during, and after the experiment: how to stop everything with one command, whom to notify, at which metrics automatic termination triggers.
• Results review procedure — a report template, criteria for "conclusion confirmed or refuted," a list of owners of discovered defects, and deadlines for closing them.
• Window and approval policy — when experiments may be launched, who approves them, and what restrictions apply during periods of increased load.

Experiment setup in Gremlin is handed over together with these artifacts: configurations in open formats, diagrams and runbooks — in your documentation. We support the first series of runs and transfer the lead role to your engineers — after that, the cycle works without us.

Case study: how an entire class of queue failures was eliminated

What it was before the experiment

The message queue in this project relied on a cluster of several nodes, and the failure of one of them was considered normal — until lag began to grow.

Incident analysis revealed something unpleasant: producers kept writing, some acknowledgements were lost, and consumers went into endless retries — without a limit on attempts and without idempotent processing.

Alerts remained silent: the lag threshold was set above the actual rate of degradation, so the failure was noticed only from complaints by adjacent teams.

What was changed as part of the implementation case

We connected Gremlin and described failures as reproducible scenarios: failure of an individual node, network delay between producer and broker, complete unavailability of the broker for several minutes.

The scenarios ran in a staging environment close to production in volume and traffic profile — this way, failure-handling defects were found before release, not during a night incident.

As a result, retries with increasing delay and an attempt limit, idempotent reprocessing keys, and a separate queue for messages that could not be processed within the allotted time were introduced.

Metrics before and after

• Degradation detection: about 40 minutes → 2 minutes — the alert was switched to the lag growth rate, not the absolute value.
• Queue recovery: about 3 hours of manual investigations → 12 minutes of automatic node eviction and partition rebalancing.
• Message loss and duplicates during node failure: isolated cases in every incident → zero across all scenario runs.

The main result is not a closed incident, but an eliminated class of failures: broker failure scenarios became regression checks in the build pipeline. Every change to the queue code now goes through them, and the reappearance of an unhandled failure is caught before rollout.

Gremlin and load testing: what is the difference?

Load testing checks behavior under peak load: traffic grows according to a profile, and we look for the point where response time exceeds the agreed threshold.

Failure testing sets a different hypothesis — the system runs at normal load, and at that moment a node or dependency is removed. The first subject is performance headroom, the second is resilience and recovery time.

They are confused because both classes of checks look the same in a report: a spike in errors, a drop in metrics, alerts.

But the causes are different: a resource bottleneck is found through a load profile and fixed with configuration, while the absence of failover and timeouts is visible only under controlled failure — and is treated with policies, not by adding capacity.

One is not enough because failures hit precisely at peak: a cluster that calmly survives node loss during quiet hours does not manage to redistribute traffic when its neighbors are already at their limit.

The opposite situation also occurs — the system handles peak, but loses everything when the only replica of a dependency fails. That is why failure types are tested at several load levels.

First, the load profile and response-time thresholds are fixed, then the list of failures and acceptance criteria: share of successful requests, failover time, absence of manual intervention.

Gremlin covers the second half — scenarios are reproducible, scoped, and stop on their own, so they are launched regularly, not as a one-off experiment.

Technical doubts before starting an experiment

Objections before the first experiment are almost always the same — this is a sign of a healthy engineering culture, not resistance. Below are answers to three questions that most often delay the start of Chaos Engineering and work with Gremlin in a cluster.

How does the experiment affect the production environment?

The first launch runs in an environment that replicates the production environment in topology and constraints. Then we proceed gradually: a small share of instances, a low-load window, only stateless services.

When an individual instance fails, traffic is redistributed, so we test what is supposed to withstand failure, not what depends on a single node.

Is a separate environment required?

For the first run — yes, otherwise there is no safe rollback point. But it should not be "toy-like": matching network policies, limits, configuration, and deployment layout is critical, otherwise the results will say nothing about the production environment.

Injection rights are granted separately and in a limited way — the experiment should not be able to do more than the scenario requires.

Let's discuss the task and come back with an estimate and plan

To make the estimate substantive rather than a rough guess, what already exists in your environment is enough for us.

The review happens in two steps: we align the picture with your team and come back with a work plan where each stage has a readiness criterion and a rollback point.

• List of services and their criticality: what tolerates degradation during failure, and what stops the product.
• Cluster and environment diagram: how many environments, how traffic is separated, where failure boundaries lie.
• Current manifests and configuration: resource descriptions, deployment templates, limits, and quotas.
• What already collects metrics, logs, and traces, and at what thresholds notifications trigger.
• Description of past incidents: what broke, how it was detected, how long recovery took.
• Restrictions: windows when experiments are allowed, and services that must not be touched until separate approval.
• Responsible people on your side: who confirms hypotheses and accepts the result.

Based on this data, we come back with a plan: a set of failure scenarios, the order of launching them from least risky, conditions for automatic stop, and a way to roll back changes with one command.

Describe the task in writing or on a call — we will discuss the details and come back with an estimate and work plan.