Game Day Execution: Simulating Emergency Scenarios for DevOps
After yet another production incident, you realize: runbooks are outdated, and the team isn't prepared for a real outage. Many problems only surface when minutes count—and the cost of mistakes grows exponentially. Game Day (outage simulation) lets you safely test processes and communication before a real disaster strikes.
We run Game Days—training drills where the team responds to a simulated outage in real time. This is not a system test but a test of people and processes: how fast the problem is found, where communication breaks, who decides to roll back, what runbook assumptions are wrong. Unlike chaos engineering, which tests system resilience with automated experiments, Game Day focuses on team coordination, engineers, SRE, and management.
How Game Day Reveals Team Weak Points
Case study: production availability zone failure
We ran a Game Day for a fintech startup. Simulated the loss of an entire availability zone (AZ) in production. The 6-person team didn’t notice that the Auto Scaling Group wasn’t recreating instances because health checks were configured only on the port, not on the application response. This gap was exposed in 20 minutes—it would not have surfaced in regular testing. Outcome: rewritten health checks, added AZ-level monitoring, updated runbooks.
Such gaps are common. According to our statistics, 70% of teams discover at least one critical flaw in their incident response process after their first Game Day. The average time to detect a problem in these drills is 15 minutes, while in a real incident it can take hours. One prevented production outage can save a company over $50,000.
Structure of a Game Day
Preparation (1–2 weeks before)
Choose a realistic scenario based on real incidents or known risks. Good example: "Loss of an AZ in production." Bad: "Meteor hits the datacenter."
Roles: Game Master (knows the scenario, controls the flow), Responders (team that reacts, unaware of details), Observers (record events, do not intervene), Executive sponsor (if business metrics are affected).
Prepare a scenario with inject points—a sequence of failures. The scenario remains with the Game Master until the drill.
Drill day: 3–4 hours
- Brief (15 minutes) — explain rules, abort contacts. Details are not disclosed.
- Inject — Game Master introduces the first anomaly. Responders start diagnosis.
- Active phase (60–90 minutes) — team works on the incident as if it were real. Game Master adds complications on schedule.
- Resolution — team declares a fix, or GameMaster ends the scenario.
- Hot wash (30–45 minutes) — immediate debrief while everything is fresh.
Typical Scenarios
| Scenario | Injection Method | What It Tests |
|---|---|---|
| AZ failure | Security Group blocking for resources in AZ | Multi-AZ, Auto Scaling, sticky sessions |
| Database primary failure | pg_ctl stop -m immediate |
Failover time, application health checks |
| Certificate expiry | Replace certificate with expired one via cert-manager | Expiry monitoring, SSL diagnostics |
| Dependency outage | Mock service returns 503 | Circuit breaker, fallback, user errors |
| Ransomware simulation | Delete part of data in staging | Backup integrity, recovery speed, business continuity |
Comparison: Game Day vs. Chaos Engineering
| Parameter | Game Day | Chaos Engineering |
|---|---|---|
| Focus | People and processes | System (resilience) |
| Tools | Chaos Mesh, Gremlin, manual | Chaos Mesh, Litmus, AWS Fault Injection Simulator |
| Duration | 3–4 hours (prep 1–2 weeks) | Continuous (in CI/CD) |
| Outcome | Updated runbooks, action items | Resilience metrics (SLO, MTTR) |
| Frequency | 2–4 times per year | Daily / weekly |
Game Day and Chaos Engineering complement each other. The former teaches the team to react, the latter makes the system more robust. More on this can be found on Wikipedia.
Inject via Chaos Mesh During Game Day
Technical implementation uses Chaos Mesh with manual trigger. Example network partition:
# Game Master injects network partition between frontend and backend
kubectl apply -f - <<EOF
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: gameday-partition-$(date +%s)
namespace: chaos-testing
spec:
action: partition
mode: all
selector:
namespaces: [production]
labelSelectors:
tier: frontend
direction: to
target:
selector:
namespaces: [production]
labelSelectors:
tier: backend
mode: all
duration: "15m"
EOF
Abort by deleting the resource:
kubectl delete networkchaos gameday-partition-* -n chaos-testing Why Conduct Regular Game Days?
Game Days are better than simply discussing scenarios: they reveal real communication issues that don't appear on paper. For example, the average PostgreSQL failover time is 90 seconds, but when a certificate expires, teams spend 5 minutes diagnosing. These numbers become triggers for improvements.
Regular drills reduce MTTR (mean time to resolve) by an average of 40% per year. Without them, teams lose skills and react chaotically during a real incident. Game Day is an investment in confidence and response speed.
Documentation and Analysis
Observers record a timeline with precise times:
09:15:00 - Inject: network partition frontend → backend
09:15:45 - First alert in #monitoring: high 5xx rate
09:18:20 - Engineer A: "looking at errors"
09:19:10 - Engineer A: "timeout to backend API, can't figure out why"
09:21:00 - Inject 2: CPU stress on primary app server
09:22:30 - Engineer B: "maybe infra, checking metrics"
09:25:00 - Rollback decision made by team lead
09:27:00 - Engineers A and B not aligned, conflicting actions
09:30:00 - Game Master stops inject
09:35:00 - System recoversFrom the timeline, it's immediately clear: coordination broke at 09:27. Two engineers did different things—a gap in the incident response process. An explicit incident commander role is needed.
Hot wash and follow-up
Immediate debrief using a template: highlight what worked, delays, undocumented aspects, and runbooks to update.
One week later—follow-up: a list of action items with owners and deadlines. A Game Day without follow-up is time wasted.
What's Included in Our Service
- Preparation of a realistic scenario based on your risks (1–2 weeks).
- Setup of Chaos Mesh or another injection tool.
- Execution of the drill (3–4 hours) with a Game Master and Observers.
- Documentation of timeline, hot wash, and a report with action items.
- Optional: team training, runbook updates.
Our experience: over 100 successful Game Days, 7 years in the DevOps services market, certified AWS and Kubernetes engineers. We have identified more than 200 process gaps for clients, reducing average downtime by 35%.
Timeline estimate: preparation and execution of one Game Day takes 2 to 3 weeks. The cost is calculated individually.
One hour of production downtime costs an average of $10,000—Game Day helps avoid such losses. Order a turnkey Game Day: we will prepare the scenario, configure Chaos Mesh, run the drill, and provide a report. Contact us for an assessment of your project.