Load Testing and Chaos Engineering Services

Why does load testing fail even after a successful release?

The release went smoothly. Three days later — a promotion, traffic spikes 10x. Four hours after the promotion starts, the database hits lock wait timeout, the API returns 504, and support drowns in tickets. Yet the staging load test passed “successfully” — because they tested with 50 users, while real traffic brought 5000. This scenario repeats in every second project that comes to us. We see the same mistakes: tests are written for a single scenario, slow queries on cache miss are not considered, and system behavior under dependent service failures is completely ignored. Load testing is not a checkbox before release. It is the process of finding the point where the system starts to degrade and understanding why exactly there. Our engineers with 10 years of experience in performance testing guarantee that after working with us, you will know your system's limit.

How to write k6 scripts from simple to production-grade?

k6 is the most convenient tool for writing load tests as code. JavaScript syntax, built-in metrics, native integration with Grafana via k6 cloud or xk6-output-prometheus-remote.

A basic script solves a simple task. Real problems start when you need to simulate complex user scenarios.

Example: a test with authentication and dependent requests:

import http from 'k6/http'; import { check, sleep } from 'k6'; import { SharedArray } from 'k6/data'; const users = new SharedArray('users', function () { return JSON.parse(open('./users.json')); }); export const options = { scenarios: { warm_up: { executor: 'ramping-vus', startVUs: 0, stages: [ { duration: '2m', target: 100 }, { duration: '5m', target: 100 }, { duration: '2m', target: 0 }, ], }, spike: { executor: 'ramping-arrival-rate', startRate: 0, timeUnit: '1s', preAllocatedVUs: 500, stages: [ { duration: '30s', target: 500 }, { duration: '1m', target: 500 }, { duration: '30s', target: 0 }, ], startTime: '9m', }, }, thresholds: { http_req_duration: ['p(95)<500', 'p(99)<1500'], http_req_failed: ['rate<0.01'], }, }; export default function () { const user = users[Math.floor(Math.random() * users.length)]; const loginRes = http.post(`${__ENV.BASE_URL}/api/auth/login`, JSON.stringify({ email: user.email, password: user.password, }), { headers: { 'Content-Type': 'application/json' } }); check(loginRes, { 'login 200': (r) => r.status === 200 }); const token = loginRes.json('access_token'); if (!token) return; const headers = { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' }; const catalogRes = http.get(`${__ENV.BASE_URL}/api/catalog?page=1`, { headers }); check(catalogRes, { 'catalog 200': (r) => r.status === 200 }); sleep(Math.random() * 2 + 1); } 

Several important details in this script. SharedArray loads data once across all VUs rather than copying to each — with 500 VUs and a file of 10,000 lines, the memory difference is significant. ramping-arrival-rate simulates real load in terms of RPS, not number of users — this is important: 100 VUs with sleep(1) give ~100 RPS, but 100 VUs without sleep can give thousands, which does not reflect real behavior.

Thresholds are the main pass/fail tool. p(95)<500 means 95% of requests must complete within 500ms. If the threshold is not met, k6 exits with code 1 — the CI pipeline can catch that.

k6 run --out influxdb=http://localhost:8086/k6 script.js — streaming metrics in real time. Combined with a Grafana dashboard (ID 2587 from grafana.com), you get a live view of the test: RPS, latency percentiles, error count, active VUs.

What to choose: JMeter, Gatling, or k6?

The choice of tool is often determined not by technical characteristics, but by who writes the tests: developers are used to code (k6/Gatling), QA teams to GUI (JMeter). k6 consumes about 5 times less memory than JMeter with the same number of virtual users, making it the best choice for continuous testing in CI/CD.

Chaos Engineering: break controllably to avoid unexpected failures

Chaos Engineering is the next level after load testing. Load testing checks that the system handles traffic. Chaos checks that the system degrades properly when components fail.

Chaos Engineering — the most mature tool for Kubernetes. Installed as CRD and managed via YAML or Web UI.

Scenario: what happens when one of three database instances stops responding?

apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: db-network-delay namespace: production spec: action: delay mode: one selector: namespaces: - production labelSelectors: app: postgresql delay: latency: "200ms" correlation: "25" jitter: "50ms" duration: "10m" scheduler: cron: "@every 1h" 

This manifest adds 200ms network delay to one random pod with label app: postgresql once an hour for 10 minutes. Run it in parallel with a load test — see how p(99) latency of the application changes.

PodChaos for resilience testing to restarts:

apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: api-pod-kill spec: action: pod-kill mode: fixed-percent value: "33" selector: namespaces: [production] labelSelectors: app: api-service gracePeriod: 0 duration: "5m" 

33% of api-service pods are killed instantly (without graceful shutdown) once per run. Expected result: error metrics temporarily rise, but Kubernetes recovers the pods, and after 30–60 seconds errors disappear. If errors persist, there is a problem with readiness probes or deployment.

GameDay principle: Chaos experiments should be conducted during working hours with the team present, not alone at night. The goal is not to break prod, but to build muscle memory for incident response.

What to test first with Chaos Engineering?

• Failure of one availability zone (NetworkChaos on entire namespace of one AZ)
• Memory exhaustion of a key service (StressChaos with memoryWorkers: 4, memoryBytes: "1GB")
• Latency to external API (HTTPChaos with delay on outgoing requests)
• Disk full (IOChaos + additional container writing garbage)

Chaos Engineering is not about breaking everything. It's about knowing in advance what exactly will break and in what order. Our certified engineers have conducted more than 50 such experiments on production projects.

How to interpret load test results?

The test is finished, numbers are available. After completion, we analyze three key scenarios.

p(50) is normal, but p(99) is catastrophic. This indicates a tail distribution — requests hitting a slow path. Often this is a cache miss, a full table scan with certain parameters, or a GC pause in JVM.

connection reset by peer errors under load — connection limit exhausted. Check ulimit -n on the server, SOMAXCONN, keepalive settings in Nginx/HAProxy, pool size in the application connection pool.

Load increases linearly, latency is stable up to 60% CPU, then jumps sharply — a classic sign of a CPU-bound bottleneck. A profiler (async-profiler for JVM, py-spy for Python) will show what is consuming CPU. Such diagnostics reduce cloud infrastructure costs by an average of 2x for our clients.

What is included in load testing?

• Architecture and business scenario analysis (identify critical endpoints)
• Script writing and debugging (k6/JMeter/Gatling)
• Smoke test, load test, spike test, soak test
• Chaos experiments (if required)
• Detailed report with graphs: latency percentiles, RPS, errors, bottleneck components
• Optimization recommendations (configs, infrastructure, code)
• Integration of tests into your CI/CD (GitLab CI / GitHub Actions)
• Post-release support: production measurements

How to integrate performance tests into CI/CD?

Continuous performance testing ensures that each code change is evaluated for performance degradation. We help integrate k6 or Gatling tests into your pipeline, automatically comparing results against baselines and thresholds. This catches regressions before they reach production.

Timelines and format

Pricing is calculated individually after analyzing the architecture and testing goals. Get a project assessment — contact us for a consultation. Order turnkey load testing and guarantee identification of bottlenecks before production finds them.