You launch a service into production. 12 hours later it crashes with OOM—memory leaked. Short load tests (5–10 minutes) showed nothing. Sound familiar? This is a classic scenario where a soak test is needed. We, the engineers at True Tech, design long-duration tests to uncover hidden degradations before they hit your production. One client lost 8 hours of work due to an undetected memory leak in a Node.js application—after a soak test, we found it in the first 3 hours of analysis.
A soak test (also called endurance test) runs a system under normal or moderate load for 4–24 hours. It reveals problems that don't show up in minutes: memory leaks, accumulation of file descriptors, database connection pool degradation, growth of slow queries due to table bloat. Without soak testing, you risk inexplicable failures after several hours of operation.
Why soak tests catch memory leaks
Memory leaks are a classic "slow boil" effect. An application grows by 100–200 MB/hour and crashes with OOM after 12 hours. Short tests simply don't notice the growth. Soak testing with RSS and heap monitoring captures the linear trend and predicts the failure point. According to k6 documentation, 8-hour soak tests detect 90% of memory leaks missed by short tests.
A soak test is 10x more effective than short tests at finding memory leaks—confirmed by our practice on 50+ projects.
What problems does a soak test uncover?
- Memory leaks: application grows by 100–200 MB/hour and crashes with OOM after 12 hours.
- Connection pool exhaustion: database connections not returned to the pool; after 6 hours the pool is empty and new requests wait until timeout.
- Heap accumulation: JVM/Node.js GC handles it for the first 2 hours, then Full GC pauses start affecting latency.
- Table bloat without autovacuum: PostgreSQL bloat after millions of UPDATE/DELETE operations degrades performance without vacuum.
- File descriptor leak: each request opens a log file or socket and doesn't close it; after 8 hours ulimit is exhausted.
| Test type | Duration | Goal | Reveals |
|---|---|---|---|
| Load test | 10–30 min | Check under expected load | Throughput, response time |
| Stress test | 5–15 min | Check under peak load | Failure point, overload errors |
| Soak test | 4–24 hours | Check under moderate load | Memory leaks, degradation, cumulative errors |
How we conduct soak tests: process and tools
Steps
- Analytics: collect your production load profile (traffic, endpoints, scenarios).
- Scenario design: write k6 scripts with a realistic mix of requests (read, write, search).
- Monitoring setup: enable memory, file descriptor, and database metrics (PostgreSQL, MySQL).
- Test execution: on a staging environment with an 8-hour window.
- Trend analysis: run regression on RSS, P95 latency, dead tuple ratio; look for statistically significant growth.
- Report preparation: visualize degradation, provide fix recommendations for code and configuration.
Example k6 scenario
// tests/soak/endurance.js import http from 'k6/http' import { check, sleep } from 'k6' import { Rate, Trend, Gauge } from 'k6/metrics' const errorRate = new Rate('errors') const p95Latency = new Trend('p95_latency_trend', true) const activeUsers = new Gauge('active_users') export const options = { stages: [ { duration: '5m', target: 50 }, // warm-up { duration: '8h', target: 50 }, // 8 hours normal load { duration: '5m', target: 0 }, // cool-down ], thresholds: { // Latency must not degrade during the test http_req_duration: ['p(95)<600'], // No errors allowed (leaks manifest as errors) errors: ['rate<0.001'], // Database connection time must not increase http_req_connecting: ['p(95)<50'], } } const BASE_URL = __ENV.BASE_URL || 'https://staging.example.com' export function setup() { const res = http.post(`${BASE_URL}/api/auth/login`, JSON.stringify({ email: '[email protected]', password: __ENV.TEST_PASSWORD }), { headers: { 'Content-Type': 'application/json' } }) return { token: res.json('token') } } export default function(data) { const headers = { 'Authorization': `Bearer ${data.token}`, 'Content-Type': 'application/json' } activeUsers.add(1) // Mix of operations typical for real traffic const scenario = Math.random() if (scenario < 0.6) { // 60%: read data const r = http.get(`${BASE_URL}/api/products?page=${Math.ceil(Math.random() * 50)}`, { headers }) check(r, { 'read: 200': (r) => r.status === 200 }) errorRate.add(r.status !== 200) } else if (scenario < 0.8) { // 20%: write data (create real records) const r = http.post(`${BASE_URL}/api/cart/items`, JSON.stringify({ productId: Math.ceil(Math.random() * 1000), quantity: 1 }), { headers }) check(r, { 'write: 2xx': (r) => r.status < 300 }) errorRate.add(r.status >= 400) } else if (scenario < 0.9) { // 10%: search const r = http.get(`${BASE_URL}/api/search?q=test&limit=20`, { headers }) check(r, { 'search: 200': (r) => r.status === 200 }) errorRate.add(r.status !== 200) } else { // 10%: user profile const r = http.get(`${BASE_URL}/api/me`, { headers }) check(r, { 'profile: 200': (r) => r.status === 200 }) errorRate.add(r.status !== 200) } // Add p95 for time series p95Latency.add(http.get(`${BASE_URL}/api/health`).timings.duration) sleep(Math.random() * 2 + 0.5) // 0.5–2.5 seconds between requests } Memory leak monitoring
In parallel with k6, run a script to monitor RSS and file descriptors:
#!/bin/bash # scripts/memory-soak-monitor.sh APP_PID=$(pgrep -f "node server.js") LOG_FILE="soak-memory-$(date +%Y%m%d-%H%M).csv" echo "timestamp,rss_mb,heap_used_mb,heap_total_mb,external_mb,fd_count" > $LOG_FILE while true; do TS=$(date -u +%Y-%m-%dT%H:%M:%SZ) METRICS=$(curl -s http://localhost:3000/metrics/memory) RSS=$(echo $METRICS | jq -r '.rss') HEAP_USED=$(echo $METRICS | jq -r '.heapUsed') HEAP_TOTAL=$(echo $METRICS | jq -r '.heapTotal') EXTERNAL=$(echo $METRICS | jq -r '.external') FD_COUNT=$(ls /proc/$APP_PID/fd 2>/dev/null | wc -l) echo "$TS,$RSS,$HEAP_USED,$HEAP_TOTAL,$EXTERNAL,$FD_COUNT" >> $LOG_FILE sleep 60 done And on the application side, expose metrics via an endpoint:
// Express/Fastify endpoint for memory exposure app.get('/metrics/memory', (req, res) => { const mem = process.memoryUsage() res.json({ rss: Math.round(mem.rss / 1024 / 1024), heapUsed: Math.round(mem.heapUsed / 1024 / 1024), heapTotal: Math.round(mem.heapTotal / 1024 / 1024), external: Math.round(mem.external / 1024 / 1024), }) }) PostgreSQL monitoring during soak
-- Table growth (bloat) SELECT relname, n_live_tup, n_dead_tup, round(n_dead_tup::numeric / nullif(n_live_tup + n_dead_tup, 0) * 100, 1) AS dead_pct, last_vacuum, last_autovacuum FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 10; -- Accumulation of idle transactions (connection leak) SELECT count(*), state, wait_event_type FROM pg_stat_activity WHERE pid != pg_backend_pid() GROUP BY state, wait_event_type ORDER BY count DESC; -- Growth of temporary files SELECT temp_files, temp_bytes FROM pg_stat_database WHERE datname = current_database(); Degradation trend analysis
After the test, run a Python script to compute RSS regression:
# analyze_soak.py import pandas as pd import numpy as np from scipy import stats def analyze_memory_trend(csv_file: str): df = pd.read_csv(csv_file, parse_dates=['timestamp']) df['minutes'] = (df['timestamp'] - df['timestamp'].iloc[0]).dt.total_seconds() / 60 slope, intercept, r_value, p_value, std_err = stats.linregress(df['minutes'], df['rss_mb']) hours_to_oom = None if slope > 0: oom_threshold = 4096 current_rss = df['rss_mb'].iloc[-1] hours_to_oom = (oom_threshold - current_rss) / (slope * 60) print(f"Memory growth rate: {slope:.2f} MB/min ({slope*60:.1f} MB/hour)") if hours_to_oom: print(f"Estimated OOM in: {hours_to_oom:.1f} hours") if p_value < 0.01 and slope > 0.1: print("MEMORY LEAK DETECTED (statistically significant growth)") else: print("No significant memory leak detected") return {'slope_mb_per_min': slope, 'r_squared': r_value**2, 'hours_to_oom': hours_to_oom, 'leak_detected': p_value < 0.01 and slope > 0.1} How to interpret soak test results
The built time series of RSS and P95 latency are key to identifying degradation. If the RSS trend slope is positive and statistically significant, it's a leak. P95 latency increasing after 2–4 hours indicates problems with GC or connection pool. Additionally, check dead tuple ratio in PostgreSQL: if it exceeds 10%, that's bloat. For each problem, we provide specific recommendations: from code optimization to database configuration changes.
What's included in our work
- Analysis of your application's architecture and load profile.
- Development of k6 scenarios with realistic user behavior.
- Setup of monitoring for memory, database connections, file descriptors.
- Execution of soak test lasting 8–24 hours on a staging environment.
- Generation of time series graphs and regression trend analysis.
- Detailed report with identified degradations and recommendations for remediation.
- Consultation on code fixes and configuration.
- Free retest if the problem was not detected.
Typical findings and solutions
| Problem | Symptom | Solution |
|---|---|---|
| EventEmitter leak (Node.js) | MaxListenersExceededWarning | Use emitter.removeListener() or once() |
| Unclosed DB connections | Growing connections in pg_stat_activity | pool.release() in finally block or ORM-level connection pooling |
| Accumulating cron jobs | Duplicate background tasks | Add mutex lock (Redis lock) |
| Redis pub/sub leak | Growing number of channel subscriptions | Unsubscribe on connection close |
Our experience and guarantees
We have been doing load testing for over 7 years. We have 50+ projects under our belt, where soak tests prevented critical production failures. We guarantee quality: after completion, you receive a detailed report with graphs and recommendations. If a problem remains undetected, we'll conduct a free retest.
Savings from preventing a single OOM incident can be substantial, while losses from an undetected memory leak are significant.
Timeline: setup and execution of an 8–24 hour soak test with trend analysis takes 2 to 4 business days. We'll assess your project in 1 day.
Order soak testing from our engineers—we'll uncover hidden degradations before they hit your production. Get a consultation and project estimate by contacting us.