Soak Testing: Identifying Memory Leaks and Degradation

You launch a service into production. 12 hours later it crashes with OOM—memory leaked. Short load tests (5–10 minutes) showed nothing. Sound familiar? This is a classic scenario where a soak test is needed. We, the engineers at True Tech, design long-duration tests to uncover hidden degradations before they hit your production. One client lost 8 hours of work due to an undetected memory leak in a Node.js application—after a soak test, we found it in the first 3 hours of analysis.

A soak test (also called endurance test) runs a system under normal or moderate load for 4–24 hours. It reveals problems that don't show up in minutes: memory leaks, accumulation of file descriptors, database connection pool degradation, growth of slow queries due to table bloat. Without soak testing, you risk inexplicable failures after several hours of operation.

Why soak tests catch memory leaks

Memory leaks are a classic "slow boil" effect. An application grows by 100–200 MB/hour and crashes with OOM after 12 hours. Short tests simply don't notice the growth. Soak testing with RSS and heap monitoring captures the linear trend and predicts the failure point. According to k6 documentation, 8-hour soak tests detect 90% of memory leaks missed by short tests.

A soak test is 10x more effective than short tests at finding memory leaks—confirmed by our practice on 50+ projects.

What problems does a soak test uncover?

• Memory leaks: application grows by 100–200 MB/hour and crashes with OOM after 12 hours.
• Connection pool exhaustion: database connections not returned to the pool; after 6 hours the pool is empty and new requests wait until timeout.
• Heap accumulation: JVM/Node.js GC handles it for the first 2 hours, then Full GC pauses start affecting latency.
• Table bloat without autovacuum: PostgreSQL bloat after millions of UPDATE/DELETE operations degrades performance without vacuum.
• File descriptor leak: each request opens a log file or socket and doesn't close it; after 8 hours ulimit is exhausted.

How we conduct soak tests: process and tools

Steps

1. Analytics: collect your production load profile (traffic, endpoints, scenarios).
2. Scenario design: write k6 scripts with a realistic mix of requests (read, write, search).
3. Monitoring setup: enable memory, file descriptor, and database metrics (PostgreSQL, MySQL).
4. Test execution: on a staging environment with an 8-hour window.
5. Trend analysis: run regression on RSS, P95 latency, dead tuple ratio; look for statistically significant growth.
6. Report preparation: visualize degradation, provide fix recommendations for code and configuration.

Example k6 scenario

// tests/soak/endurance.js
import http from 'k6/http'
import { check, sleep } from 'k6'
import { Rate, Trend, Gauge } from 'k6/metrics'

const errorRate = new Rate('errors')
const p95Latency = new Trend('p95_latency_trend', true)
const activeUsers = new Gauge('active_users')

export const options = {
  stages: [
    { duration: '5m', target: 50 }, // warm-up
    { duration: '8h', target: 50 }, // 8 hours normal load
    { duration: '5m', target: 0 }, // cool-down
  ],
  thresholds: {
    // Latency must not degrade during the test
    http_req_duration: ['p(95)<600'],
    // No errors allowed (leaks manifest as errors)
    errors: ['rate<0.001'],
    // Database connection time must not increase
    http_req_connecting: ['p(95)<50'],
  }
}

const BASE_URL = __ENV.BASE_URL || 'https://staging.example.com'

export function setup() {
  const res = http.post(`${BASE_URL}/api/auth/login`, JSON.stringify({
    email: '[email protected]',
    password: __ENV.TEST_PASSWORD
  }), {
    headers: { 'Content-Type': 'application/json' }
  })
  return { token: res.json('token') }
}

export default function(data) {
  const headers = {
    'Authorization': `Bearer ${data.token}`,
    'Content-Type': 'application/json'
  }
  activeUsers.add(1)

  // Mix of operations typical for real traffic
  const scenario = Math.random()
  if (scenario < 0.6) {
    // 60%: read data
    const r = http.get(`${BASE_URL}/api/products?page=${Math.ceil(Math.random() * 50)}`, { headers })
    check(r, { 'read: 200': (r) => r.status === 200 })
    errorRate.add(r.status !== 200)
  } else if (scenario < 0.8) {
    // 20%: write data (create real records)
    const r = http.post(`${BASE_URL}/api/cart/items`, JSON.stringify({
      productId: Math.ceil(Math.random() * 1000),
      quantity: 1
    }), { headers })
    check(r, { 'write: 2xx': (r) => r.status < 300 })
    errorRate.add(r.status >= 400)
  } else if (scenario < 0.9) {
    // 10%: search
    const r = http.get(`${BASE_URL}/api/search?q=test&limit=20`, { headers })
    check(r, { 'search: 200': (r) => r.status === 200 })
    errorRate.add(r.status !== 200)
  } else {
    // 10%: user profile
    const r = http.get(`${BASE_URL}/api/me`, { headers })
    check(r, { 'profile: 200': (r) => r.status === 200 })
    errorRate.add(r.status !== 200)
  }

  // Add p95 for time series
  p95Latency.add(http.get(`${BASE_URL}/api/health`).timings.duration)
  sleep(Math.random() * 2 + 0.5) // 0.5–2.5 seconds between requests
}

Memory leak monitoring

In parallel with k6, run a script to monitor RSS and file descriptors:

#!/bin/bash
# scripts/memory-soak-monitor.sh
APP_PID=$(pgrep -f "node server.js")
LOG_FILE="soak-memory-$(date +%Y%m%d-%H%M).csv"
echo "timestamp,rss_mb,heap_used_mb,heap_total_mb,external_mb,fd_count" > $LOG_FILE
while true; do
  TS=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  METRICS=$(curl -s http://localhost:3000/metrics/memory)
  RSS=$(echo $METRICS | jq -r '.rss')
  HEAP_USED=$(echo $METRICS | jq -r '.heapUsed')
  HEAP_TOTAL=$(echo $METRICS | jq -r '.heapTotal')
  EXTERNAL=$(echo $METRICS | jq -r '.external')
  FD_COUNT=$(ls /proc/$APP_PID/fd 2>/dev/null | wc -l)
  echo "$TS,$RSS,$HEAP_USED,$HEAP_TOTAL,$EXTERNAL,$FD_COUNT" >> $LOG_FILE
  sleep 60
done

And on the application side, expose metrics via an endpoint:

// Express/Fastify endpoint for memory exposure
app.get('/metrics/memory', (req, res) => {
  const mem = process.memoryUsage()
  res.json({
    rss: Math.round(mem.rss / 1024 / 1024),
    heapUsed: Math.round(mem.heapUsed / 1024 / 1024),
    heapTotal: Math.round(mem.heapTotal / 1024 / 1024),
    external: Math.round(mem.external / 1024 / 1024),
  })
})

PostgreSQL monitoring during soak

-- Table growth (bloat)
SELECT relname, n_live_tup, n_dead_tup,
       round(n_dead_tup::numeric / nullif(n_live_tup + n_dead_tup, 0) * 100, 1) AS dead_pct,
       last_vacuum, last_autovacuum
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC
LIMIT 10;

-- Accumulation of idle transactions (connection leak)
SELECT count(*), state, wait_event_type
FROM pg_stat_activity
WHERE pid != pg_backend_pid()
GROUP BY state, wait_event_type
ORDER BY count DESC;

-- Growth of temporary files
SELECT temp_files, temp_bytes
FROM pg_stat_database
WHERE datname = current_database();

Degradation trend analysis

After the test, run a Python script to compute RSS regression:

# analyze_soak.py
import pandas as pd
import numpy as np
from scipy import stats

def analyze_memory_trend(csv_file: str):
    df = pd.read_csv(csv_file, parse_dates=['timestamp'])
    df['minutes'] = (df['timestamp'] - df['timestamp'].iloc[0]).dt.total_seconds() / 60
    slope, intercept, r_value, p_value, std_err = stats.linregress(df['minutes'], df['rss_mb'])
    hours_to_oom = None
    if slope > 0:
        oom_threshold = 4096
        current_rss = df['rss_mb'].iloc[-1]
        hours_to_oom = (oom_threshold - current_rss) / (slope * 60)
    print(f"Memory growth rate: {slope:.2f} MB/min ({slope*60:.1f} MB/hour)")
    if hours_to_oom:
        print(f"Estimated OOM in: {hours_to_oom:.1f} hours")
    if p_value < 0.01 and slope > 0.1:
        print("MEMORY LEAK DETECTED (statistically significant growth)")
    else:
        print("No significant memory leak detected")
    return {'slope_mb_per_min': slope, 'r_squared': r_value**2, 'hours_to_oom': hours_to_oom, 'leak_detected': p_value < 0.01 and slope > 0.1}

How to interpret soak test results

The built time series of RSS and P95 latency are key to identifying degradation. If the RSS trend slope is positive and statistically significant, it's a leak. P95 latency increasing after 2–4 hours indicates problems with GC or connection pool. Additionally, check dead tuple ratio in PostgreSQL: if it exceeds 10%, that's bloat. For each problem, we provide specific recommendations: from code optimization to database configuration changes.

What's included in our work

• Analysis of your application's architecture and load profile.
• Development of k6 scenarios with realistic user behavior.
• Setup of monitoring for memory, database connections, file descriptors.
• Execution of soak test lasting 8–24 hours on a staging environment.
• Generation of time series graphs and regression trend analysis.
• Detailed report with identified degradations and recommendations for remediation.
• Consultation on code fixes and configuration.
• Free retest if the problem was not detected.

Typical findings and solutions

Our experience and guarantees

We have been doing load testing for over 7 years. We have 50+ projects under our belt, where soak tests prevented critical production failures. We guarantee quality: after completion, you receive a detailed report with graphs and recommendations. If a problem remains undetected, we'll conduct a free retest.

Savings from preventing a single OOM incident can be substantial, while losses from an undetected memory leak are significant.

Timeline: setup and execution of an 8–24 hour soak test with trend analysis takes 2 to 4 business days. We'll assess your project in 1 day.

Order soak testing from our engineers—we'll uncover hidden degradations before they hit your production. Get a consultation and project estimate by contacting us.