Recreating Real Traffic: From Logs to k6 Scenarios

Recreating Real Traffic: From Logs to k6 Scenarios

A typical mistake in load testing is using uniform request pacing with a fixed number of virtual users. In reality, traffic has peaks (morning and evening), different user types (mobile browsers, API clients), random pauses, and an 80/20 distribution. For example, 80% of requests hit 20% of pages. Synthetic tests often miss issues with caching, session state, and concurrency.

We offer an approach based on analyzing real traffic from Nginx logs or Google Analytics, and generating k6 scenarios that accurately reproduce real user behavior. Realistic testing is 3 times better than uniform load in identifying performance issues. Clients achieve significant savings on debugging by catching issues early. Contact us to discuss your project.

How Log Analysis Works

# Extract patterns from nginx access log import re from collections import Counter, defaultdict import json def analyze_access_log(log_file: str): pattern = re.compile( r'(?P<ip>\S+) .+ \[(?P<time>[^\]]+)\] ' r'"(?P<method>\w+) (?P<path>[^"]+) HTTP/\d+" ' r'(?P<status>\d+) (?P<bytes>\d+)' ) endpoint_counts = Counter() method_counts = Counter() hourly_traffic = defaultdict(int) with open(log_file) as f: for line in f: m = pattern.match(line) if not m: continue # Normalize path (remove IDs) path = re.sub(r'/\d+', '/{id}', m.group('path').split('?')[0]) endpoint_counts[f"{m.group('method')} {path}"] += 1 method_counts[m.group('method')] += 1 # Hourly distribution hour = m.group('time').split(':')[1] hourly_traffic[hour] += 1 total = sum(endpoint_counts.values()) print("=== Top Endpoints (% of traffic) ===") for endpoint, count in endpoint_counts.most_common(20): pct = count / total * 100 print(f" {pct:.1f}% {endpoint}") print("\n=== Hourly Distribution ===") for hour in sorted(hourly_traffic): bar = '█' * (hourly_traffic[hour] // 100) print(f" {hour}:00 {bar} {hourly_traffic[hour]}") # Export for k6 scenario weights = {ep: round(cnt/total, 3) for ep, cnt in endpoint_counts.most_common(20)} return weights 

The extracted weights are exported to JSON and used to generate k6 scenarios. Analyzing access logs lets us identify the 20% of endpoints generating 80% of traffic, following the Pareto principle.

Limitations of Uniform Traffic

Uniform load doesn't create the "crowd" effect: when 1000 users simultaneously navigate to a product after a social media post. It doesn't test session caching, database locks under concurrent writes, or degradation under sustained peaks. Realistic simulation with a Pareto (80/20) distribution and session behavior reproduces such scenarios, uncovering bottlenecks before production deployment.

How to Build a Scenario Based on Logs

From the extracted weights we generate k6 scenarios. For each user type we create a separate executor with different intensity. For example, 40% traffic – anonymous browsers, 50% – logged-in users, 10% – API clients. Scenarios include random pauses, branching, and probabilistic transitions.

// tests/realistic/user-journey.js import http from 'k6/http' import { check, sleep } from 'k6' import { SharedArray } from 'k6/data' import { randomItem, randomIntBetween } from 'https://jslib.k6.io/k6-utils/1.4.0/index.js' // Load test data from CSV const users = new SharedArray('users', function() { return open('./data/test-users.csv').split('\n') .slice(1) .map(row => { const [email, token, userId] = row.split(',') return { email, token, userId } }) }) const searchTerms = new SharedArray('searches', function() { return open('./data/popular-searches.txt').split('\n').filter(Boolean) }) export const options = { scenarios: { // Anonymous browsers (40% of traffic) anonymous_browse: { executor: 'ramping-vus', startVUs: 0, stages: [ { duration: '5m', target: 40 }, { duration: '30m', target: 40 }, { duration: '5m', target: 0 } ], exec: 'anonymousBrowse' }, // Logged-in users (50% of traffic) logged_in_users: { executor: 'ramping-vus', startVUs: 0, stages: [ { duration: '5m', target: 50 }, { duration: '30m', target: 50 }, { duration: '5m', target: 0 } ], exec: 'loggedInJourney' }, // API clients (10% of traffic) api_clients: { executor: 'constant-arrival-rate', rate: 10, timeUnit: '1s', duration: '40m', preAllocatedVUs: 20, exec: 'apiClient' } }, thresholds: { http_req_duration: ['p(95)<800'], http_req_failed: ['rate<0.01'], } } const BASE = __ENV.BASE_URL || 'https://staging.example.com' // Scenario: anonymous browser export function anonymousBrowse() { // Landing → catalog → product → exit http.get(`${BASE}/`) sleep(randomIntBetween(1, 4)) const category = randomItem(['electronics', 'clothing', 'books', 'sports']) http.get(`${BASE}/api/products?category=${category}&limit=20`) sleep(randomIntBetween(2, 8)) // 30% leave immediately, 70% view product if (Math.random() > 0.3) { const productId = randomIntBetween(1, 500) http.get(`${BASE}/api/products/${productId}`) sleep(randomIntBetween(3, 15)) } // 20% perform a search if (Math.random() < 0.2) { const term = randomItem(searchTerms) http.get(`${BASE}/api/search?q=${encodeURIComponent(term)}`) sleep(randomIntBetween(1, 5)) } } // Scenario: logged-in user export function loggedInJourney() { const user = randomItem(users) const headers = { 'Authorization': `Bearer ${user.token}`, 'Content-Type': 'application/json' } // Profile http.get(`${BASE}/api/me`, { headers }) sleep(randomIntBetween(1, 3)) // Browse products for (let i = 0; i < randomIntBetween(2, 8); i++) { const productId = randomIntBetween(1, 500) http.get(`${BASE}/api/products/${productId}`, { headers }) sleep(randomIntBetween(2, 10)) } // 40% add to cart if (Math.random() < 0.4) { http.post(`${BASE}/api/cart/items`, JSON.stringify({ productId: randomIntBetween(1, 500), quantity: randomIntBetween(1, 3) }), { headers }) sleep(randomIntBetween(1, 3)) // 60% of those who added — checkout if (Math.random() < 0.6) { http.get(`${BASE}/api/cart`, { headers }) sleep(randomIntBetween(2, 5)) const checkout = http.post(`${BASE}/api/orders`, JSON.stringify({ paymentMethod: 'saved_card', shippingAddressId: 1 }), { headers }) check(checkout, { 'order created': (r) => r.status === 201 }) } } } // Scenario: API client (integration) export function apiClient() { const apiKey = __ENV.API_KEY const headers = { 'X-API-Key': apiKey, 'Content-Type': 'application/json' } // Sync products const r = http.get(`${BASE}/api/v1/products?since=${Date.now() - 3600000}`, { headers }) check(r, { 'api: 200': (r) => r.status === 200 }) } 

A realistic scenario provides 1.5 times more accurate modeling compared to uniform load.

Pareto Distribution in k6

Real traffic: 20% of pages receive 80% of traffic. This is modeled in k6 with a function that generates IDs according to a power law:

// Pareto distribution ID generator function paretoId(maxId, shape = 1.5) { const u = Math.random() return Math.ceil(maxId * Math.pow(1 - u, 1 / shape)) } // Usage const productId = paretoId(10000) // mostly IDs 1-200, rarely ID 9000+ 

Comparison: Synthetic vs Realistic Testing

How We Work

1. Log analysis: Collect Nginx access logs or Google Analytics data, extract patterns (endpoints, statuses, hourly distribution).
2. Scenario design: Determine user types (anonymous, logged-in, API), build probabilistic behavior models.
3. k6 implementation: Write JavaScript scenarios with executors, random pauses, and branching.
4. Test run: Execute test on staging environment, collect metrics (LCP, CLS, TTFB, errors).
5. Analysis and report: Identify bottlenecks, compare with baseline, provide optimization recommendations.

What's Included

• Scenario documentation describing behavior of each user type.
• k6 configurations (options, thresholds, threshold values).
• Test results report: load graphs, response time percentiles, errors.
• Performance optimization recommendations (database indexes, caching, async queues).
• Support during first run and result interpretation.

Company Metrics

With 5+ years of experience in performance testing and 200+ projects delivered, we guarantee all scenarios are verified on our test benches. Our clients report 15–30% improvement in response times after implementing our recommendations.

Pricing and Timelines

Development of a realistic load test scenario based on real traffic analysis takes 2 to 5 working days. Pricing starts from $2500 for a basic scenario and goes up to $7500 for complex multi-user tests. Typical savings: $15000 on early bug detection.

Order a realistic load testing scenario turnkey. Get a consultation from engineers.