Parallel Tests in CI/CD: Sharding and Isolation

Parallel Tests in CI/CD: Sharding and Isolation

Tests that locally pass in 2 minutes suddenly take 25 in CI. You look at the log — the runner spends 8 minutes installing dependencies, another 3 on build, and then 14 minutes linearly running 800 test cases one by one. Parallel test execution is not just a --parallel flag. It's proper sharding, shared state management, database isolation, and smart caching. Our experience shows: proper setup cuts pipeline time by 3-5 times, and runner resource savings reach 40%. On one project with 1500 tests, we reduced time from 45 to 12 minutes.

Why parallel tests fail in CI?

Most test frameworks support parallel execution out of the box. Jest documentation recommends using --shard for large test suites. RSpec via parallel_tests gem, pytest via pytest-xdist. But simply enabling them is not enough.

The shared state problem: tests that write to the same database cannot safely run in parallel. user_id = 1 in one test conflicts with user_id = 1 in another. For databases, the solution is a separate DB per worker or transaction rollback after each test. In Rails — DatabaseCleaner with :transaction strategy for parallel tests. In Django — --keepdb with isolated schemas.

The test order problem: some tests have side effects — create files, modify env variables, mock global singletons. Under parallel execution, these side effects collide. It manifests unpleasantly: a test passes when run alone and fails in CI when parallel.

How to achieve uniform sharding?

For large test suites, parallelism inside a single runner is insufficient — tests must be distributed across multiple runners. This is test sharding.

In GitLab CI, the parallel parameter launches multiple job copies:

---
test:
  parallel: 6 # launches 6 job copies
  script:
    - bundle exec rspec --format progress \
      $(bundle exec ruby scripts/split_tests.rb $CI_NODE_INDEX $CI_NODE_TOTAL)
---

GitLab sets CI_NODE_INDEX (0..5) and CI_NODE_TOTAL (6) for each shard. The split_tests.rb script divides the spec file list into equal parts by index.

In GitHub Actions via matrix:

---
test:
  strategy:
    matrix:
      shard: [1, 2, 3, 4, 5, 6]
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - name: Run tests (shard ${{ matrix.shard }}/6)
      run: |
        npx jest --shard=${{ matrix.shard }}/6 --coverage
---

Jest natively supports --shard=N/TOTAL since version 28. No need to write a splitting script — Jest divides tests by file name hash.

When to apply time-based sharding?

Simple division by file count yields uneven shards: one may contain 5 long integration tests, another 50 fast unit tests. The right approach is to divide by expected execution time.

RSpec can use rspec_junit_formatter to save each test's time in JUnit XML. knapsack_pro or a custom script reads historical timings and balances shards so all finish around the same time. Knapsack Pro does this via cloud API — it collects stats from all runners and optimizes distribution with each run.

For pytest — pytest-split with a similar approach: pytest --splits 6 --group 1 --durations-path .test_durations.

Compare approaches:

Time-based sharding (Knapsack) gives 1.5-2 times more even distribution than file count, as confirmed on our projects.

What if tests depend on shared state?

If tests use shared files, global variables, or external services, parallel execution will cause race conditions. Solution — full isolation: each worker gets its own temporary files (via Tempfile), its own env variables (via dotenv or CI configuration). For external services — use mocks or separate containers per worker. In Kubernetes, each shard can run in a separate pod with isolated resources.

Database isolation for parallel tests

In Rails with parallel_tests, each worker gets its own DB: test, test2, test3... Created via rake parallel:create parallel:migrate. In database.yml:

test: database: myapp_test<%= ENV['TEST_ENV_NUMBER'] %> 

In Django when sharding by runner — each runner creates its own test DB. If using PostgreSQL, ensure the CI user has CREATEDB permissions, otherwise manage.py test will fail with permission denied to create database.

Test reports and analyzing slow tests

With parallel execution, JUnit XML reports are generated separately on each shard. The CI system must aggregate them:

# GitLab — artifacts from all shards are merged automatically artifacts: reports: junit: reports/rspec-*.xml 

CircleCI Test Analytics, Buildkite Test Analytics, GitHub Actions with dorny/test-reporter — all can aggregate reports from multiple shards and build historical trends for each test's execution time. This helps find tests that slowed down after a specific commit.

Comparison of parallel testing support in CI systems

Typical mistakes when configuring parallel tests

What is included in the work

We provide turnkey setup from audit to implementation. As a result, you get:

• Analysis of the current pipeline and identification of bottlenecks.
• Audit of tests for shared state issues with recommendations for fixes.
• Selection and implementation of sharding strategy (by file or by time).
• Configuration of database isolation for parallel runners.
• Setup of report aggregation and alerts for slow tests.
• Documentation for maintenance and team training.

Work stages

1. Analysis of current pipeline time — where exactly is time lost, bottlenecks.
2. Audit of tests for shared state problems — run with --random-seed in different orders.
3. Selection of sharding strategy — by count or by timing history.
4. Configuration of parallelism in CI — parallel: in GitLab, matrix in GitHub Actions.
5. Configuration of DB isolation on parallel runners.
6. Configuration of JUnit report aggregation.
7. Measurement of final pipeline time.

Timeline: 1-3 days depending on test infrastructure complexity. Cost is calculated individually. We guarantee transparent results — you will see pipeline time reduction after the first stage. If you want to speed up your pipeline — request an audit today. Contact us to evaluate your project.