Automating integration tests in CI/CD: Docker and parallelism

Why CI/CD integration tests slow down deployment

The pipeline stalls not at image build, but at the stage where tests start hitting real dependencies: the database, message queue, external service.

If the run is sequential, each scenario first brings up its own environment, waits for it to become ready, and only then makes the first request — across dozens of scenarios this amounts to hours of waiting paid for by every deployment.

A separate problem is when there is one environment for everyone. Tests write to shared tables, see data from a neighboring scenario, and fail for reasons unrelated to the code change.

Reproducing such a failure locally is almost impossible: a developer's machine has a different execution order and a different data set, so investigation takes hours rather than minutes.

Then the main destroyer kicks in — false failures. A couple of flaky scenarios turn the build red, it is restarted manually, and soon the result stops being read: changes are merged on top of a failed build.

Thus build time and an unstable pipeline become the same risk — more and more edits accumulate in a single release, and the cause of a failure in production has to be sought across the entire accumulated volume.

We look at how this is addressed: by isolating environments at the Docker level, running independent suites in parallel, and cutting off flaky scenarios before they enter the mandatory set.

Parallel execution and test environment isolation

Independent streams instead of one long run

The test suite is divided into groups that share neither data nor external resources, and the groups run in parallel in separate containers.

Each stream brings up its own environment from the same image, so the result does not depend on what a neighbor in the run has managed to do. Duration is determined by the longest group, not the sum of all: with even division, the speedup is close to the number of streams.

Test data isolation

Each stream works in its own database schema or in a separate instance brought up for the duration of the run, rather than in a shared test database.

Data is created by a setup script before the start and deleted afterward, so an order update scenario will not see a record created by a parallel test. Test data isolation removes an entire class of flaky failures that were reproducible only under a certain execution order.

Run stability and honest retries

A retry is triggered only for operations where network instability is acceptable, not for the whole test. The report marks a failure as infrastructure-related or logic-related, so a red build again means a defect, not noise. The share of false failures drops severalfold, and the team stops ignoring build notifications.

Fast build recovery

Build recovery time is reduced because only the failed group is restarted on a warm dependency cache, rather than the entire suite. Diagnostics are based on the logs of a specific container with preserved state, so the cause is found in minutes rather than over a series of runs.

Parallel test execution and environment isolation together provide predictable duration: variation between runs is measured in tens of seconds and does not depend on which scenarios happened to coincide in a single run.

Options for organizing the test environment

The choice of environment determines two things: how many tests can be run simultaneously and how stable the result is.

Integration tests depend on the database, queues, and external services — on a shared environment, runs compete for data and fail for reasons unrelated to the code.

The main dividing line is whether the environment lives permanently or is brought up for the duration of the run. The former starts instantly and is suitable for manual acceptance, while the latter provides isolation and true parallelism. Below are the options that are actually chosen between.

In practice, an ephemeral environment is most often used for the main suite and a permanent environment for scenarios that cannot be isolated.

Implementation order: from audit to handover

The route is built so that each step produces a verifiable result and does not stop the current release. First we record how the pipeline works now — only then do we change the order of execution.

1. Build pipeline audit. We analyze where time is spent: environment setup, duplicate steps, tests failing not because of code. The output is a map of bottlenecks with an estimate of gains for each item.
2. Test environment scheme. We determine the set of services for one run, the method for isolating databases and queues, and the order of populating test data. The scheme is agreed before work: it defines what can be parallelized at all.
3. Container setup. We build images with cached dependencies, and instead of fixed pauses we add service readiness checks. Environment setup stops being the longest stage.
4. Parallelization. We divide the suite into independent groups, separate them across individual environment copies, and remove shared mutable resources. Run time decreases approximately in proportion to the number of groups.
5. Stability check. We run the suite in series, mark flaky tests, and fix the causes: races, shared data, dependence on start order. Quarantine is formalized as a fix task, not as exclusion from the suite.
6. Handover to operations. We hand over configurations, the environment scheme, the process, and an analysis of typical failures, and hold a review with the team. After that, changes are made according to the documented order, not from memory.

Configurations, schemes, and policies as output

We hand over not a pipeline that “works for us” but a kit that the customer's team maintains itself.

Below is what physically remains in the repository and wiki after delivery: configurations, schemes, policies, and documents by which a new engineer brings up the environment and investigates a failure without messaging us.

• Build configuration files — description of stages, image layer caching, branch run matrix: tests run on every change, not on a schedule.
• Environment scheme — which services are brought up for the integration run, their connections, initialization order, and readiness checkpoints; the scheme matches the real environment.
• Secret access rules — where credentials are stored, what is visible from pull requests from forks, how a secret is injected only into protected branches and does not end up in logs.
• Parallelism policy — how test suites are split, how many environment copies are brought up simultaneously, and by what criterion tests are considered independent.
• Test documentation — what each suite covers, what data it needs, how to reproduce a specific scenario locally with a single command.
• Failure investigation process — who reviews a failed run, within what time, how to distinguish a flaky test from a real regression, and when a test is removed from the run.
• Report and notification templates — where the result goes, what slices the on-call engineer sees, where the run history for recent weeks is stored.
• Change procedure — how edits to the test configuration go through review and are validated on the infrastructure itself, not after deployment.

The kit is designed for handover: the customer's team changes suites and environments itself, and we get involved only where the infrastructure itself needs further development.

Case study: how run time was reduced threefold

The project is a service with a dozen interconnected components: an API, a queue, two databases, and an external payment circuit. Integration tests ran on a shared environment that had a life of its own: data from previous runs, other people's branches, manual edits.

The run took more than forty minutes, and about every fifth failure was false — the test failed not because of code but because of the environment.

The team stopped trusting the signal: a red build was manually rechecked, restarted “just in case”, and the release came out once every two weeks. Below is exactly what was changed and how it was measured.

What we did

Each run received an isolated set of containers that is brought up for the duration of the test and removed afterward. Database state is no longer inherited: the schema is applied from scratch, and initial data is set by fixtures.

The pipeline on GitLab CI, instead of one long stage, launches parallel groups by domain — orders, payments, notifications — each in its own environment.

External dependencies were replaced with stubs with fixed scenarios, including timeouts and partial failures. Instead of a fixed pause before tests, service readiness waiting was introduced — this removed the class of failures where something “did not have time to start”.

Results

The signal again means what it should: a red run is a real break in the code, not an environment failure. The freed-up time went not into reporting but into shortening the interval between a change and its verification on a live environment.

How CI/CD integration tests differ from unit tests

The boundary between levels runs along process and network. A unit test lives inside a single unit of code: dependencies are replaced with stubs, and branching, data transformations, and error handling are checked.

It does not answer whether a database query was assembled correctly, whether a message reached the queue, and whether the response schema of an external service matched.

An integration test checks exactly this — the seams between services.

The database is brought up in a container, the broker and external APIs are replaced with predictable stubs, and the suite captures contracts, migrations, transaction rollback, retries on network failures, and serialization. Such a run catches a class of defects unreachable at the unit level.

Mixing both levels in one run is expensive for two reasons. An integration suite is an order of magnitude slower: bringing up the environment and waiting for services to become ready consume most of the time.

And diagnostics are worse — a failure does not say where the defect is: in the logic, in the data schema, or in an unready environment.

Hence the test pyramid: many fast unit tests, fewer integration tests, and a few end-to-end scenarios. The proportions are chosen so that the bulk of defects are caught by cheap levels, while expensive ones verify what they physically cannot see.

The separation of suites is enforced technically: separate run commands, tagging, different pipeline stages.

Unit tests run on every change in a branch and provide feedback in minutes; integration tests run on merge to the main branch and before release, with container-level parallelism.

Thus the slow level does not slow down development, and the fast one does not pretend to have checked the seams.

Technical doubts and limitations of the approach

The doubts here are justified: integration tests depend on the environment, and naively moving them into the pipeline produces an unstable result instead of a signal. In practice, most limitations are addressed by configuring the run, not by rewriting the checks themselves.

Is a permanent environment needed?

No. The environment is described declaratively and brought up for the duration of the run, so the test starts on a clean database and does not inherit data from previous runs. We keep a permanent circuit only for acceptance checks and demonstrations.

Where is the limit of parallelism?

The limitation is one — shared state. Tests writing to the same schema begin to interfere with each other: locks, races, false failures.

This is solved by isolation — a separate schema or prefix per stream, wrapping in a transaction with rollback, a separate database instance for the heaviest suites. After that, the number of parallel streams is limited by node resources rather than conflicts.

Can the run be sped up without rewriting the tests?

Yes, and primarily by optimizing environment setup: image layer cache, a prebuilt application image instead of building on every run, preloading dependencies.

Then — splitting the suite into fast and slow groups, retrying only failed checks, warming up connections before the first request. We prepare test data with migrations and seeds from a snapshot, rather than accumulating it manually.

Let's discuss the task and come back with an estimate and plan

To come back with a plan rather than general words, we need the context of your build. Describe the task: which scenarios must pass on every commit, how long the run currently takes, and which tests fail most often. That is enough to make the conversation concrete.

In a call with an engineer, we will analyze where the bottleneck is — environment startup in Docker, sequential execution, or flaky tests — and what can realistically be sped up with parallelism. What to send in advance and what we will clarify:

• current build configuration file and the list of services brought up for tests;
• where the build runs: your own servers or an external platform, and whether there are access restrictions;
• the list of scenarios that need to be covered first, and those that can be postponed;
• how test data is currently stored and whether there is an environment close to production;
• rollback requirements: what should happen if the run fails at the deployment stage;
• who on your team will accept the work and by what criteria they will consider it complete.

In response, we send an automation work plan: stages, dependencies between them, and a timeline estimate for each. As a separate item — preparation for the start: what we will set up on your side before the first run and what will be needed from your engineers. Once the plan is agreed, we begin.