Configuring dependency caching in a CI/CD pipeline

Caching CI/CD dependencies: where the build loses minutes

Every CI/CD pipeline run starts from a clean environment, and the first thing it does is re-download and rebuild the same dependencies.

Re-downloading packages takes from half to two-thirds of the pipeline build time: this means minutes of waiting for a fix after every commit instead of working. And when the external registry responds slowly or rate-limits requests, the pipeline fails not because of code.

Caching CI/CD dependencies removes exactly this part, but it works only with a precise key. A key tied to the lock file invalidates the cache exactly when the set of versions changes and gives a hit on almost every run.

A key by branch or build number gives the opposite: either regular misses or silent sticking to outdated versions, which surfaces later in the staging environment.

The second bottleneck of the build is the path to external registries, not computation. A local proxy registry and artifact storage move downloading to the internal network: packages arrive from disk next to the runners and stop depending on the availability of the public source.

Scheduled cache warm-up also fits here, so the first run after an update doesn't pull everything again.

As a result, pipeline build time is reduced severalfold, and the build itself no longer depends on external infrastructure. Let's break down how this is handled in practice: cache keys and layers, a shared cache for parallel jobs, and a cleanup policy that keeps it from growing.

What changes in build speed and delivery reliability

Dependency cache changes the build profile: the install step stops being the most expensive.

When the storage is warmed, time is determined by compilation and tests, not downloading—the change verification cycle is reduced by 2-4 times, and a fix is verified while the task context is still in your head.

The cache key is calculated from the checksum of the dependency file, so steps that previously went to the registry take ready-made artifacts from local storage. The network drops out of 80-90% of builds: external requests remain only where dependencies actually changed.

The load on the package registry drops by a factor—queues, rate limits, and timeouts at peaks disappear.

At the same time, the registry stops being a single point of failure: when it is unavailable, the pipeline reaches the end on a warmed cache, and only the dependency update step fails.

A separate effect is the classes of failures that disappear completely. Mid-download interruptions, divergence of transitive dependency versions between runs, and builds that go green on the second run are no longer reproduced. Pipeline stability becomes a property of configuration, not luck.

Predictable build time is a direct consequence: the spread between fast and slow runs shrinks to 10-15% instead of twofold. With such a duration, you can already plan releases and keep the delivery window fixed.

Methods of storing build artifacts: comparison of options

Where the cache lives determines how much the dependency installation step adds to each build.

A local cache on the runner takes files directly from disk and works faster than all others, but it is tied to a specific node: if the job moved to another runner or the machine was recreated, the cache is empty, and installation goes from scratch.

A shared cache in object storage removes the node binding, but requires a well-thought-out key and limits for simultaneous writes from multiple builds.

The scope is set by the key: if it includes the hash of the dependency file and the environment version, outdated data is not reused, and branches do not displace each other.

Invalidation rules are more important than volume—a cache without a key and lifetime limit grows faster than it brings acceleration. Therefore, the option is chosen by the pair "speed + maintenance", not by maximum speed in isolation from everything else.

Implementation order: from audit to handover to the team

You don't configure cache by eye: first we measure what share of the cycle time dependency installation takes and how often repeated runs reuse already downloaded packages.

We take a baseline snapshot—time by steps, hit rate, volume of downloaded archives—and only then touch the configuration.

1. Pipeline audit. We break down the jobs: where dependencies are installed, whether the local cache is read between runs, which branches and environments use the same set. We record the current behavior and the causes of misses.
2. Cache key scheme. We build the key from immutable attributes: dependency lock file, runtime version, job type, base image. It must change when the package set is updated and remain stable when application code is edited.
3. Separation of scopes. A separate cache for build, tests, and deployment—mixing makes the key too broad, and jobs start receiving other packages' sets.
4. Hit verification. A series of cold and repeated builds: we compare the hit rate and the install step time. Separately, we run a branch with a dependency update—the key must change.
5. Invalidation. Retention rule: lifetime and volume limit, so the cache does not grow and does not serve outdated package versions.
6. Handover of settings. The configuration lies in the repository next to the code; we give the team the key scheme, cleanup rule, and procedure for misses in short documentation.

Delivery package: configurations, policies, regulations

The delivery package is what remains in the repository and in the system after caching works.

Without it, build acceleration rests on one engineer: when the team changes or storage is migrated, the pipeline quietly returns to full dependency downloads, and the cause is visible only by increased build time.

• Pipeline configuration with explicit cache restore and save steps and their order relative to dependency installation.
• Key scheme: what the key is formed from (lock file, branch, task type), which layers are shared, and which are tied to a specific build.
• Retention policy: record lifetime, maximum volume, eviction rule for old layers, and frequency of full invalidation.
• Access and separation rules: who can overwrite the shared cache, what fundamentally does not get into it—primarily secrets and private registries.
• Fault-tolerance scheme: build behavior when storage is unavailable—the pipeline degrades to normal downloading but does not fail.
• Build documentation: how to reproduce the environment locally, what to do when the key does not match, and how to verify that dependencies were taken from cache rather than downloaded again.
• Maintenance regulations: frequency of record review, invalidation procedure when the base image changes, and the person responsible for key scheme changes.
• Diagnostics instructions for typical failures: false hits after dependency updates, storage volume growth, slowdown at the unpacking stage.

Separately, we fix the entry point: how to add a cache for a new service without breaking already configured ones. This is the minimum after which build acceleration is reproducible rather than one-off.

Case: monorepo build accelerated severalfold

A monorepo with 40+ packages built in 35–40 minutes, and almost all this time went to re-downloading the same dependencies from external sources.

The cache was formally enabled, but the key was built by branch number—it changed on every commit, and the storage did not hit the target.

Worse, during network failures on the registry side, the entire pipeline failed: 5–7 failed builds per week, each one a manual restart and lost team hours.

What was changed

We rebuilt the key scheme: the base layer with system packages and tools of a fixed branch is updated on a schedule, the dependency layer is tied to the lock file, and the artifact layer is tied to the content of the packages themselves.

Thus, a code change in one module does not invalidate the cache of neighboring ones. Alongside, we raised a local registry mirror with warm-up for the most frequent packages—the pipeline stopped depending on external network availability.

On GitLab CI, this also gave parallel warm-up: the graph branch affected by the edit is rebuilt, the rest is pulled from storage.

Metrics before and after

We separately checked invalidation: when the lock file is updated, only the affected part of the graph is rebuilt, and rollback to the previous dependency set remains possible.

Changes in the cache configuration go through review like ordinary code—regressions are caught on the very first run.

How is dependency caching different from layer storage?

These are different levels of build acceleration, and they are often confused. Layer storage of images caches the result of building the image itself: each layer is a set of file system changes tied to the instruction that created it.

Dependency cache lives separately from the image and stores package manager artifacts downloaded or built during pipeline execution.

Their invalidation is different. Layer invalidation goes in a chain: a change to any instruction higher in the description resets all subsequent layers, so the dependency manifest is copied before the source code—then code edits stop rebuilding package installation.

Dependency cache is invalidated by key: the hash of the manifest with pinned versions. On a miss, the nearest previous key is taken and only the missing part is downloaded.

This is the point of separating levels: the image is responsible for the runtime environment—system libraries, runtime, built application. Dependency cache accelerates steps that do not get into the image at all: unit tests, style checks, static analysis, artifact builds. If you cache only images, these steps pull the same packages again on every run.

The effect is visible in the contrast between hot and cold runs: a cold one is measured in minutes, a hot one in tens of seconds, and the bottleneck shifts from dependency installation to the test run itself.

In practice, the cache is kept shared for the branch so that parallel builds do not overwrite each other's records.

This is verified by step logs: a cache hit and the reason for a miss are printed explicitly. If a miss repeats on every run, a changing value such as the build number has crept into the key, or the dependency layer is placed after source code copying.

Storage limitations and typical configuration mistakes

Dependency cache behaves predictably exactly until what lies outside it changes: library version, environment composition, or package source. Below are four situations that most often surface already in the working process.

Let's break them down one by one, with the failure sign and configuration method.

Two environments overwrite each other's cache

This is a key conflict: the key is built from the branch name, and identical names in different repositories or environments land in the same record. We make the key composite—repository identifier, environment, build platform.

The impact of this mistake is small: on a miss, dependencies are rebuilt, and someone else's artifact will not get into the build.

What to do with a closed package registry?

The internal registry does not expose packages externally and requires authorization on every request. We put the cache on the build agent side, inside the perimeter: only verified artifacts go outside.

We store credentials in secrets, not in the cache configuration, otherwise they leak along with the dump.

Storage grows indefinitely—how to limit it?

Without a cleanup policy, the cache grows faster than code: every build adds layers, and no one deletes old ones. The retention rule is the last few records per key plus deletion of everything not requested during a set period. Volume reaches a plateau, and the miss rate remains within acceptable limits.

Let's discuss the task: what to clarify before starting

To come back with an estimate and work plan, rather than general promises, we need inputs about your pipeline. Below is what we clarify in the first conversation: this set is enough to calculate the scope, stage order, and risks.

• How the pipeline is structured: one build per repository or a matrix of targets, which stages are mandatory, what is already cached.
• Package managers and version pinning files: how many there are and how often they change—the cache key and behavior during dependency updates depend on this.
• Where the cache is stored: local disk of the build agent, shared network volume, or object storage—affects speed and cleanup of outdated data.
• Agent topology: own machines or cloud, how many concurrent builds, whether there are network restrictions to external registries.
• Structure of repositories and shared libraries: keep the cache shared or split by project.
• Access and perimeter: who owns agent settings, closed network, whether proxy is allowed.
• Target build time: how long a typical build takes now and what the time goal is—from this we calculate the effect.

With these inputs, we come back with a timeline estimate, stage order, and a list of changes to agent configuration. Describe the task in text or with a link to the current configuration—we will propose a work plan.