Why Docker layer caching stops working: where layers get lost
Docker layer caching relies on simple mechanics: each instruction in the image description creates a layer, and on the next build the engine reuses it if neither the instruction itself, nor the contents of copied files, nor the parent layer has changed.
The dependency is chained: one changed step invalidates all subsequent ones, and the build proceeds from scratch.
The most common cause is instruction order. If the entire project is copied first and dependencies are installed afterward, any edit in the source code changes the copy layer, layer invalidation cascades down to package installation, and it runs again on every commit.
The reverse order — first the dependency list, then their installation, and only then the source code — keeps the layer cache intact during ordinary changes.
The second source is mutable labels and the build context. A timestamp or build number inserted into an instruction produces a new fingerprint on every run.
The same applies to a missing ignore list: build directories, logs, and local settings get into the context, and its checksum changes even without code edits.
The cost is slow image builds. Where tens of seconds were enough, it now takes several minutes, and they multiply by the number of builds per day and the number of engineers.
Checks in shared branches stop keeping up with commits, feedback arrives with a delay, and resetting the build cache turns into a manual operation that everyone does differently.
What layer reuse provides: build time and hit rate
Build time and failure rate
Docker image layers are cached by instruction content: as long as the dependency file has not changed, the package installation step is taken ready-made.
On a typical project, the build shortens from 8–12 minutes to 2–3, and the cache hit rate stays at 80–90% during edits in application code.
The build failure rate drops separately: a network failure while downloading packages no longer brings the whole run down, and a retry does not start from scratch.
Storage volume and deployment predictability
Layers are reused between builds, so only what changed gets into the registry — the image storage volume grows in fractions, not multiplied by the number of deployments.
Cleanup rules based on the number of versions per tag prevent old builds from accumulating, while rollback to the previous release remains possible.
Deployment predictability rests on reproducibility: the same layer sequence produces the same image, and what was tested in the test environment arrives in production with the same composition.
The effect is visible not in a single number but in the combination: the higher the hit rate, the shorter the edit cycle and the fewer reasons to restart a deployment.
We measure this on every project — image build time, hit rate, number of reruns — so that acceleration is manageable rather than "it seems to have gotten faster."
Caching methods: local cache, image registry, shared volume
Docker layer cache is not a checkbox in settings but a dependency on the build topology. A local build cache lives on a specific machine and provides maximum benefit on a permanent node, but it resets as soon as the node is recreated.
When builds run on multiple machines, the question is no longer speed but where layers are stored and who shares them.
Single-team mode
On a single node and with a stable dependency set, a local cache is enough: it is the fastest and requires no infrastructure.
As soon as parallel branches and multiple executors appear, we hit caching limitations — layers are not transferred between machines, and changing the base image rebuilds everything further down the chain.
| Option | When it fits | Limitations |
|---|---|---|
| Local build cache | one permanent node, frequent rebuilds of one project | does not survive node recreation, is not shared between machines |
| Cache in an image registry | multiple nodes and branches, builds in isolated environments | requires registry access and tag cleanup; private layers must not leak outside |
| Shared cache volume | a fleet of build nodes with shared network storage | locking during parallel writes, the network becomes a bottleneck |
When there are multiple nodes
Here, a cache between branches and machines is needed constantly; otherwise each node warms the cache again. Cache in an image registry is more convenient: a node pulls ready-made layers by tag and does not depend on who built the previous one.
A shared cache volume gives more hits but requires discipline — two simultaneous builds into one volume conflict, and some layers have to be invalidated.
If stability matters more than maximum matching, volumes are separated by node groups and a separate cache is kept for each build branch.
How we set up the Docker build cache: work stages
The work order is the same for any project: first we figure out which layers are rebuilt on each commit, then we build the caching scheme, and only then do we modify the pipeline.
This way releases do not stall during edits, and the effect is visible in numbers — in build time and the share of layers restored from cache.
Preparation and verification
- We audit the build pipeline: we look at the instruction order in the image description, what changes on every commit and what changes once a month. Frequently changing layers are moved higher, rarely changing ones are left at the bottom — those are the ones that become cacheable.
- We assemble the caching scheme: we determine which layers are shared across branches and where the cache will live — on the build node or in external storage accessible to all nodes.
- We configure cache in the registry: export and import of cache into the image, separate tags for branches and base images so that layers do not overwrite each other.
- We verify hits: using the build log, we check how many layers were restored from cache, measure time before and after, and remove unnecessary files from the build context — they break the cache prematurely.
Rollout and handover
- We roll out to working branches gradually: first the main one, then the rest, monitoring build time and cache storage growth.
- We hand over build guidelines: rules for editing the image description, what must not be changed without approval, a memo for developers, and the procedure for a full cache reset.
What remains in the repository: configurations, schemes, and guidelines
The result of the setup is measured not only by build time but also by what remains with the client's team. We hand over not a cache working on our side but files, schemes, and rules by which the build can be changed and reproduced without us.
Code and infrastructure
- Build pipeline configuration — build files in the repository: stage order, layer breakdown, extracted dependencies.
- Cache scheme — which Docker layers are cached, which key is used, what it depends on, and where it is stored: on the builder or in a shared registry.
- Invalidation guideline — what resets the cache (change of base image, dependency file) and the procedure for an emergency full reset.
- Registry access policies — read and write roles, separate credentials for builds, prohibition of anonymous writes, token lifetime.
- Observability — cache hit rate and alerting when it drops: slowdown is visible before it becomes a problem.
Documentation and access
- Build documentation — how to build the image locally, how to reproduce it in the pipeline, where to view logs, and how to distinguish a cache miss from a layer error.
- Ownership map — who is responsible for the base image, who for registry policies.
- Rollback instruction — how to restore the previous configuration if the build slowed down or became unstable.
- Acceptance checklist — checks by which the update is accepted: build time, hit rate, reproducibility on a clean builder.
All this lies next to the code and goes through review like any ordinary change — configuration does not live in chat and is not lost when an engineer changes.
Case study: monorepo build accelerated, failure rate dropped
What it was like before the work
A monorepo with forty services was built in a single Docker pipeline: a shared image registry, a shared executor, one branch for all teams.
Dependency layers were rebuilt on any change in the tree — an edit in one service triggered package reinstallation across the entire repository, and a full pass took 22–26 minutes.
The queue grew: while one service was building, the rest waited for the executor to become free. Engineers stopped waiting for the result and began checking locally, and some errors reached the main branch.
The failure rate held at 18%, and the class of failures "built on outdated dependencies" reproduced regularly.
What changed afterward
The build was split into stages and the instruction order was rewritten: dependencies are installed in a separate layer, mutable code is copied last.
The cache is mounted on the executor's shared volume and reused between jobs, and the key is tied to the checksum of the dependency file — version substitution does not allow building on outdated layers.
A full pass was reduced from 22–26 to 6–8 minutes, and rebuilding an unchanged service fits within a minute. The failure rate dropped from 18% to 4%: that same class of errors is closed by the cache design, not by a guideline.
The registry stopped growing — the lifecycle policy keeps the last ten images per tag and cleans up the rest, and rollback to the previous build remains possible.
How does layer cache differ from dependency and package cache?
There are three caching levels in a build, and they do not replace each other. Layer cache works at the instruction level: Docker calculates a layer key from the instruction text and the parent layer hash.
Dependency cache lives inside the package manager — it is its own store of downloaded libraries. A mounted volume is a directory on the build host that survives container removal and does not get into the finished image.
Layer cache breaks in a cascade: one instruction changed — everything below is rebuilt from scratch. That is why layer order matters more than any settings.
First we copy only the file with the dependency list, then install them, and only then copy the source code — a code edit stops dragging package reinstallation along with it.
Dependency cache covers a different scenario. The installation layer may be invalidated while the package store remains intact — the manager will take everything from it except new versions.
A mounted volume for the store gives the same effect but lives separately from the image and survives build cache resets and edits to the dependency file.
Hence the difference in approaches. Layer cache is responsible for reproducibility: the image is built exactly from fixed filesystem states. Dependency cache and the volume are responsible for speed and traffic to external repositories.
The volume needs a separate key for each build branch — otherwise parallel builds overwrite each other's store and produce unstable results.
Common technical doubts about build caching
Layer cache speeds up builds only when it behaves predictably. Below are answers to questions engineers ask before implementation: parallel builds, base image updates, cache transfer between nodes, and correctness verification.
Two builds run at the same time — will they mix up the cache?
The key is built from the task and branch content, not from a shared tag, so layers from one build do not get into another. We move the shared cache to separate storage with write permission per task, and the local one is populated from it.
The base image was updated — will the cache remain valid?
Invalidation goes by content, not by tag name: we pin the base image by immutable identifier, and a change of base layers itself resets dependent ones. Once a day we run a full rebuild without cache and compare artifacts — a discrepancy means the key rule is too broad.
Will the cache transfer to another node or build agent?
Yes: we put the shared cache layer in a registry or object storage, the new agent reads it read-only and warms up the local one. The first build on a cold node receives most layers ready-made instead of a full rebuild.
Is the cache secure? Will secrets remain in it?
Intermediate layers can retain what got into the image. We inject secrets at the build stage rather than copying them into a layer, restrict access to storage, and maintain a lifetime with regular cleanup — then the cache does not turn into a leak channel.
Let's discuss your task: we will come back with a plan and estimate
To come back with an estimate rather than general promises, we need to see your build pipeline as a whole: where images are built, how many steps there are between a repository edit and the finished artifact, which layers change on every change and which remain unchanged.
A link to the repository, the build file, logs of two or three typical builds, and a description of where the image storage and build agents live are enough.
With this set, we calculate what exactly will provide acceleration and what will not work in your case — and we will say so directly.
What we deliver in response:
- Caching scheme for the project: which layers to cache, where and how much to store, how to share cache between branches, agents, and parallel builds.
- Step-by-step implementation plan: the order of edits in the pipeline, checkpoints, and a way to return to the previous scheme without downtime.
- Work estimate: scope, effort, timelines by stage, and what is needed from your side — access, environment, change window.
- Invalidation and cleanup rules: so that storage does not grow and builds do not pick up outdated layers.
- Ready-made configurations and build files that can be applied in your repository without rewriting from scratch.
- Handover and team training: walkthrough of the scheme, typical failures, and what the on-call engineer should do when a build bypasses the cache.
- Short documentation: what is where, how to change the policy, and how to measure the gain in build time.
Describe the task as it is — with current build times and bottlenecks. We will come back with an implementation plan and work estimate, and we will go over the caching scheme using your real builds.