GitLab Container Registry: where CI/CD breaks
The image registry in GitLab almost never breaks "on its own" — it breaks at the seams: between building, tagging, and cleanup. The first typical failure is storage bloat.
Every run publishes a new layer, old tags remain, and after a few months the registry grows faster than the product. As long as there is enough space, no one sees the problem; when space runs out, it is no longer the builds that go down, but the entire server with the registry.
The second source is tag conflicts. If a build overwrites latest or reuses an immutable tag, then rolling back to a previous release becomes impossible: a different image already sits under the same name.
Diagnosing this is hard — the build passed, the deploy passed, but what is running on the environment is not what was expected.
A separate category is authorization errors. An expired runner token, incorrect permissions for a service account, or a scope mismatch cause a failure not at the publishing stage, but at the stage of pulling the image in another environment. The symptom looks like "the build passed, but we can't deploy."
Finally, build degradation: the more garbage there is in the registry, the longer the pull and cleanup operations take, and the more often runs hit timeouts. Let's break down how this is addressed at the level of retention policies, tagging schemes, and access rights.
What a well-structured image registry provides
A registry is not a "warehouse of images" but the point where build speed, rollback speed, and the integrity of what ships to prod converge.
When it is well-structured, the effect is visible in measurable terms: builds stop pulling base layers from external sources every time, because they reside locally and are reused across jobs, and recovery time after a failure is determined not by searching for the right version but by a single rollback command to a specific tag.
The first measurable effect is speed. Layer caching on the registry side and reuse of base images within the network eliminate re-downloading the same dependencies: a typical build step is cut by multiples, not by percentages. The second is recovery time.
When every release is tagged with an immutable tag and linked to a commit, rollback stops being archaeology: an engineer restores the previous working image in minutes rather than rebuilding it from an old branch.
The third effect is eliminated classes of failures.
Gone is "it built on my machine but failed on the staging environment" due to differing base layers, gone is manual image pushing bypassing the pipeline, gone is the situation where a tag overwritten on top of an old one shipped to prod.
Integrity is ensured by the fact that a tag points to the same digest, and traceability is ensured by the "image → commit → task → author" chain.
This is exactly the logic we build in when configuring a GitLab Container Registry: retention policies, access separation, and automatic publication from the pipeline, so that the registry reflects the actual state of the code rather than intentions.
Comparison of approaches to image storage
The choice between a built-in registry, external storage, and a hybrid scheme is determined not by convenience, but by where the build, runtime, and access policy live.
The built-in registry in GitLab offers a minimum of moving parts: authentication is inherited from the project, read and write permissions match the roles, and image cleanup is governed by the same lifecycle rules as other artifacts.
The limitation is that the registry lives on the same perimeter as GitLab itself and shares disk space and bandwidth with it.
External storage relieves the load on the main perimeter and allows the build and delivery of images to be separated across different availability zones.
The price for this is a separate authentication layer, permission synchronization, and its own mechanism for deleting outdated tags: a forgotten cleanup policy turns the storage into a dump that grows faster than releases arrive.
The hybrid scheme takes the best of both: the build goes to the built-in registry, and delivery to the runtime goes through an external mirror with pre-warming.
| Option | When it fits | Limitations |
|---|---|---|
| Built-in registry | Small team, single perimeter, short path from build to deploy | Shares resources with GitLab, limited by the scale of a single perimeter |
| External storage | Multiple clusters, distributed zones, high frequency of image delivery | Separate authentication, manual synchronization of permissions and cleanup |
| Hybrid scheme | Build in one place, runtime in several, cache pre-warming needed | More complex to set up, requires consistent tagging policies |
In practice, the choice comes down to two questions: how many perimeters consume the images and who is responsible for deleting them. If the answer is one — the built-in registry is sufficient.
If there are several perimeters and there are network boundaries between them, the hybrid scheme pays off in image delivery time during deployment.
Implementation and setup stages
We start with an audit of the current pipeline: we look at where images are built, where they are published, how access rights are structured, and how long the cycle from commit to finished artifact takes.
At this stage we identify bottlenecks — duplicate builds, missing layer caching, manual steps during release.
Based on the audit results, we agree on a scheme: which repositories and tags to create, how to separate access between teams and environments, and where secrets for registry authentication are stored.
Setup and verification
- Audit. We analyze current builds, image publishing, rights, and pipeline completion time — we get a map of bottlenecks.
- Scheme. We design the structure of repositories and tags, role-based access policy, and rules for cleaning up old images.
- Registry setup. We bring up storage, connect authentication, and configure limits and retention policies.
- Pipeline integration. We embed image building and publishing into automated release, add layer caching and security checks.
- Verification. We run a test release: build, publish, deploy, rollback to the previous tag — we make sure everything is reproducible.
- Handover. We provide the team with documentation, access credentials, and the regulations for working with the registry, and train engineers.
After verification, we record metrics: build time, share of automated steps, rollback speed. Handover includes not only access credentials but also clear regulations — how to release images, how to clean the registry, and what to do if publishing fails. Describe your task — we will get back to you with an estimate and a work plan.
What the customer receives
After the project is delivered, you are left not with "server access" but with a package that lets the infrastructure be handed over to another team or rebuilt from scratch without us.
Everything we configure is recorded in the repository and documentation — no manual actions "from memory."
- Storage and build configurations: registry settings files, deployment manifests, image storage and cleanup parameters — in version control, with change history.
- Image flow diagram: from build in a job to publication in the registry and delivery to the environment — with the points where permissions and signatures are verified.
- Access policies: roles and token scopes, separation of read and write permissions, rules for external consumers and build bots.
- Cleanup policy: how many images and tags are stored, what is deleted automatically, how the ability to roll back to a previous release is preserved.
- Operations documentation: how to release an image, how to revoke access, how to restore the registry from a backup, whom to contact in case of failure.
- Backup and recovery diagram: frequency, location of copy storage, a verified recovery scenario after a failure.
- Acceptance and handover checklist: a list of checks that confirm the pipeline and registry operate normally after a change of responsible person.
We discuss each item with your team during the handover stage — so that the document does not remain a formality but matches the actual state of the system. Describe your task — we will get back to you with an estimate and a plan.
Case study: how build time was cut by three times
The client is a product team of 14 developers, a monorepo, 6 services. Building the image took 19 minutes: on every commit all layers were rebuilt because the cache was not reused between runners.
The registry grew uncontrollably — tags accumulated without a policy, storage volume exceeded a terabyte, and half of the images did not build at all.
GitLab Container Registry was connected as the primary registry alongside the pipeline. The build was split into a base layer with dependencies and an application layer — the base is rebuilt only when the manifest changes, the rest is taken from the cache.
Distributed cache between runners was enabled so that parallel jobs would not build the same thing twice.
What changed in numbers
| Metric | Before | After |
|---|---|---|
| Image build time | 19 min | 6 min |
| Share of failed builds | 11% | 2% |
| Storage volume | 1.1 TB | 340 GB |
Failures almost disappeared thanks to two things: the build no longer depends on external downloads at startup, and the tag always points to an immutable image — rolling back to a previous release takes minutes instead of hours of investigation.
What was done with storage
A lifecycle rule keeps the last 10 images per tag and cleans up the rest — the registry stopped growing, while rolling back to a previous version remains possible.
Separately, deletion of "dangling" layers that remain after a tag is overwritten was configured: they were the ones accounting for a third of the volume.
The build was cut by three times, storage volume by more than three times, and the share of failures dropped fivefold. The team stopped waiting for the pipeline and cleaning the registry manually.
How is GitLab Container Registry different from an external registry?
The built-in registry is part of GitLab itself: images are stored alongside code, pipelines, and issues, rather than in a separate service.
For builds, this means that after docker build, the image goes to the same project where the source lives, and the next pipeline step pulls it without configuring external credentials and network routes.
An external registry requires separate integration: its own tokens, its own access rules, a separate perimeter for CI runners, and all of this needs to be synchronized with the project lifecycle in GitLab.
Access management in both cases is built on tokens, but the logic is different. In the built-in registry, permissions are inherited from project roles: a developer pushes images for their branch, a release engineer pushes tags, a guest only reads.
Access is revoked in one place — when a participant is removed from the project, they also lose access to images, which closes the typical hole where a departed employee retains a pull token for an external registry.
In an external solution, roles have to be maintained in parallel and reconciled manually, and desynchronization between the two systems is a source of incidents involving leaks of private images.
The cost of ownership is made up not of the tariff, but of the accompanying work. The built-in registry does not require a separate service, its backup, availability monitoring, or updates — these tasks are already covered by the operation of GitLab itself.
An external registry adds yet another component to the fault-tolerant perimeter: network policies between runners and the registry, separate encryption keys, its own update schedule.
Plus retention policies: a lifecycle rule cleans up old tags, but it has to be configured separately in each system.
The practical choice depends on scale. While the team and the number of projects are moderate, the built-in registry gives fewer moving parts and predictable behavior during rollback — the image of the previous release is stored right where it was built.
An external registry is justified when images are consumed by clusters outside GitLab, a single point is needed for several platforms, or geo-distributed delivery is required.
Then integration and synchronization of access become a deliberate price for the storage's independence from the build system.
Technical doubts and answers
Let's go through the questions that most often come up during the design phase and after launch: access, volume growth, migration between registries, and compatibility with current builds.
How do you separate access to images between teams and environments?
We grant rights at the project and group level: developers push to their own branches, the builder pushes to intermediate tags, and production tags are locked down by a separate role. For external consumers we use tokens with a limited lifetime and a read-only scope.
Secrets are not stored in the build configuration — they are passed through protected variables available only on protected branches.
What happens to the registry when there are thousands of images?
Volume growth is a matter of policy, not disk size. Lifecycle rules keep the last N images per tag and delete the rest on a schedule, so the registry does not grow uncontrollably while rollback to a previous release remains possible.
Additionally, we move the build cache to a separate volume so it does not compete for space with working images.
How do you migrate from another registry without downtime?
We do the migration in stages: first we set up synchronization, then we switch the builds to the new address, and then the consumers one by one, with verification at each step.
The old registry remains available read-only until we are sure that all deployments are coming from the new one. A rollback is possible at any stage until the old address is disabled.
Is it compatible with our current builds and deployment?
The registry works over the standard image protocol, so existing builds and the orchestrator connect by changing the address and credentials — there is no need to rewrite the pipelines.
We separately check scenarios with multiple architectures and image signing, if they are already in use. If your pipeline has its own security checks, they continue to work on top of the new addresses.
How do you know the registry is configured correctly?
The criterion is simple: the build publishes an image, the deployment pulls exactly that one, and old versions are deleted by the rule without affecting the current ones. Plus the access log shows who accessed the images and when. We run this check before handing it over to production, not after the first failure.
Let's discuss your task
To come back with an estimate and a plan, rather than generalities, we need the context of your task. Describe how your image release process is currently set up: where images are built, where they are published, who pulls them to the environments and how. That is enough to understand the scope of work and the risks.
From there, the work proceeds in steps:
- We analyze the current image build and publish process — where the manual operations are, where the bottlenecks are, what breaks most often.
- We clarify the requirements: how many environments, what retention policies, whether separate registries are needed for teams and projects.
- We prepare a plan for implementing GitLab Container Registry tied to your repositories and build jobs.
- We agree on the access scheme and role-based separation — who publishes, who only pulls images.
- We set up automatic cleanup of old versions so the registry does not grow uncontrollably.
- We run the build and delivery on one service as a pilot, measuring time and stability.
- We migrate the remaining services and hand over the documentation to your team.
What we need from you at the start:
- a list of services and environments to be connected;
- how images are currently stored and versioned;
- requirements for retention periods and rollback to previous versions;
- who on your team will accept the work;
- infrastructure constraints, if any.
Describe the task in any convenient form — we will come back with a timeline estimate, the scope of work, and a plan by stages.