Base image policy: what breaks without it
Until there is a single rule, each team takes a base image from wherever is convenient and updates it whenever it remembers.
At night a patch version changes in the parent image — in the morning one of the services comes up with a new library and fails on an incompatible call, and the cause is not visible: same code, same branch, different parent.
There is no base image policy, and the build begins to depend on what someone published to the shared registry.
Then environments start to diverge. The staging environment is built on the previous parent, production on the current one, while a developer locally has an image from cache entirely, and the bug reproduces on one of three environments.
At the same time outdated dependencies accumulate: updating the parent is scary because no one checked what will come with it — the version lives in production until the first incident.
Shadow builds close this pain in their own way: some images are built bypassing the pipeline, manually, from local machines, and end up in the registry without scanning and without metadata about composition.
This yields a non-reproducible build — different artifacts come out of the same commit, and rolling back to the previous image ceases to be a rollback.
The cost shows up at the most inconvenient moment — during a planned parent update or an urgent vulnerability fix. First you have to figure out what is even built and on what, and only then change something.
We break down how this is closed: a single parent image, fixed tags, an update window, and staging validation before production.
Measurable effect: build time and environment stability
A base image policy is measured not by the number of documents, but by four values: release lead time, share of successful updates, number of manual configuration edits per release, and reproducibility of environments.
We capture these metrics before implementation — from build history in the registry and deployment logs — otherwise there is nothing to confirm the effect with.
The main shift is in release lead time. When all services are built from one fixed base image, and security updates arrive centrally, the team does not need to coordinate library versions on every release and rebuild layers manually.
Release preparation is reduced severalfold, and the build itself fits into predictable time instead of “depending on cache luck.”
The share of successful updates grows by eliminating a whole class of failures — environment divergence. Test, pre-production, and production environments stop differing in their set of system packages, so configuration divergence disappears as an incident category.
What remains are errors in code and logic — things that truly require an engineer rather than a rebuild.
Manual configuration edits leave the process: a change to the base layer is made in one repository and goes through the same checks as code.
When setting up the reference image, we rely on CIS Benchmarks — security requirements are checked automatically, not by the eyes of the on-call engineer.
Predictability of environments becomes measurable: the same image digest produces the same behavior on any node, and rollback comes down to switching to the previous tag.
Update modes and image version control
Base image policy answers one question: who and when decides that the foundation of a service has changed.
A failure does not manifest at the moment of change — library incompatibility or an unpatched vulnerability surfaces several releases later, when the cause is already lost in build history.
The key fork is hard version pinning versus floating labels. Pinning by immutable hash gives repeatability: same foundation, same result. A floating label brings fresh fixes without code changes, but makes the update an uncontrolled event.
| Variant | When it fits | Limitations |
|---|---|---|
| Hard version pinning by hash | Regulated services, reproducible builds, audit | Patches do not arrive on their own — an update procedure is needed |
| Floating labels | Internal services, test environments | The build breaks at a random moment, rollback is non-obvious |
| Full build recreation | Planned update, preparation for a major release | Long check run, a parallel environment is needed |
| Overlay layers | Custom patches on top of an existing foundation | Number of layers grows, provenance is harder to trace |
Next is version compatibility. We maintain a “foundation version × service dependency” matrix and run the build against it: if a library requires an older branch, a specific service stays on it, while the others move forward.
We build overlay layers only from the internal repository — updating the foundation does not pull in unverified code, and build recreation is triggered on a schedule, not after a failure.
Implementation stages: from registry audit to policy
The order of steps here is not a formality: rules written before inventory break the builds of teams whose dependencies you have not yet seen. Therefore we go from facts to policy — each stage provides input for the next.
- Registry audit. We export images and tags, mark what runs in production and what is pulled as a parent layer. The output is a list with build dates and service owners.
- Dependency schema. We build a graph: service → base image → shared layers. It shows how many teams the image change will affect and which builds to migrate first.
- Configuring build rules. We fix allowed base images and their source, tag policy, and signature verification. Builds from an untrusted source are blocked automatically, not by persuasion in chat.
- Staging validation. We run one or two services on the new rules: system libraries, certificates, access rights. A conflict is cheaper to catch here than at release.
- Rollback and exceptions. We make sure previous images are available and rollback takes minutes, and that an old service has a temporary exception with a review date.
- Policy handover. We hand over the document and train the team: how to add a new base image, who approves it, what to do when a build is rejected.
After that, the rule lives in the build process, not in an instruction: a violation is visible immediately, not during an incident review.
Update policies, schemas, and dependency map
A policy is not a document for the sake of a document, but a set of artifacts your team uses without us.
Each item is tied to an action: who has the right to change the base image, in what order a build enters the working environment, what to do at night if a critical vulnerability is found in the base layer. Below is the kit that remains with you after implementing a base image policy.
- Update policy — rebuild frequency, tag lifetime, and the person responsible for release.
- Dependency map — which service is built on which base image, who the owner is, and which layers it inherits; updated with every build.
- Access policies — who publishes an image to the registry, who only reads, who approves exceptions; rights are separated by role.
- Exception procedure — a temporary deviation form: justification, term, owner, review date; none are lost and all go into the report.
- Environment schema — which tags are allowed in the test environment, which in the production one, how a build is promoted between them.
- On-call shift instructions — step-by-step procedure for a critical vulnerability: scope assessment, rebuild, rollback.
- Reporting — share of services on the current image, open exceptions, time from fix release to deployment.
Case: how environment divergence was eliminated and releases accelerated
Before implementing the base image policy, nine teams each built the base layer their own way: dependencies were pulled in at build time, and the same service on six environments received different system libraries.
Environment divergence cost more than it appeared in reports. About every fourth rollout failed at startup due to mismatched library versions, and root-cause analysis took longer than the build itself.
The full build cycle took about 11 minutes, six of which went to installing dependencies from external sources.
We fixed one parent image with a described package composition and consolidated build and artifact storage in Harbor.
Child images are built only from the verified parent, updates happen by scheduled rebuild and when vulnerabilities appear in base packages, rather than by edits in each repository.
After the transition, environments stopped diverging — the share of differing packages fell from a third to zero. The build shrank to three minutes thanks to the shared parent and layer caching, and image size decreased threefold after removing unnecessary utilities.
The failure class “it worked in staging, not in production” disappeared, while rollback to the previous tag remained available: old images are stored in the registry and signed.
How does hard pinning differ from advisory mode?
The difference manifests when a check fails. A hard rule blocks the build: if the base image is not from a trusted registry or has not passed vulnerability scanning, the artifact is not released.
Advisory mode records the same violation but lets the build through: the deviation goes into the report.
Hard rules pay off where the build stream is large and there are many teams. In soft mode, warnings accumulate, and old base layers live for months — one day a vulnerability is found in twenty services at once. Blocking aligns everyone: updating the base layer stops being the good will of an individual team.
The downside is emergency fixes. When a critical vulnerability must be closed now, but the needed image has not yet passed the full check cycle, a hard policy gets in the way: the patch is ready, but it cannot be released.
Engineers go around it — manually edit the policy or pull an unaccounted branch — and build controllability drops as a result.
The compromise between modes looks like two routes within one policy. The general stream follows hard rules, while a separate path remains for emergency fixes: advisory mode with mandatory exception registration — who released it, for how long, and with what justification. The exception lives for a limited time and then automatically returns the build to hard mode.
The policy is versioned together with code, every build knows which revision it passed under, and the number of active exceptions is a measurable value.
A growing exception list itself suggests where rules are too strict and where base images are updated less often than teams need.
Technical concerns: scanning, signatures, compatibility
Provenance is verified twice: the base layer is fixed by digest, not by a floating tag, and on output the image receives a signature and attestation.
Before rollout, cluster admission control compares the signature with the trust key — a substituted image simply does not reach deployment. Build integrity stops depending on discipline, and ready-made signing tools such as Sigstore cover this part without custom scripts.
We move to the policy in stages, otherwise the first release runs into someone else's build. First it operates in audit mode, then it blocks production namespaces, and only after that — all the rest.
A build that still pulls a legacy image receives a temporary exception with an owner and review date.
Will compatibility with current builds break?
Only what pulls unmaintained images breaks. For non-standard foundations — custom runtimes, native modules — we build a minimal layer and add it to the allowed list. The application builds as before; only the parent layer changes.
Will vulnerability scanning slow down the pipeline?
We scan the delta and cache the report by digest: the same layer is not checked again. Critical levels are checked on every commit, and a full report — on a schedule.
How do we roll back if the new base breaks a release?
The old image remains in the registry: the retention rule keeps the latest builds per tag, so returning is a digest change in the manifest, not a rebuild. An exception to the policy is described as code and rolled back together with it.
Let's discuss the task: registry audit and implementation plan
A registry audit begins not with a report, but with access to facts: without inventorying images and understanding who builds them and how, any base image policy will remain a declaration.
Therefore the starting order is this — you send the inputs, we review the registry and build pipelines, then return with an implementation plan and an estimate of work by stage.
For an estimate, it is enough to attach:
- list of base images and tags in the registry with service owners indicated;
- several examples of build files from different teams — from long-standing services to recent ones;
- build pipeline settings and image vulnerability scanning rules;
- registry access schema: who publishes, who only reads, where keys are stored;
- security and environment isolation requirements that the policy must cover;
- list of services that cannot be stopped, and a convenient window for changes.
Then we compare facts with practice: we look at duplicate base images, tag drift, layer lifetime, and how many builds bypass checks.
The output is a staged implementation plan: from freezing the list of approved base images to automatic verification in the pipeline and rollback to the previous image if a rule blocks a release.
Let's discuss the task — describe it in free form, and we will return with an estimate of work, timelines, and starting order.