Why Docker image signing fails at the deployment stage
A classic failure looks like this: the build completes, the image goes to the registry under a floating tag, and an hour later a completely different set of layers is running on a node in production.
The causes can be mundane — someone rebuilt locally and overwrote the tag, the build job failed midway and deployed the previous result, the registry mirror served a cached copy. A tag is a mutable label, and it says nothing about what was actually loaded.
It is nearly impossible to distinguish a substitution from a normal release by external signs: the same tag, the same port, the application starts and passes basic probes.
An unsigned image in production lives until the first incident — it is noticed after a leak, unexpected network traffic, or lost data, not at the deployment stage.
The investigation then takes days, because there is no way to link a running container to a specific set of source code and build run number after the fact.
Image provenance verification is needed precisely so that this question is resolved in seconds.
A cryptographic signature records that the artifact was built by a specific build job from a specific source code state, and the verifying mechanism in the cluster simply does not allow anything whose signature does not match to run.
Then artifact substitution between build and deployment hits a rejection at the entrance, not an investigation of consequences.
We'll break down how this is covered: Docker image signing, Keyless via Sigstore, and Kyverno policies that only allow verified artifacts into the cluster.
What a verifiable supply chain delivers: effect in measurable terms
Image signing turns the question "where did this artifact come from" into a machine check. Until an image is signed by the build process key, deployment is rejected at the cluster entrance — before the container gains access to the network and secrets.
This is artifact integrity control: what is compared is not the tag name but the specific build snapshot.
Release preparation stops waiting for manual confirmation: the build pipeline signs and publishes the artifact itself, and the engineer intervenes only where a decision is required.
The share of manual operations in deployment drops to a few steps, deployment time is reduced from hours to minutes — on every release, not once.
During incident investigation, the verification log answers which build and which signer produced the artifact in a specific environment.
Localization takes minutes instead of manually sorting through versions, and the downtime window is limited to one rollback to the previous signed build.
A separate consequence is an eliminated class of failures. Tag substitution, an image from a local build, rollback to an unsigned artifact no longer reach the production environment: they are rejected by policy at the cluster entrance.
A verifiable supply chain stops being an audit report and becomes a standard mechanism that saves hours on every release and every investigation.
Artifact signing: key-based, keyless, and hybrid modes
The modes differ not by algorithm but by how the verifying side links the signature to an identity. Key-based signing holds a long-lived private key, keyless signing issues a short-lived certificate for pipeline identity, and a hybrid identity scheme combines both trust models.
| Mode | When it fits | Limitations |
|---|---|---|
| Key-based | Closed environments without outbound access: the signature is made from a secure store, and the cluster verifies the image by public key | Key rotation and protection are required: its compromise means re-signing the entire registry, and the signature does not say who exactly made it or from which branch |
| Keyless | Build in a shared pipeline: an external certificate authority issues a certificate for minutes, and no private material remains on disk | Requires outbound access to the identity service and transparency log; in an isolated environment without proxying, the mode does not work at all |
| Hybrid | Policy requires both offline verification and binding to a specific pipeline, branch, environment | More expensive to set up: you have to synchronize the trust chain and maintain two verification sources at once |
On the cluster side, the mode defines the admission rule: verification by public key or by identity and certificate issuer. A typical trap is a rule that looks only at the fact of a signature: then an image signed by any identity passes, and the point of verification is lost.
That is why signing modes are compared before implementation, and the verification scopes are fixed explicitly.
Step-by-step implementation: from pipeline audit to operations handover
We structure the work order so that image signing does not stop current deployments: first we capture the actual state, then we introduce rules, and only then do we enable blocking.
-
Build pipeline audit. Inventory: where and how images are built, which registries are involved, who deploys, what admission policies already exist. The result is a dependency map and insertion points for signing.
-
Trust scheme. We decide what attests the builder's identity: a keyless mode with identity from the build system or a key in a secret store. We fix the list of trusted identities.
-
Signing setup. Cosign signs the image after publication, and the record goes to the transparency log. We check on one service that tags and digests do not diverge.
-
Artifact verification setup. An admission rule in the cluster compares the signature and identity before container startup — first in warning mode, to see what will break.
-
Staging run. Full cycle: build, sign, verify, rollback to the previous image, behavior when the log is unavailable. We measure deployment latency and the false rejection rate.
-
Acceptance and operations handover. We switch verification to blocking mode and hand over the regulation: how to sign manually, how to add a trusted identity, what to do during an incident.
Handover package: admission policies, diagrams, and regulations
A working verification is half the result. The other half is a package by which another engineer can maintain, change, and if necessary cancel this verification without the author. Below is what exactly remains with your team after handover.
- Admission policies. Which images run in the cluster, whether signing is mandatory, what to do with unsigned builds, which exceptions are open and until what date they apply.
- Role separation. Who is allowed to sign an artifact, who can change policy, who can only read the log. The right to sign and the right to admit are separated: the builder cannot single-handedly allow their own deployment.
- Build and deployment flow diagram. One page: where the signature is applied, where verification takes place, what happens on rejection, and where the notification goes. With it, a new engineer enters the environment without repeated questions.
- Incident response regulation. Compromise of a key or signing account: how to revoke, how to re-check what has already been deployed, and within what time the on-call engineer must complete it.
- Signing documentation. How a short-lived certificate is issued, which build identity it is bound to, and how to reproduce verification manually on your own machine outside the environment.
- On-call instructions. Analysis of typical rejections — image not signed, signature expired, identity service unavailable — with commands and the boundary "do it myself / escalate".
- Verification log. What is stored and for how long, how to export a report for internal audit, and how to prove that no unsigned artifact ever reached production.
The package is assembled for your processes and handed over together with a walkthrough: we show how to change policy without breaking deployment.
A policy without a regulation turns into a setting people are afraid to touch — that is why documentation is part of the delivery, not left in a chat thread.
Case study: eliminating artifact substitution between build and deployment
At the customer's site, build and deployment lived in one pipeline, but the release right remained with an engineer, and images were addressed by tags.
One day an image from the wrong branch went to production: the tag had been overwritten between build and deployment, and there was no way to verify the link between the release and the source commit. The incident took more than three hours to localize — the connection was searched for in logs.
We put Docker image signing into the same pipeline: deployment goes by digest, not by tag, and each build receives a short-lived certificate in keyless mode Sigstore — no long-lived key is stored.
At the cluster entrance, a Kyverno policy rejects an image if the signature is invalid or was issued by the wrong pipeline.
Metrics before and after
- Rejected deployment: previously there were no checks at all — now roughly every twentieth attempt is filtered out at the entrance, and these are substitutions, not working releases.
- Incident localization time: from three hours to fifteen minutes — the signature shows the pipeline, branch, and commit.
- Manual operations per release: from five to zero; rollback follows the previously signed digest, not the "previous tag".
Caveats for applying the result
The scheme works where the build is centralized and has its own account. If some services are built by contractor teams, we start in warning mode: the policy writes an event to the log but does not block deployment — and we enable blocking after all branches are signed.
For private registries, we configure trust in the certificate authority separately; otherwise legitimate images will not pass verification.
How does deployment-time verification differ from registry verification?
Verification at deployment and control in the image registry are two different control points, and they cover different classes of failures.
Docker image signing records who released what exactly, but the signature itself can be verified in two places: the registry filters out unsigned artifacts at the entrance, and the runtime confirms authenticity at startup.
A registry policy triggers at write time: if an image does not have a valid Sigstore signature or does not meet base layer requirements, it does not enter the store.
The developer learns about this immediately, not after deployment, and artifacts with unclear origin do not accumulate in the registry. But there is only one control point here — write only.
The admission controller in the environment catches what comes outside the usual channel: images from another registry, locally loaded archives, tags overwritten after publication.
It rejects startup if a signature formally exists but was issued by someone who should not be signing: with keyless signing, a specific account and its issuer are checked, not the mere presence of a certificate.
Without this clarification, verification works but lets everything through — any valid certificate opens access to the environment.
And the admission controller has its own limitation: it checks only creation; it will not stop an already running container, so registry policy remains mandatory. Both control points are needed — one cleans the entrance, the other guarantees the fact in the environment.
Engineers' technical doubts: keys, revocation, trusted sources
Signing helps exactly as much as the answers to uncomfortable questions are predictable: what to do if a key is compromised, what about images deployed before implementation, who changes the admission policy and how.
Below is what we most often discuss during alignment, because without these answers verification in the cluster remains a formality.
We work with Cosign: the signature lies next to the image in the registry, and verification is performed at admission time. Then rotation, revocation, and trusted sources are solved infrastructurally, not through chat messages.
What happens if the signing key is compromised?
The key is revoked, its fingerprint goes into the untrusted list, and all images signed by it stop passing admission verification — including those already in the registry.
Artifacts do not need to be rebuilt: re-signing takes minutes because the signature does not change the image contents. Revocation of the signature takes effect immediately on the next verification, not "sometime during a rebuild."
Already deployed unsigned images — should they be deleted?
No. We enable the policy in observation mode: it only records violations without blocking anything. Then the restriction applies to new deployments, and running containers are not shut down — they are replaced with signed versions as scheduled updates occur.
How is a trusted source determined?
The admission rule binds the image to a specific registry and build identity issued by a trusted authority. Then for new builds, the signature need not be stored as a long-lived key: it cannot be forged, and there is nothing to leak.
Who owns the admission policies?
Policies are part of the repository, not a setting in a personal dashboard: any change goes through review and enters history. Each rule has an owner from the team, so tightening requirements does not turn into an unexpected release block.
Let's discuss the task: we'll review your pipeline and come back with a plan
Reviewing the deployment pipeline starts with facts, not general questions: from a snapshot of your infrastructure, it is clear where image signing will fit into the current process and where it will require changes in permissions and release order.
Send what you already have — that is enough to come back with an estimate of the scope of work, not with guesses.
- deployment flow from repository change to production: which steps are automated, where manual confirmations remain;
- list of environments and differences between them in access and policies;
- current build and publication permissions: who releases images and who has write access to the registry;
- registry configuration: storage rules, tags, lifetime of old versions;
- list of pre-release checks: scanning, tests, approvals, and who confirms them;
- environment constraints: closed network, key storage requirements, separation of duties.
In response, you receive an implementation plan: the sequence of stages, scope of work for each, rollback points, permission changes, and criteria by which you will verify that signing actually works. Describe the task and attach a snapshot — we will come back with an estimate and plan.