Container image vulnerabilities: where Trivy CI/CD breaks
An image with known vulnerabilities reaches production more often not because there is no scanning, but because it is in the wrong step. A report is generated, stored as an artifact, someone reviews it — and the container is already in the registry and already deployed.
Then the expensive part begins: emergency re-release, service rollback, unscheduled change approval during working hours.
The cost multiplies by the number of services. A vulnerability in a base layer is not one release, but all images built from the same base, including those no one has touched for months.
A separate trap is base image substitution: the tag points to a rebuilt parent, the contents have changed, and previous checks did not notice because they compared the name, not the immutable reference.
Typical failure causes look mundane: the check step comes after publication to the registry; a non-zero exit code is suppressed so the build “does not go red” before release; layers pulled from cache are not re-analyzed.
Then build-time scanning formally exists, but no one makes the decision to stop, and vulnerable images pass review by inertia.
All this is not about the absence of a tool in the pipeline, but about its place in the process and about what to consider blocking.
We break down how this is closed: where to put the check, which thresholds stop release, and how to keep container security at a level that is not bypassed in a hurry.
What image scanning in Trivy CI/CD gives
Vulnerability control on every commit
The image is checked for layers and application dependencies immediately after build, not once a quarter by procedure. The report is tied to a specific tag, so you can see which library produced the finding and whose change brought it.
Critical vulnerabilities stop accumulating for months — they are closed when they appear, while the fix costs one line in the dependency manifest.
Build grows by seconds, not minutes
The vulnerability database is pulled once and reused between jobs, the changed layer is checked, and the result returns to the same build pipeline report. The engineer does not switch to a separate interface or wait for a manual check at the end of the week.
Rollout blocked by threshold
The build pipeline policy stops publication on critical findings and marks significant ones as a warning for review. An image with a known hole physically does not reach operation, and the failure class “a vulnerable build was rolled out” leaves the process.
Registry stops growing
The Trivy scanner cuts off problematic images before they are sent to the registry: failed builds are not published at all, and a lifecycle rule keeps the latest builds per branch and cleans up the rest.
The registry grows by the number of releases, not by the number of build attempts — storage volume is reduced severalfold, while rollback to the previous release remains possible.
Scan modes: from manual run to build blocking
A scan mode is not about whether Trivy is enabled in the pipeline, but about where exactly the decision is made on whether an image can move further.
The severity threshold is set once and determines which vulnerability levels count as a build failure: without it, a report appears but there is no reaction to it.
Below are four scanning modes — from the softest to the strictest. In practice, they are not chosen forever but combined: fast feedback to the developer at build time plus insurance at entry to the production environment.
| Mode | When it fits | Limitations |
|---|---|---|
| Manual run | Getting acquainted with images, dealing with accumulated debt | Regularity depends on a person, the result does not remain in the build history |
| Build check without blocking | The team is just introducing the practice, many hits on base layers | The report is not always read, the vulnerability goes to the registry along with the image |
| Build blocking by severity threshold | Published images, it is important not to ship a known vulnerability | An exception policy is needed, otherwise builds stop on third-party dependencies |
| Registry check on schedule | Images sit for a long time and are not rebuilt | Finds the problem after publication: it is unclear what to do with the released tag |
| Environment admission control | Critical environments where running an unchecked image is unacceptable | Failure at startup is the most expensive delay, a clear rollback path is needed |
The choice of mode is the choice of how early the team is ready to learn about a problem and how much manual work it is ready to spend on it.
Implementation order: from image audit to policy
Image scanning rarely breaks in the scanner itself — usually everything rests on the order of work: the check is enabled on all builds at once, hundreds of hits appear in the first report, and a week later it is disabled because the release stopped.
We take the opposite path: from inventorying what is already being built and run, to rules the team is ready to maintain for a long time.
- Image audit. We compile a list of base images, registries, and build jobs, look at what is pulled from external sources and where there is no checksum pinning — we get a map of what will need to be checked at all.
- Admission scheme. We agree on what to consider blocking: critical vulnerabilities with an available fix stop the build, the rest goes into the report and the review queue. Without this rule, implementation slides into an argument about every line.
- Scanning setup. We embed Trivy into the build and into the check before sending to the registry, separate strictness levels for development and release branches, and cache the vulnerability database so the check does not pull it anew.
- Testing on real images. We run the rules on current artifacts, not training ones: we look at how many hits actually block the build and calibrate thresholds until the added time becomes unnoticeable.
- Exceptions and their lifetime. We formalize justified exceptions with an owner and review date — so a temporary waiver does not turn into a permanent hole in the admission scheme.
- Policy and handover. We fix the procedure for responding to a hit, rollback points, and the procedure for updating rules, hold a review with the team, and hand over documentation along with the settings.
Handover package: policies, configurations, and documentation
After delivery, the repository contains all the tooling around Trivy: not only runnable checks in CI/CD, but also configuration, policies, and procedures.
All of it is versioned alongside the code and changed through normal change review, rather than through correspondence with the contractor.
- Scan configuration — a scan settings file with rule sets, severity levels, and an exceptions list; each exception is tagged with an owner and review date.
- Image admission policies — which vulnerabilities block the build, which are only logged, and for which branches the strict mode applies.
- Scan artifacts — a machine-readable report and image composition in CycloneDX or SPDX formats, tied to the build tag.
- Response policy — a table “severity level → response time → owner,” and the procedure for confirming and removing exceptions.
- On-call instructions — how to read the report, what to do about a critical finding in a published image, how to roll back to the previous tag.
- Integration scheme — at which pipeline stages the scan runs, where the results go, and how they relate to the build log.
- Decision log — review of disputed findings with the team and an extension point: where to add new rules and how to test them.
Case study: how a product team closed the flow of vulnerable images
Before implementation, the team checked images manually once a quarter: the report arrived in a ticket when the image was already in production, and the tag had time to be overwritten. No reproducible build remained, and there was nothing to use to find the source of the finding.
A third of the images in the registry carried vulnerabilities with a ready fix, and nine days passed from detection to rebuild.
What changed
Scanning with Trivy was placed as a separate step before publication to the registry — on the same build that goes to production. The build fails on critical and high findings for which a fixed dependency version exists; the rest goes into the report for review.
The list of justified exceptions is in the repository next to the code — each with a reason and owner, so the rule is not muted entirely. We cache the vulnerability database on the build node: the step adds seconds, not minutes, to the build.
What became
The share of images with critical vulnerabilities fell from a third to a few percent — only what passed the check reaches production. From detection to rebuild now takes a few hours instead of nine days: the dependency fix and the new tag go in one pass.
A whole class of “vulnerability in the base layer” incidents is gone — previously they were handled after the fact, now an image with such a layer is simply not published. The release engineer freed up about six hours a week that went into manual list reconciliation.
How does registry scanning differ from build-time scanning?
Build-time scanning answers the question “can this image be released at all.” A scanner like Trivy runs before sending to the registry, the result becomes part of the build report, and the severity threshold triggers: the build fails, and the artifact does not go further.
This is the only mode that prevents something — but it sees only the images that passed through the pipeline, and only at build time.
Registry scanning answers another question: what is already stored and has it become dangerous.
Image composition is immutable, while the vulnerability database is constantly updated — an image clean at build time gets critical findings a few weeks later because they were disclosed later.
Only regular registry scanning can catch this, but it does not block anything: it is a signal, not a barrier.
The difference in approaches is also visible in coverage. At build time, only what the pipeline builds comes under control; images brought from outside, built manually, or taken from an external supplier remain outside the check.
The registry closes this gap — it scans by manifest everything pushed, including old versions that no one rebuilds. There is also a limitation: without access to layers or with an incomplete manifest, some findings are lost.
The practical conclusion is not “either,” but “both, with different roles.” At build time, we keep a blocking threshold for critical vulnerabilities and an explicit list of exceptions so release does not break due to false positives.
In the registry — regular review with a report and a clear rule: which image to rebuild or revoke and within what time.
Technical concerns from engineers: addressing objections
Before a pilot, engineers ask the same questions: will the check turn into a stream of false positives, will it slow down the build, what to do in an air-gapped environment. We address each point as it is resolved in real builds.
We place the Trivy scanner at the image build stage, before publication to the registry. We set the blocking threshold by severity level — the build fails only on what actually requires a fix.
Will false positives block the build?
The threshold cuts off most of the noise: we check only fixable package vulnerabilities, findings without an available update are not raised to blocking. Disputed cases go into the exceptions file — each line is tied to a component and has an owner with a review date.
How are exceptions formalized in the rules?
We do not disable the check entirely. An exception is a record with the package, vulnerability code, and reason, plus a term after which it stops applying.
Such changes go through review along with the code, and the report shows how many exceptions are active and which are overdue.
Does this work in an air-gapped environment?
Yes. Vulnerability database updates are loaded into an internal mirror and served to the scanner without external network access. The database cache is reused between jobs, so the check does not slow down parallel builds and does not create extra load on the registry.
Where is the boundary of responsibility?
Our part is the build, scanner setup, thresholds, registry integration, and reports. Yours is prioritizing findings and updating base images: the scanner shows what to fix, the service owner decides on the patch. We fix the list at the start so disputed findings do not hang.
Let's discuss the task and come back with an estimate and work plan
To start, we need access to the repository and image registry — read access is enough to look at the current pipeline, Dockerfiles, and how artifacts are built and published.
Plus a short list of services where the cost of a missed vulnerability is highest: usually these are things that face outward or work with payment and personal data.
If there is a window in which it is acceptable to change the build, tell us about it right away — timelines depend on it.
Then we come back not with a generic “we need to implement Trivy,” but with a plan where every item can be checked and accepted:
- implementation plan by stages: first scanning in report mode, then blocking checks at build time and in the registry;
- policies by severity level — what stops the build, what goes into the report, and what is closed by deadline;
- integration scheme into the pipeline: at which step the check runs, which artifacts are scanned, where the result goes;
- procedure for handling false positives and a list of justified exceptions, so checks do not turn into noise;
- procedure for updating vulnerability databases and reconciling with the dependency versions you already use;
- instructions for developers: how to read the report, what to do on a hit, how to approve an exception;
- timelines for each stage and acceptance points where we record the result.
Describe the task and the current state of the build — we will come back with an estimate of the scope of work and a plan for the order in which to implement checks without stopping releases.