Kyverno in Kubernetes: why policies fail to trigger
Most often policies formally exist: manifests are applied, rules described, reports collected. But some deployments slip past checks, some fail with an unclear admission webhook error, and cluster configuration drifts further between environments. The cause is almost never the rules themselves — it is how their enforcement is configured.
The first source is the admission webhook itself. If it is set to a mode where unavailability of the policy controller does not block requests, deployments at that moment pass without checks.
The same applies to namespace and label selectors: only part of the workloads falls under the rule. And pods that controllers create after the check, without background scanning, remain outside policies.
The second is the order in which mutation rules and validation policies trigger. Mutation changes the object before validation, so a check designed for fields injected by another rule behaves differently than expected.
Rejected deployments arrive as a webhook message without the rule name and specific field — such text cannot reconstruct the cause.
The third is policy management. When rules live in a repository but are applied manually, exceptions accumulate in one cluster and do not reach another: in the test environment mutation worked, in production it is absent, and validation rejects what passed in test.
Cluster configuration control falls apart quietly, without obvious errors — next we discuss how this is closed.
What changes after implementation: measurable effects
Kyverno is embedded at the moment an object is admitted to the cluster, so the effect is measured by maintenance metrics, not feelings. Below are four values that change in the first weeks after migration.
Share of changes that passed without manual correction
A requirement is described once: mandatory labels, resource limits, prohibited image sources.
The engineer does not compare configurations by eye but receives a rejection with the reason directly in the command output — the share of edits returned to the author for revision falls from dozens to a handful.
Change review time
Configuration review shrinks from hours to minutes: the reviewer looks at the meaning of the change, not compliance with the regulation. Typical deviations are filtered out automatically, before they reach the main branch, so reviews cease to be the release bottleneck.
Class of failures that disappears
A separate source of failures is environment drift: on the test environment the rule was followed, on production it was forgotten. Mutation brings the object to a uniform form on creation, and environment requirements become identical by design, not by agreement in a document.
Recovery time after incorrect configuration
If an incorrect change does pass, generation of dependent objects and an explicit rejection reason reduce troubleshooting. Downtime stays within minutes: rollback is described by the same policy as the prohibition, not by a separate scenario.
Rule enforcement modes: audit, block, mutate
The same rule in Kyverno is described once, and the mode determines what happens on non-compliance: a record in the report, denial of object creation, or fixing the manifest on the fly.
The implementation order is audit on a live cluster, then a dry run on real manifests, and only then blocking: otherwise the first strict rule halts the rollout.
Mutation and generation are a separate category: they do not reject the object but bring it to the desired form or create related objects. They are enabled after the audit shows exactly what resources lack.
| Variant | When suitable | Limitations |
|---|---|---|
| Audit mode | First run on a live cluster: collect violations without stopping rollout | Prohibits nothing — violations continue to be created |
| Dry run | Check a new rule against real manifests before enabling | Does not see decisions depending on the current cluster state |
| Blocking mode | When violations are resolved and there are no false positives | An invalid manifest fails entirely — the rollout stops |
| Resource mutation | Auto-injection of labels, limits and safe defaults | Modifies the original manifest: divergence from the repository |
| Object generation | Related objects are needed: quotas, network policies, access permissions | Requires deletion rules — objects remain after label removal |
Within a single pass, rules are applied sequentially: mutation, then validation, then generation. A blocking rule therefore validates the already corrected object.
Implementation stages: from cluster assessment to handover
The order of work is built so that no stage slows development before we see the real cluster picture. First inventory and rule design, then a soft run, false positive triage, and only then blocking.
This removes the main risk: a policy enabled immediately in strict mode stops a release rollout because of a single non-standard manifest.
- Cluster audit. Inventory objects and exceptions: privileged containers, host volumes, images without a pinned tag, containers without limits. Separately record namespaces and system workloads that must not be touched.
- Rule set design. Classify findings by violation classes — one rule per class, with clear rejection text. Agree on what goes into blocking and what remains a warning.
- Pilot on staging. Run Kyverno rules on a cluster copy, review reports and refine wording until false positives are gone.
- Soft launch. Enable the set in audit mode on the working environment: see how many objects violate it, who creates them, and whether this interferes with rollout.
- Switch to blocking. Switch one rule at a time, starting with those where reports are clean. For inherited workloads — targeted exceptions with a review deadline.
- Rule acceptance. Run bypass scenarios: image substitution, running as superuser, missing limits. Verify that the rule triggers and does not break regular releases.
- Operations handover. Hand over the repository with policies, the priority scheme, the exception procedure, and the set update procedure.
Access policies and handover documentation package
Policies that no one described in words turn in six months into a set of rules that people are afraid to touch: it is unclear what each directive checks and what will happen if it is disabled.
Therefore, along with the configured cluster, the customer receives a package where each rule has a description, and the team has an agreed workflow.
- Rule registry. For each policy, state what it checks, which resources it affects, whether it runs in blocking mode or report-only, and why.
- Roles and access. Editing policies and approving their enablement are separated between different people; rights are granted by roles, not one shared account.
- Enforcement scheme. The path of a rule from repository to cluster: where the source is, how it is tested, when it takes effect, and how it is rolled back.
- Change policy. A new rule first collects hits in report mode and only after review is moved to blocking — mass rollout stops do not happen.
- Exception procedure. How to file an exception by namespace or label, who approves it, and when it is removed without requiring edits to the rule itself.
- On-call shift instructions. What to do if a rule stopped a rollout: how to distinguish a false positive from a real violation, where to look at the log, and in what order to remove the block.
- Trigger reports. Configured export of events by rule — see which checks most often obstruct work and which are due for tightening.
How manual configuration edits were eliminated in one cluster
A cluster with several dozen services and three environments — development, pre-production, production. Any manifest edit went through manual review: the on-call engineer opened the diff, checked limits, labels, security policies and the image registry. Divergences between environments surfaced after release.
What was
60-80 edits per week came to review, each took from ten minutes. About 10% reached production — the rest returned for revision after build and tests. The author did not understand what exactly they violated: there were no formal rules, only agreements.
What was changed
The rules were moved to Kyverno. Validation rejects the manifest before apply: no resource limits, no owner label, floating image tag, writing to another namespace.
Mutation sets missing fields itself and brings environments to a common template, generation creates quotas and network rules along with the application.
What became
Rejection arrives within a minute and indicates the triggered rule — early rejection instead of after-the-fact analysis. Manual checks decreased by an order of magnitude, error response time dropped from hours to a minute.
Configuration rework after release almost disappeared, and environment drift stopped being a separate task.
What to do if the cluster already has its own rules and exceptions?
Rules do not cancel what has already been done — they stand alongside and gradually replace existing work.
We start with audit mode: the new set records violations but does not block deployment, and it is immediately clear where it diverges from previously granted exceptions and which workloads it affects first. Rules are moved to blocking mode one at a time and only after reviewing these divergences.
Rule conflicts most often arise at the intersection of mutation and validation. If a rule first adds a label or sidecar container, validation sees the already modified object and triggers differently than expected.
Therefore we describe the target state with mutation, and requirements with validation on top of it, and keep them together so the enforcement order is predictable.
We build inheritance top-down: base requirements live at the cluster level, extensions in individual namespaces.
An exception declared above applies everywhere; an exception below must not quietly remove a requirement for neighbors, otherwise one namespace weakens the policy for the entire cluster.
We migrate each old exception as a separate record with a reason and review deadline, not as a general bypass of checks. What cannot be explained in words is closed by a requirement — usually these are rules added at some point "temporarily."
Custom checks are retired last: until the new set in audit shows a match with them across all workloads, scripts remain a safety net. Then they are disabled one by one, and the cluster retains a single source of requirements with a clear enforcement order.
Technical concerns: limitations and compatibility
The syntax of Kyverno rules is rarely a source of difficulty; questions begin in operations.
What happens to a release if the controller is unavailable, how not to affect system components, and what to change during version upgrades — below we answer the concerns most often raised by the cluster maintenance team.
What happens if the policy controller is unavailable?
The cluster follows the selected failure mode: soft passes requests without checks, strict denies them. We apply strict selectively, to critical rules, with a short timeout and narrow validation scope.
Can policies stop an application release?
Before enforcement, we run the rule in audit mode: reports show how many objects will violate it, while the release is not blocked. Moving to strict mode only after divergences are resolved.
How to exclude cluster system components?
We exclude networking, autoscaling and storage from rules via namespace label exceptions. Otherwise a single check can stop system workloads before application ones.
What limitations do the rules themselves have?
Not everything can be expressed declaratively: cross-resource dependencies and mutation order are described explicitly. Generation creates objects in new namespaces — it does not apply to existing ones.
How does version upgrade work?
Rules are tied to the cluster API version. We test upgrades on staging with the same set of manifests and roll them out in stages, preserving compatibility with current policies.
Let's discuss the task: where setup begins
To keep task assessment to one short conversation rather than a series of clarifications, it helps to gather several inputs in advance.
Nothing exotic is needed — what your team already has: read access to the cluster, current policy manifests, and a description of which checks are already covered by built-in tools and where they fail.
Then we go through the cluster and see which validation, mutation and generation rules will produce a real effect, and which conflict with existing workloads.
We return with a timeline estimate and work plan: what we do first, which policies we enable in audit mode, which in blocking mode, and how each change is rolled back.
What is needed from the customer to start:
- List of clusters and environments with the order in which we touch them.
- Read access to the API server and the manifest repository — can be through your engineer.
- Current policies: what is configured manually, what through admission controllers, what exceptions have already accumulated.
- Pain points: which deployments broke due to incorrect tags, limits or missing labels.
- Security and audit requirements: which checks must block deployment, and which only need to be logged.
- Contact of the engineer who will confirm decisions on policies and access rights.
- Change window constraints — when it is acceptable to enable blocking mode in the production environment.
On our side, we provide a policy placement scheme, configuration split by namespaces, a staging validation procedure, and a description of the rollback procedure.
If some inputs are still unknown, that is not a reason to postpone: we will start with an audit of the current state and close the gaps as we go.