OPA and policies as code: where cluster settings diverge
Access rules, resource limits, and label requirements rarely live in one place.
Some settle in manifests, some in role rights, some in chat: “that’s how it’s done on this environment, don’t touch it.” While there is only one cluster, agreements rest on engineers’ memory and on reviews conducted by the same person who came up with them.
As soon as there are several environments, settings begin to diverge.
In the test cluster, someone spun up a privileged pod to test a hypothesis faster and forgot to remove it; a container without memory limits made it into prod; in the pre-production environment, network rules were loosened for debugging.
Configuration drift is not visible on monitoring dashboards—it manifests at the moment of an incident.
The cost of such configuration divergence is measurable. A defect reproduces in one environment and does not reproduce in another, so diagnostics take hours.
Checks remain manual: before a release, an engineer visually verifies that the manifest has the required values, and an audit turns into a survey of who granted privileges and why.
Policies as code move these agreements from chat into declarative rules that are applied to the cluster automatically. OPA together with Gatekeeper checks an object before it enters the cluster and rejects anything that diverges from the agreed-upon set. Below we discuss how this works and how to move cluster control from manual mode to reproducible.
What policy-as-code management gives: effects
Rules stop living in chat and documentation pages and become part of the repository: described declaratively, they pass review and enter the cluster from the same change as the application.
Open Policy Agent with Gatekeeper checks objects at cluster ingress, so there is a single source of rules for all environments—divergence between test and production environments disappears not because of engineers’ attentiveness but structurally.
Automated checking shifts feedback left: a rejected manifest is visible at build time, not a day later in the production environment.
Manual approval of each configuration leaves the release process—typical changes pass according to the described rule in minutes, not days of waiting for a chat response.
Downtime reduction rests on reversibility: a policy is a versioned artifact, so an erroneous rule is rolled back with a single revision, without manually analyzing the cluster state.
Recovery time after a failed change is measured in minutes, and there is no longer any need to look for who changed what in the console.
The share of incidents due to manual edits falls: changes bypassing the repository are either forbidden or recorded and brought to the described state.
Risk reduction proceeds in stages—first rules operate in observation mode and only log violations, and only after analyzing false positives is blocking enabled.
OPA Gatekeeper operating modes: audit, deny, auto-fix
Gatekeeper treats an object in three ways: it records a violation, rejects it, or brings the configuration to the reference.
The mode is set by the action when a constraint is triggered—report only, denial, or warning—and the scope of checking is determined by selectors by namespaces and labels.
Therefore, the maturity of an environment is described not by “enabled or not” but by where and how strictly it was enabled.
A new rule is usually rolled out along the route: audit across the entire cluster → warning → denial on pilot namespaces.
The fate of auto-fix is decided separately: it is appropriate where the requirement is unambiguous and gets in the way where the manifest author should see the divergence themselves. Below is how the modes differ in practice.
| Option | When it fits | Limitations |
|---|---|---|
| Audit mode | The rule has just been written or is being revised; you need to understand the volume of violations across the cluster | Protects nothing: an incorrect object is applied; divergence surfaces only in the report and until the next pass |
| Object denial | The rule has been tested on a real stream of manifests, the requirement is critical, there are few exceptions | Blocks releases and system components without an exception list; an error in the rule manifests at rollout time |
| Configuration auto-fix | The requirement boils down to substituting values: labels, resources, image references | The edit is not visible to the manifest author—the configuration in the repository diverges from the cluster, and debugging becomes more difficult |
| Gradual enablement by namespaces | The cluster is large, teams have different maturity levels, a sharp denial will stop work | You need to maintain an exception list and ensure it does not become a permanent bypass of the rules |
The enablement sequence leaves a rollback at each step: a rule can be returned from denial to warning without removing the check from the cluster.
Policy implementation order: from audit to handover
Policies enabled with a single commit in blocking mode almost always break workflows: old manifests and service accounts that no one remembered fall under them. Therefore, policy implementation proceeds in waves—from inventory to denials on a limited environment.
-
Resource inventory. We collect a snapshot of the cluster: which objects are created, by whom, and with which labels. Sources—the API server audit log and manifests from git.
-
Rule design. Based on the inventory, we formulate what exactly we prohibit and by which attribute: rule template, match selectors, parameters. A rule that cannot be explained in one phrase is split into several.
-
Observation mode. We enable policies in check mode without blocking and observe triggers. What matters is not the volume but the noise: a rule with a stream of false violations is rewritten, not pushed through.
-
Gradual enablement of denials. First non-production environments, then production—by namespace, not all at once. Confirmed exceptions are formalized as explicit lists, not by disabling the rule entirely.
-
Policy testing. On a staging environment, we run service deployment: standard manifests pass, forbidden ones are rejected with a clear reason. We separately test rollback—the rule is removed with a single edit.
-
Handover to operations. Policies live in git and pass review as code, triggers go to monitoring, and teams receive a regulation: how to request an exception and how to change a rule.
Deliverable package: policies, schemas, regulations, reports
The deliverable package is not an archive of manifests but a working set of artifacts in your repository. Using it, an engineer who already knows the cluster adds a policy or exception themselves—in the same way we did.
- Policy library — OPA modules and constraint templates by domain: security, resources, network. Warning mode and denial mode are kept separate, so a rule is enabled gradually.
- Exception rules — a separate file with labels for namespaces and workloads, indicating the owner and expiration date. Expired ones are visible in the report; they do not dissolve in the code.
- Access schemas — who changes policies, who only reads reports, who approves exceptions; roles are separated in your existing user directory.
- Update regulation — the path of a rule from proposal to blocking: testing on a cluster copy, warning period, then denial of application.
- Audit reports — regular exports of violations by clusters and owners; they show what prevents moving a rule to strict mode.
- Documentation and maintenance guide — for each rule: what it prohibits, whom it affects, how to fix the object; plus the rollout procedure through the validation pipeline.
None of the artifacts require our participation in edits: we hand over the package together with a walkthrough with your team; after that, the rules evolve within your environment.
Case study: how configuration divergences between environments were eliminated
The environments diverged for a long time: access rules and limits were edited manually in each environment, and identical services in test and production behaved differently.
Incorrect objects made it to release, and investigating a single access case took hours—an engineer manually compared manifests and logs across several environments.
What it was like before
Before release, a checklist was completed manually: about forty items per release, some of which were skipped under tight deadlines.
The share of rejected incorrect objects in the test environment remained at the level of every fourth, and some divergences surfaced already in production. Restoring configurations took on average a working day.
What was changed
The rules were moved into declarative policies and OPA Gatekeeper was connected: a manifest is checked before application, not after.
Policies are versioned together with the configuration, so an environment is brought up from a single set, and a rejection is always accompanied by a clear message—what was violated and how to fix it.
Result
The share of rejected incorrect objects grew to 92%: divergences are cut off at ingress rather than in production. Access investigation time dropped from hours to minutes—a policy answers the question about rights instead of manual searching through manifests.
Manual pre-release checks are no longer needed: the controller performs them on every change. Restoring an environment configuration takes minutes—the rule set resides in the repository and is deployed together with it.
How Gatekeeper differs from built-in cluster tools
Built-in tools—RBAC, pod security levels, quotas, and resource limits—trigger at object creation time and require nothing installed. They cover a narrow class of tasks: who has access to the API, what privilege level is allowed, how many resources a namespace will take. As long as requirements fit within these frames, an external policy engine is not needed.
The limitation of the built-in is its narrowness and non-composability: one rule expresses one condition.
A composite requirement (image only from a trusted registry, mandatory owner labels, prohibition of mounting host paths) cannot be described—it is covered by manual review or the gap is tolerated. The rule set is copied between clusters manually, and divergences accumulate unnoticed.
Manual regulations pick up the remainder: a pre-release checklist, manifest review, agreements in documentation. It works until the first crunch—under pressure a step is skipped, and no trace remains in the API: there is no way to recover who violated a rule and when.
Gatekeeper moves rules outside controllers: a policy is described declaratively, stored in the repository alongside manifests, and applied as a single set to all clusters.
Audit checks existing objects too—divergence is visible before an incident; parameters (trusted registries, mandatory labels) are externalized, so one policy works in both test and production environments.
The choice is simple: one cluster with basic access and resource restrictions—built-in tools are sufficient.
As soon as there are several clusters and more rules than conditions in RBAC, reuse begins—we discuss how to assemble such a set so that it does not become a source of failures.
Common technical doubts before implementation
Before implementing OPA Gatekeeper, engineers ask the same questions: will a policy stop a legitimate release, how will checks affect running workloads, how to separate rules between environments, and who will maintain them after handover. Below are answers to each.
Will a policy not block a normal release?
We start with observation mode: rules record violations in the report without rejecting anything. Then we enable warning—the developer sees the reason and the rule wording. We set denial only for unambiguous rules; for the rest, an agreed exception mechanism operates.
How do checks affect running applications?
The check occurs at object creation or modification, before it enters the cluster; it does not affect already running workloads.
We separately configure behavior when the checking environment is unavailable and the scope by namespaces, so that a rule failure does not stop the entire cluster.
How to manage policies in multiple environments?
We maintain a common rule library and per-environment overrides: in test—warning; in production—denial for the same set. Parameters reside alongside manifests and undergo the same review, so divergences are visible during edit review, not at release time.
What happens with maintenance after handover?
We hand over the policy repository with edit history and a regulation: who owns a rule, how to add an exception, how a false positive is investigated. We separately describe updating the set for new cluster versions.
Let’s discuss the task: policy audit and implementation plan
Discussion of the task begins not with a presentation but with your data. Send what you already have—that is enough to estimate the scope of the policy audit and not include stages you do not need in the plan.
- cluster inventory: distribution, current branch, number of namespaces and workloads;
- manifests or deployment charts—a link to the repository is sufficient;
- list of policies currently in effect: review, manual agreements, or admission controller;
- list of exceptions—system namespaces, privileged workloads, operators;
- mode requirements: where only violation audit is needed and where blocking is needed;
- roles and access method to the cluster control plane;
- windows during which changes are allowed in production.
We will clarify the scope of work: which requirements we translate into OPA constraint templates, what we first run in audit mode, who owns exceptions, and how violations surface in the build before rollout.
We will separately discuss where policies live in the repository and how their changes pass review—otherwise, in a month the rules will diverge from reality.
We will return with an implementation plan: stages, the order of moving checks from warning to denial, scope estimate, and the points where your engineers take over from us. Describe the task—we will compare it with what you sent and propose a work sequence.