Drift Detection in GitOps

GitOps Drift Detection: why configuration diverges from the repository

The environment lives its own life: someone adjusted resource limits or replica count directly in the cluster because it was "on fire," and the change remained outside commit history.

The repository branch still describes the reference, but different values are actually running. A month later, no one remembers either the author or the reason, and configuration drift surfaces only during an incident — when you need to quickly understand what is actually there.

The mechanics of state drift are monotonous. An engineer bypasses the release pipeline: changes a ConfigMap, adds an annotation, expands the rights of a service account, edits Ingress rules directly.

Undeclared changes do not enter review or rollback — they can be neither reproduced on a staging environment nor transferred to another environment. Argo CD and similar controllers see such objects as diverged, but detection itself is not a fix.

Hence the fork.

Auto-sync silently returns the object to the manifest and breaks a working workaround — we get an outage because "the system fixed itself." With manual sync, the drift lives for months and accumulates: dozens of small manual edits in the environment add up to a state that can no longer be brought up from scratch from the repository.

The cost manifests at the moment of failure: diagnostics begins with checking "what is actually there," not with logs, and rolling back a release returns not the state that existed before it.

GitOps drift detection turns this verification from an investigation into a regular check with an explicit list of divergences. Below we discuss how this is addressed.

The effect of drift control: time, failures, downtime

As long as configuration lives only in Git, any change in the cluster — a manual edit via the console, a deleted resource, an object created bypassing the repository — remains unnoticed until the next incident.

The controller continuously compares the desired state from the repository with the actual state in the environment, so GitOps drift detection turns a deviation into an observable event with a timestamp and the author of the edit, rather than a surprise during an outage.

Reaction time to a deviation drops from hours to minutes: the signal comes from the controller immediately, not from an on-call engineer recalling who changed the object and when.

Automatic reconciliation returns the resource to the state from the repository — with self-healing enabled, a manual edit lives only until the next comparison.

A whole class of failures that no application check catches also goes away: an environment where settings have diverged between environments, an access rule added bypassing review, an extra resource forgotten after debugging.

Such deviations are not reproduced on the test environment because they exist only in one environment and only from the moment of the edit.

Recovery time stops growing with the lifespan of the environment. Restoration is done by applying the repository, not by manual reconstruction from memory, so the longest phase of incident analysis is shortened — downtime goes away with it.

Transparency of environment state is added as a side effect: for each environment, it is visible whether the actual matches the declared, and the drift report is read in the normal cycle rather than assembled by a separate investigation.

Drift control modes: comparison of approaches

Drift control modes differ in two parameters: what triggers the comparison and what is allowed to be done outside the repository. The first determines how quickly a divergence becomes visible; the second determines how long it lives before resolution. Below are the real options that are chosen between, with their weak points.

In practice, modes are combined: an event provides a quick response, a periodic cycle insures against lost notifications, and a pause is issued for a limited period with a journal entry.

Argo CD and related controllers support all the listed options, so the choice is not based on the tool's capabilities but on the cost of error in a specific environment.

Implementing drift control: stages of engineering work

Drift control is not enabled with a single flag: first you need to understand what actually lives in the cluster and what of that is described declaratively.

The work goes from inventory to regulation, and at each step the result is verifiable — otherwise the comparison quickly turns into a source of false positives that engineers learn to ignore.

1. Infrastructure audit. We collect the actual state: which objects are deployed, what was created manually or by third-party operators, where manifests in the repository diverge from reality and for what reason — an edit via the console, autoscaling, an external controller, custom calculation.
2. Coverage scoping. We decide what we transfer to automatic comparison and self-healing, and what remains outside it: objects with noisy fields, generated secrets, resources that the cluster itself changes. Without this, rollback will overwrite legitimate changes.
3. Repository layout. We arrange environments and teams by the structure of repositories and paths: separate branches or directories per environment, shared base layers, synchronization order of dependent components. The layout determines who makes a change, how, and how it reaches production.
4. Configuring state comparison. We enable periodic comparison with an interval that catches drift without excessive load, add event-based comparison, set behavior on divergence — signal only or reconciliation to the reference — and cleanup rules for objects deleted from the repository.
5. Notification routing. Divergences go where they will be seen: we separate "changed manually" and "deleted in the cluster," configure recipients and priority so the signal is not lost in the general flow.
6. Staging verification. We run scenarios: manual editing of an object, deleting a pod, rolling back a release, unavailability of an external service. We see what the system restores, what it leaves as is, and how long it takes.
7. Handover of regulations. We provide documentation on adding services, analyzing triggers, and emergency disabling of self-healing, and conduct training for on-call engineers.

Then we enable control in production gradually — by namespaces or teams, monitoring the first triggers to distinguish real drift from configuration noise.

Implementation artifacts: manifests, policies, regulations

Drift detection works exactly as well as your repository is described.

Therefore, at the end of implementation, the customer is left not with a "configured system," but with a set of files and agreements according to which any change passes through a verifiable path — from a change in the manifest to a confirmed cluster state.

• Manifests and overlays — the base resource description plus overrides for each environment, so that differences between environments are declared explicitly rather than reconstructed from an on-call engineer's memory.
• Repository structure — what is considered the source of truth, where environment overlays are located, where policies are, how branches relate to environments, and what happens on merge.
• Access policies — who changes the resource description and in what role, which paths are protected from direct edits, what requires a second approval.
• Configuration of the checks themselves — schedule, thresholds, list of exceptions for fields that the cluster changes itself, so reports do not drown in noise.
• Team instructions — how to read the report, how to distinguish a harmless deviation from a real divergence, how to safely return the state to the described one.
• Change regulations — the order of editing, review, running checks, and the moment of application: a rule under which bypassing the repository becomes visible rather than remaining unnoticed.
• Responsibility map — who owns the repository, who approves exceptions, and to whom the signal goes on divergence.

We hand over the package together with a walkthrough: we go through each file, show what it does, and verify that the documentation matches the actual state of the environments. Then it is updated as the infrastructure grows — according to the same regulations as other changes.

Case study: how divergences stopped reaching production

The customer is a product team: more than thirty services in Kubernetes, configuration delivered from the repository via Argo CD.

The repository was considered the single source of truth, but during incidents the on-call engineer edited objects in the cluster manually to bring the service up faster, and such changes were not returned back to code.

The bottleneck was not the manual edit itself, but the moment when it became known: the divergence surfaced either during the next deployment, when automation overwrote manual changes and the service fell again, or in a post-mortem a day later.

Detection time for a deviation was measured in hours, and half of production rollbacks were due to desynchronization between the repository and the cluster.

What was changed: periodic comparison of desired and actual state was enabled and namespaces were separated by mode — for critical ones, only notification was left; for the rest, automatic reconciliation to the state from the repository.

The notification goes to the on-call channel indicating the object, field, and author of the change, and editing bypassing the repository is closed: a change passes only through verification and a commit to the main branch.

Metrics before and after

After a quarter, manual edits during incidents decreased by approximately 80%, and post-mortems asking "why did the configuration revert after deployment" disappeared as a class: drift is caught by automation, not by a human.

Drift after synchronization: what to do if divergences return?

Repeated divergences after successful synchronization are almost never a failure of the controller itself, but a situation where the same object is edited by two systems.

Until the second author of the change is found, synchronization catches up with reality rather than setting it, and the divergence returns within the next comparison cycle.

Therefore, we start not with the tool, but with the edit history of a specific field: who last changed it and when.

A significant part of the causes lies in the environment itself. Automatic operators (storage drivers, certificate managers, load controllers) add their own fields to objects: identifiers, labels, statuses, replica count.

For them, the manifest in the repository is incomplete by definition, and such drift is not an error but a normal mode.

External changes in the environment also come from cloud services: the load balancer, autoscaling, external DNS edit resources bypassing the repository, and this is also expected.

A separate category is manual work. Editing a field directly in production during an incident lives until the next synchronization and returns again if it is not transferred back to the repository.

The issue here is not the engineer's discipline, but the absence of a procedure: direct access has no mandatory return step, so the "temporary" solution becomes permanent.

The working approach is not to crank control down to zero divergences, but to describe exceptions to control explicitly. For operators, exceptions are set by specific paths: what they are allowed to rewrite, and what is closed and must strictly match the repository.

Manual changes get a fast track through the repository with mandatory return, otherwise they will again become a source of drift.

Recurring drift is cured by separation of ownership: each field has one owner — either the repository or an external system. Then the signal is not any divergence, but only an unexpected one, and it immediately points to a specific author of the change.

Technical doubts: secrets, permissions, manual edits

Before starting, doubts are usually the same: where to store sensitive values, whether permissions are sufficient, what to do with resources created manually. Below are short answers on five points that most often come up during the approval stage.

Will automation get excessive permissions in the environment?

We grant permissions on a least-privilege basis: synchronization covers only target namespaces; everything else is closed to it. Access to the production environment is separated from test and granted by name.

Can a change be rolled back?

Yes. The state is described declaratively, so rollback is returning to the previous revision and reapplying. The return takes minutes and does not require manual editing of resources.

What to do with resources that were already created manually?

We import the existing descriptions into the repository and compare them with the actual state — divergences are shown before the first apply. We do not delete anything blindly: first we capture the current view of the resource.

What if some systems live outside the repository?

We connect such systems with separate descriptions or leave them outside the synchronization loop. Then divergences on them are visible in the report but do not block the operation of other services.

Let's discuss the task and come back with an implementation plan

To come back with an estimate of the scope of work rather than general words, we need a snapshot of your environment: where the infrastructure description currently resides, who and how makes changes besides it, and which environments need to be taken under control first.

Based on these inputs, we assemble a work scheme and sequence of actions — from an observable environment to blocking checks.

• How many environments and clusters are in scope, which of them are shared and which are isolated by individual teams.
• Where and in what form the current infrastructure description is stored: a single storage or scattered files that no one synchronizes.
• How access is organized: roles, service accounts, where secrets are stored, and who has the right to write to the main branch.
• Which changes are made manually — emergency edits via the console, manual load balancer settings, resource tuning in the cloud panel.
• Approval procedure and change windows: when blocking checks can be enabled so as not to stop releases.
• Requirements for rollback and the journal: how long to store divergence history, who analyzes triggers, and by what procedure.

Then we agree on the starting environment — one environment where we test detection and response — and if the result matches expectations, we roll it out to the rest.

Describe the task to the extent it exists now: a draft will also do, and we will ask clarifying questions ourselves.