The cluster diverges from Git: where auto-sync kubernetes breaks
Cluster synchronization with Git rarely fails outright — usually it is enabled, but with exceptions. Someone tweaked a pod's limits via kubectl "for five minutes," someone added an annotation by hand while dealing with an incident.
As long as auto-sync kubernetes is set up on the "apply and forget" principle, such edits live quietly in the cluster until the next release and never make it into a single commit.
A few weeks later, cluster desynchronization stops being a collection of small things. Manual edits in the cluster pile up, Git describes one environment while another one is actually running, and the difference is recorded nowhere.
Configuration drift strikes in the worst-case scenario: the environment is recreated from scratch from the repository, and the application comes up differently than in production — because part of the working state lived only in the live cluster.
It breaks in predictable places. Automatic re-application (self-heal) is disabled, so the divergence simply accumulates. Deletion of objects that have disappeared from the repository is not enabled — orphaned resources stay in the cluster.
The exclusion list for state comparison has grown so large that the tool stops noticing real changes. On top of that, engineers retain direct write access, which means bypassing Git is possible again at any moment.
Environment reproducibility rests on exactly one condition: the repository is the single source of truth, and nothing else deviates from it unnoticed. We break down how this is closed in ArgoCD and Flux, and what needs to be checked before enabling automatic application.
What automatic cluster state synchronization delivers
The main effect: cluster state stops being the result of manual actions.
Manifests, policies, and environment parameters live in the repository as the single source of truth, and the controller compares the actual state with the described one and brings divergences to the required form.
Automatic synchronization in Kubernetes removes the "an engineer remembered and fixed it" layer: a change reaches the cluster only after it has passed review and been captured in a commit.
Hence the first measurable effect — recovery time. Previously, losing a node, a failed update, or a destroyed environment meant manual investigation: recalling what was configured and when, finding the current configurations, repeating the steps in the right order.
Now recovery comes down to a rollout from the repository, and the count is in minutes instead of hours. The same mechanism closes an entire class of failures caused by undocumented edits — the ones made directly in the cluster that nobody remembers.
The controller finds such drift on the next reconciliation and either returns the object to the described state or flags it as a divergence visible in the interface.
Release rollback becomes predictable. A release is rolled back not with a set of manual commands under pressure, but with a revert commit: the previous described state is restored, and the controller itself brings the cluster to it, including deletion of superfluous objects. We set this up in advance and test it in a training environment, not in the middle of an incident.
Finally, change control stops resting on discipline. The commit history shows who changed what and why, every edit is tied to a review, and the divergence between repository and cluster is visible as a metric rather than as a surprise at the next release.
Comparing cluster-to-repository synchronization modes
The synchronization mode determines who applies changes from the repository to the cluster and when. ArgoCD and Flux have the same set of modes: auto-apply on commit, auto-apply with self-healing, confirmation before rollout, and scheduled runs. The only difference is what happens to a manual edit in the cluster and to a faulty commit.
Usually a mode is tied to an environment: the development branch goes to the development contour with full automation, the release branch to pre-production, a tag or a separate branch to production with confirmation.
This coexists within a single cluster because the mode is set at the application level, not for the entire cluster.
| Mode | When it fits | Limitations |
|---|---|---|
| Auto-sync with self-healing | development and testing contours where drift is unacceptable | a manual edit is rolled back within minutes, a faulty commit goes out immediately |
| Auto-sync without self-healing | environments where engineers edit resources directly | divergence accumulates silently, visible only in the report |
| Sync with confirmation before rollout | production and regulated contours | application depends on the on-call engineer: the merge exists, the rollout does not |
| Scheduled runs or repository-event triggers | rare contours, nightly windows, heavy migrations | delay between commit and apply — minutes to hours |
| Observation without synchronization | audit, inventory, acceptance of a third-party cluster | makes no changes to the cluster, no write-back |
Limitations here matter more than capabilities: automation speeds up rollout only when configuration checks pass before a merge into the repository.
Implementation stages: audit, scheme, setup, verification, handover
We work in stages: from inventorying the current state to handing over for maintenance. Automatic Kubernetes synchronization is a change in how you work with the cluster, not a toggle switch, so each step relies on the artifacts of the previous one.
- Cluster audit. We build an inventory of objects and separately mark those created by hand and those managed outside the repository. These divergences become the first candidates for being brought to the described state.
- Synchronization scheme. We define the boundaries: which contours are managed from Git, which remain manual, where separate branches per environment are needed. We agree on the access model and the order in which changes are applied.
- Repository preparation. We lay out manifests into directories, move environment parameters out, and connect checks at the commit stage — so an error is caught before it reaches the cluster, with the cost calculated on an individual basisController setup. We deploy ArgoCD or Flux in observation mode: automatic application is off, and only the actual divergence from the repository is visible.
- Testing on a staging environment. We run through scenarios: editing a manifest, rollback, an unavailable repository, a manual change directly in the cluster. We watch how long it takes for the state to converge and how the controller behaves under load.
- Switching to automatic mode. Non-critical contours first, then the rest. We enable deletion of objects that are not in the repository last — once the inventory matches reality.
- Handover for maintenance. We hand over the scheme, the change procedure, and the set of checks, and train the team — after that, changes go through a normal commit.
Configurations, schemes, and procedures delivered by the project
A project closes not with a presentation but with a set of artifacts that are used without our involvement.
Below is what the client keeps after setting up automatic cluster-to-Git synchronization: a working configuration, a relationship diagram, permissions, and the procedures the maintenance team works by independently.
- Working configurations. Cluster and application manifests in Git, laid out by environment; the differences between them are moved into separate value files rather than copied across several branches.
- Synchronization scheme. Which repository is linked to which cluster, which policies apply on divergence — deletion of superfluous objects, automatic alignment to the state in Git — and where the zone of manual changes ends.
- Access role matrix. Who edits manifests, who approves a merge, who holds rights to secrets, and who has read-only access to the cluster.
- Rollback procedure. The step-by-step order for returning to the previous state: reverting the commit, synchronizing, verifying the result — stating how long the operation takes.
- Secrets handling procedure. Where they are stored, how they are rotated, what to do in case of compromise; edits directly in the cluster are ruled out as a class.
- Maintenance documentation. What to do on divergence, when the synchronization controller is unavailable, how to connect a new environment or a new cluster.
- Signals and alerts. Which synchronization state metrics are collected, at which values a notification reaches the on-call engineer, and who handles it.
Acceptance follows the same list: we deploy an environment from scratch strictly according to the documentation, deliberately introduce a manual change and a divergence, and then check whether the rollback fits within the stated procedure.
Case study: restoring a cluster in minutes instead of hours
A cluster spanning three environments — development, pre-release, production — rested on manual edits: engineers connected directly and changed objects in place. Some settings never made it to the repository at all, others diverged from it.
Recovery after a failed update or a node failure took 4–6 hours: restore from a backup, then manually catch up on everything that had changed since it was taken, and compare environments against each other from the participants' memory.
What was changed
Object definitions were moved into Git as the single source of truth, and reconciliation and state alignment were delegated to Argo CD: it compares the cluster with the repository on a schedule and returns any deviation back on its own.
Write access to the cluster was granted only to the synchronization controller; people kept read access and incident triage. Changes now go through a merge request — with review, history, and rollback in a single revision.
Metrics before and after
- Production environment recovery time: 4–6 hours → 12–20 minutes, and without the on-call engineer's involvement.
- Share of changes that bypassed the repository: about a third → less than 2%, and those are only one-off emergency edits.
- Configuration divergence between environments: several incidents per month → none in a quarter.
An entire class of failures is gone — "we fixed it in prod and forgot to port it": on divergence from the repository, the cluster returns to the described state on its own instead of accumulating into a unique snowflake that is scary to touch.
The cluster is already set up, but auto-sync kubernetes is failing: what to do
A synchronization conflict almost always means not a controller failure but two sources of truth in one cluster. Some objects are edited by hand — through the control panel, local kubectl, or release scripts — while others come from the repository.
With self-healing enabled, the controller silently returns its version, and the manual edit disappears; without it, the application hangs in divergence and auto-sync kubernetes does not pass.
For an already deployed cluster, the first step is an ownership inventory: the metadata of the last applied config and tracking annotations show which fields were set manually and which objects were never created from the repository. Usually this surfaces resources that were edited during an incident and forgotten in Git.
Then comes a staged transition, without removing all divergences at once. First the controller (ArgoCD or Flux) runs in comparison mode: it shows the difference but applies nothing.
Then configuration migration — live manifests are pulled from the cluster and stripped of fields that the cluster itself generates.
After that, auto-sync is enabled without deleting objects outside the repository, and only once divergence is consistently zero — cleanup and self-healing.
Boundaries are drawn in advance: secrets, operator objects, and hardware-bound storage are not moved into the repository, and exclusions are set by specific paths rather than by disabling control over an object entirely.
The result: cluster state is read from the repository again, rollback is done with a single revision, and any manual edit either lands in Git or is visible as a deviation.
Technical concerns: permissions, secrets, and change rollbacks
Automatic synchronization runs into not the controller installation but the boundaries: who can change cluster state, where sensitive data lives, and what happens if the connection to the repository is lost. Below are the things engineers ask before work starts.
Who gets the right to change prod via Git?
Rights are granted in layers: a developer pushes to their own branch, a change to the environment branch passes a check and an approval, and manual access to the cluster is closed off.
Otherwise the state in Git and in the cluster diverges — and the next synchronization rolls back someone else's edit.
Where are secrets stored if everything is described in the repository?
Open passwords and keys do not remain in Git: they are either encrypted before commit or pulled from an external vault at apply time, leaving only a reference in the description. Decryption is available to the synchronization controller and a narrow circle of administrators.
Can automation be enabled partially rather than for the whole cluster?
Yes, and this is the usual implementation order: auto-apply is enabled first on the test contour and non-critical services, while prod runs on confirmation for a while.
The apply order is configured too — dependencies first, then services — and partial synchronization is visible line by line: which objects are behind and which check they failed.
What happens if the synchronization service fails?
The cluster keeps running on the last applied state — running applications do not stop, changes from Git simply are not applied.
Once the connection is restored, synchronization catches up on the difference, and a connection-loss alert arrives before a user notices the divergence.
Let's discuss the task: what is needed to assess your scheme
To come back with an assessment and a plan rather than general words, we need a snapshot of your current scheme: how environments are arranged, who has rights to change the cluster, what is already described declaratively, and what is done by hand. Below is what removes most of the uncertainty during assessment.
- Environment composition: how many clusters and namespaces, how the development, pre-production, and production contours differ.
- Current deployment procedure: how changes reach the cluster today — manual commands, scripts, a build pipeline — and at which step the process most often stalls.
- What is already in the repository: manifests, templates, environment configurations, and which objects remain outside it.
- Roles and access: who commits, who approves a production rollout, how credentials for automatic synchronization are issued.
- Constraints: security policies, private image registries, change windows, contours without external access.
- Target behavior: full auto-sync or confirmation for prod, whether a return to the previous version with a single operation is needed.
Describe the task in free form — that is enough for us to come back with an assessment and a plan.
A diagram of your environments or an example of the current process will speed up the review: they show where synchronization will deliver an effect immediately and where a staged transition will be required.