ArgoCD without GitOps: why cluster state diverges from the repository
Most often ArgoCD lands in a cluster as a convenient status panel: you can see what is deployed, you can see discrepancies, but changes are still made manually.
During incident triage, someone increases the replica count, expands limits, or swaps an image directly in the cluster — “temporarily, to go faster.” The patch remains in the live state, and no one remembers it in the repository.
This is how configuration drift begins: the actual state stops matching what is described in the repository. As long as synchronization is not triggered, the mismatch goes unnoticed — the application works, metrics are normal.
But once auto-sync is enabled, the cluster returns to the repository version, and along with it the workaround change that the service relies on is lost.
Then the state mismatch accumulates: temporary patches layer on top of each other, manual edits in the cluster become part of the working environment, and the source of truth remains an edit that no one recorded.
Rolling back to a previous release does not help in this scheme either — the repository describes a configuration that is not the one that actually worked.
The cost is unplanned downtime at the worst possible moment: during a rollout, rollback, or nightly node restart. ArgoCD without a closed GitOps loop remains a mirror, not a control mechanism. We examine how this is closed.
What managed delivery provides: speed, rollback, and predictability
A single source of truth — a declarative description of applications and infrastructure in the repository. The cluster is brought to this state automatically, and discrepancies are detected and resolved without an on-call engineer. This is the starting point from which all other effects are calculated.
Rollback time stops depending on approvals and manual edits: it is a single operation returning to the previous recorded state, comparable in duration to a normal apply of changes.
The share of successful rollouts grows because the order and checks are described in advance and are the same for all environments — the test environment stops differing from production in how delivery is done.
Manual labor decreases: the engineer does not assemble a release according to instructions but confirms a change that has already passed checks.
The change history stops being oral. Every edit has an author, a reason, and a link to a task, and a rollback leaves an equally clear trace.
From this grows release predictability: the team knows in advance what exactly and when will end up in the cluster, and can show this for any point in time.
Separately, it is worth naming the eliminated class of failures — “it worked in the test environment, but in production the configuration is different.” A cluster edit that bypasses the repository becomes a visible anomaly rather than a hidden norm, and it is noticed before it reaches users.
In such a scheme, ArgoCD only executes what is recorded in the repository — therefore the environment state is reproducible, not restored from memory.
These are exactly the metrics we capture at acceptance: rollback time, share of successful rollouts, and number of manual operations per release.
Change delivery modes: synchronization from the repository and alternatives
The synchronization mode is an application-level policy: whether the controller monitors divergence between the cluster and the repository and what it does when divergence is detected.
It determines who presses “deploy”: the build pipeline, the on-call engineer, or the controller itself. In ArgoCD these are two independent settings — auto-sync and self-heal — and their combinations produce different delivery models.
The choice almost always comes down to the cost of error: in a test environment it is low, in production it is not. Below are the options that teams actually choose between when there is more than one service and cluster.
| Delivery mode | When it fits | Limitations |
|---|---|---|
| Automatic rollout with self-healing | test environment, frequent small changes, readiness to fix quickly | manual edits in the cluster do not survive — the controller erases them; a faulty manifest is rolled out without pause |
| Rollout on image readiness (the pipeline updates the tag in the repository) | the release must go immediately after build and checks | depends on image registry availability; tag and manifest divergence is not caught immediately |
| Manual release confirmation (auto-sync disabled) | production environment with change windows | divergence accumulates, an on-call engineer is needed; deviations are noticed later |
| One template for multiple clusters | test and production environments with identical manifests | an error in the template propagates to all clusters simultaneously |
A practical rule: auto-sync where rollback is cheaper than approval, manual confirmation where the cost of error is higher than speed.
An intermediate option is scheduled synchronization or synchronization with ignored fields: the controller does not touch what other tools change and does not fight with them.
How implementation proceeds: from cluster assessment to handover of procedures
GitOps implementation is not installing ArgoCD into a cluster but restructuring how changes reach production.
The order of stages matters more than speed: without an inventory and a well-thought-out repository scheme, synchronization will begin to diverge from the real state within a week.
- Cluster inventory. We collect workloads, namespaces, role bindings, and manual edits via kubectl. We record discrepancies between manifests and the live state — this is the real scope of work.
- Repository scheme. We separate application and environment configuration, define the directory structure and team responsibility boundaries. It determines how many manual approvals remain with each rollout.
- Synchronization setup. We connect ArgoCD, describe Application and AppProject, enable auto-sync and state recovery after manual edits. Secrets go into Git only in encrypted form.
- Policy checks. We limit available namespaces and resource types by project, enable manifest validation on review. Configuration errors are caught before they reach the cluster. Cost calculated after audit. Test rollout. We run the release on the test environment, then on a non-critical service in production: we look at synchronization time and rollback behavior. We verify rollback by reverting a commit, not with manual commands.
- Handover to support. We hand over procedures, describe roles and access, show the on-call engineer how to read synchronization status and what to do when a release is stuck.
After acceptance, the team releases changes by commit, and we remain on support for the first releases.
What we hand over: manifests, roles, procedures, and diagrams
We hand over not a set of charts but a working environment: Argo CD is already configured, repositories are connected, the on-call engineer can reproduce the environment, roll back a release, and see who changed what.
Below is what exactly remains with the customer's team along with sources and access to the configuration storage.
- Manifests and configurations — descriptions of applications and application sets with synchronization policy: auto-sync, recovery after manual edits in the cluster, deletion of resources no longer in Git. The single source of truth is the repository, not the live cluster.
- Role model and access policies — projects with restrictions by repositories, namespaces, and resource types; integration with single sign-on and groups, a separate right to synchronize to production.
- Environment diagram — a map of development, pre-production, and production: which branches and overlays correspond to them, in what order deployment proceeds, and where configurations intentionally diverge.
- Release procedure — step-by-step order: change review, merge, synchronization, service health check, rollback decision, and recording the result.
- Secrets management — where credentials are stored, what decrypts them inside the cluster, who has access, and how key rotation is performed.
- On-call instructions — what to look at when state diverges, how to distinguish a deployment failure from an application failure, typical actions, and the escalation procedure.
- Diagrams and documentation — a diagram of the change flow from commit to cluster, a repository map, and a checklist for connecting a new service.
We review each artifact with your team during handover — so that a month later no one has to search for where the deployment logic is hardcoded.
Case study: ArgoCD removed manual deployments in a cluster of 40 services
Before GitOps implementation, a cluster with 40 services ran on manual rollouts: an engineer built an image, edited manifests directly in the cluster, and applied them via the command line.
No one could reproduce the environment state — descriptions diverged from what actually worked, and “one-minute” edits settled in the cluster forever.
We moved all descriptions to Git and handed synchronization to Argo CD: the repository became the single source of truth, and divergence between it and the cluster became visible and resolvable.
We enabled automatic synchronization and self-healing: if a resource is changed manually or a pod fails, the controller returns the declared state without human involvement.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Release rollout time | 40–60 minutes per service | 4–7 minutes, without manual steps |
| Configuration desyncs | 10–15 per month, found via failures | isolated, visible in the panel immediately |
| Rollback share | every fourth rollout | about 6% |
| Recovery after failure | 2–3 hours to investigate and manually roll back | 10–15 minutes by reverting the version in Git |
Rollback is now not a separate operation under pressure but a revert of the previous commit: the controller itself brings the cluster to the desired state.
Incident analysis shifted from the question “what did they tweak manually” to working with a clear version difference, and the change history for each service remains in the repository.
How does ArgoCD differ from a custom delivery pipeline?
ArgoCD stores the desired state in Git and continuously compares it with the cluster: divergence is visible immediately and does not require a manual run. A custom pipeline is a sequence of commands that runs once and knows nothing about the actual cluster state.
Hence the difference in observability and rollback.
In the declarative model, you can see which resources have deviated from the repository, in which environment synchronization is stuck, and which commit preceded it; rollback of changes is a return to the previous state, not a rerun. In scripts, rollback is often partial: some steps have already applied, some have not.
Responsibility for the actual state is also different. In the declarative model, the reference is the repository: if a resource is edited directly, drift will manifest and be eliminated.
In a custom pipeline, no one describes the actual state — it is composed of the side effects of runs.
Pipeline maintenance shifts from code to rules, roles, and the repository change history. Next we show how this looks in practice.
Questions about access, environments, and already-running clusters
Before starting, we discuss five questions with the customer's team — they determine whether the implementation becomes a managed environment or just another layer that engineers will start to bypass. Below is the technical essence of the answers, without generalities.
What minimal permissions does Argo CD need in the cluster?
A separate service account, not an administrator one. It only needs to read object descriptions and write to target namespaces; access to repositories is via a read-only token.
Cluster administrator roles are not granted: if the controller is compromised, the damage is limited to one environment.
How to isolate development, testing, and production environments?
We separate them by instances or by projects with namespace restrictions. The production environment is distinguished by mandatory manual synchronization confirmation and a prohibition on auto-apply from a shared branch.
Will an already-running cluster connect?
Yes, without downtime. First, the service is taken into observation mode: state is read, discrepancies with manifests are recorded, but nothing is applied. When descriptions match reality, we enable auto-apply, one service at a time.
What about the rollback plan?
Rollback is a return to the previous revision of manifests in Git: the cluster state is brought to it within minutes. The second switch is disabling auto-apply, after which changes go through only on an explicit engineer command.
What is included in support after implementation?
Incident on-call, updating the controller and access rules, connecting new services, reviewing discrepancies between description and reality. Separately — analyzing which services remained outside the environment and why.
If at least one item is not closed before the first synchronization, implementation runs into manual workarounds — that is why we record the answers in writing and attach them to the work plan.Let's discuss the task: we'll assess your cluster and come back with a plan
A substantive assessment of ArgoCD implementation is based on facts about your cluster, not on generalities. The more precise the input, the fewer assumptions remain in the plan and the faster work begins.
- Environment layout: how many clusters and environments, where they are located, whether they are connected by network and a shared image registry.
- Current rollout method: manual commands, build pipeline jobs, charts, or separate manifests — and what exactly is done manually before release.
- Repository structure: where resource descriptions are stored, whether there are separate branches for environments, who has the right to merge changes.
- Access restrictions: whether a read-only account can be issued, whether there is an isolated environment without external network, your own certificate authorities.
- Rollback requirements and release windows: how quickly the previous version needs to be restored, whether there are periods when rollout is prohibited.
- What you have already tried: an ArgoCD pilot, custom scripts, restrictions that stopped everything.
In return, we send not general considerations but a working package:
- environment layout and target topology — where the management environment is, which clusters are under observation, how rights are divided between teams;
- implementation plan by stages, with a verifiable result at each: trial on one environment, migration of the rest, disabling manual operations;
- collaboration procedure: where your engineers are needed, how changes are accepted, and who confirms the rollout;
- set of accesses and conditions for starting work — what is required from your side for the first stage.
Describe the task and attach what you have: this is enough for us to come back with an assessment and a plan.







