Flux CD instead of manual edits: where delivery breaks
While manifests sit in the repository and are applied by hand, the described and actual state diverge.
An urgent fix in a nightly rollout, a forgotten parameter in values, a release deployed from the wrong branch — and a month later no one can confidently say what is actually running in the cluster.
Configuration drift accumulates unnoticed: manual deploy bypasses checks, cluster desync grows from environment to environment, and the environment stops being reproducible from scratch.
Incident analysis of a rollout turns into archaeology — they compare logs, edit history, and what was actually applied, instead of looking at a single source of truth.
At night, the cost of error is higher: the on-call engineer edits a resource manually, the state diverges even more, and rollback is done from memory.
Without automatic reconciliation of the cluster to the described view, every edit in prod moves the contour further from the repository, and the next rollout proceeds from an already incorrect baseline.
Flux CD removes the very possibility of such divergence: the agent reconciles the cluster with the repository and brings it to the described state, rather than waiting for someone to apply an edit manually.
Next, we break down how this is addressed: how the change flow is built, what to do with already accumulated drift, and how to see a failure before the user reports it.
What managed delivery provides: speed, rollbacks, downtime
Delivery is described declaratively: manifests and release values are in Git, and Flux CD compares the actual cluster state with the described one and brings it to the desired state.
Reconciliation runs in cycles, so divergence is eliminated by itself — without logging into servers and manual commands in the cluster console.
The unit of change becomes a commit, not a set of on-call actions: what is deployed in each environment is visible from repository history.
Release speed increases due to the absence of manual steps between image build and prod apply.
Updating a manifest in the environment branch triggers a rollout, and the controller itself brings the release to readiness — the path from change merge to running instance is measured in minutes and does not depend on who is on call.
Rollback of changes is one operation: reverting a commit or switching to the previous revision of the manifest, after which reconciliation itself returns the cluster to the previous state.
A specific HelmRelease is rolled back, not the entire release train: neighboring services continue running on their versions.
Service downtime is reduced in two places. Configuration drift — edits bypassing the repository, forgotten hotfixes, diverged values — is removed during the next reconciliation cycle.
The failure class "someone changed something on the server and no one remembers what" goes away: the source of truth is one, and divergence is visible and reversible.
Rollout reliability becomes measurable: the share of changes that reached prod without manual intervention, and the time to return to the previous version. Both indicators are set by reconciliation configuration, not by the care of an individual engineer.
Flux CD: comparing state synchronization methods
Cluster state diverges from the repository for two reasons: an object was edited manually or a manifest was applied from an external build system. The speed of rollback, reproducibility, and whether we notice divergence at all depend on who initiates the apply.
The flexibility of Flux CD is that continuous reconciliation and external rollout triggering do not exclude each other. The key criterion is reconciliation: only it returns an object to the described form after a manual edit or pod failure.
External triggering gives control over the moment of apply, but between triggers drift is not eliminated.
| Option | When it fits | Limitations of the approach |
|---|---|---|
| Agent in cluster with reconciliation | Production contours where automatic return to repository state is needed | Requires permissions to change objects and discipline: the repository is the only source of truth |
| Rollout from external pipeline | Releases with checks on the build system side | Drift is eliminated only at trigger time; between rollouts divergences accumulate |
| Hybrid scheme | Different pace of change for applications and platform services | Two sources of initiative: without clear responsibility boundaries, apply conflicts arise |
| Direct edits in cluster | Incident troubleshooting, urgent patch | Not reproducible: the next reconciliation overwrites the edit, change history is lost |
The working option in practice is hybrid: platform services live on reconciliation, application releases go through rollout, and direct edits remain an emergency tool with mandatory transfer to the repository.
Before enabling reconciliation, make sure all objects in the contour are already described in the repository — otherwise the first cycle will overwrite what was deployed manually.
Implementation order: from audit to handover to support
GitOps implementation starts not with installing the controller, but with an inventory of what already runs in the cluster. Until there is an inventory of services and dependencies between them, automation reproduces not the desired state but accumulated chaos. The order is therefore: audit, description storage scheme, reconciliation, roles, failure testing.
- Cluster audit: inventory of workloads, ingress rules, volumes, and external dependencies, comparison with current manifests. Output — divergences and a list of services that cannot be handed to automation without rework.
- Repository scheme: platform base layer, shared components and applications are separated, environments and layer apply order are defined. It is visible what lies in the shared layer and what is in the settings of a specific contour.
- Reconciliation setup: deploy Flux CD, connect the repository to the cluster, set the reconciliation interval, reaction to manual edits, and pruning of deleted objects. Helm releases are described declaratively, with explicit dependencies between them.
- Role setup: permissions for the main branch, change approval, and the controller's own authority are separated. Service accounts receive the minimally required access.
- Fault tolerance testing: simulate node failure, object edit bypassing Git, and loss of connection to the repository — see whether the state returns on its own and how quickly.
- Handover to support: change policy, repository scheme, and rollback procedure go into documentation; incidents are analyzed together with the customer's team.
Deliverables: manifests, schemes, policies
Upon completion, the customer's team retains not only a working contour, but also a set of artifacts by which it is developed and handed over to other engineers.
Everything lies in the git repository next to the application code: cluster state is restored from change history, not from a specific person's memory.
- HelmRelease manifests for each service: pinned chart versions, values for each environment, explicit dependencies and installation order.
- Release descriptions — what is included in the rollout, which changes affect state, where a manual step is needed, and how to roll back to the previous version.
- Storage structure in the repository: separation by environments, base values and per-environment overrides, a separate branch for the cluster's own configuration.
- Access policies: roles and permissions of service accounts, prohibition of direct edits in the cluster bypassing the repository, separation of responsibility zones between teams.
- Contour scheme: connections between repository, controller, image registry, and monitoring, system behavior when desired and actual state diverge.
- Contour documentation: recovery procedure after failure, control points, where to look on a synchronization error, and how to read release status.
- Release policy: who approves the rollout, which checks are mandatory before change merge, how rollback is documented, and who is on call after it.
The set remains with you: edits go through git, review, and merge; then the controller itself brings the cluster to the described state.
Case study: manual rollouts were removed in a contour of twelve services
A contour of twelve services lived on manual rollouts: an engineer built an image, edited manifests in place, and applied them to the cluster.
A release took 14 manual operations per service — from updating the tag to checking state after rollout, and any of them could be performed differently.
Divergences between the described state in the repository and the actual state in the cluster accumulated: a week after release, answering "what is running now" required walking through environments manually.
What was changed
Took Flux CD as the single source of truth: service descriptions were moved to the repository, initial controller installation took one run, and manifests were laid out by HelmRelease with separate environments.
Manual operations were replaced with synchronization from the repository, and state reconciliation was configured at an interval at which divergences do not have time to accumulate. Update and rollback became one operation with a predictable result.
What became
The number of manual operations per release fell from 14 to one or two — editing a value in the repository and watching the rollout.
Recovery time after failure dropped from forty minutes to three or four: rollback is switching to the previous description, not searching for a working manifest across environments.
The share of divergences between described and actual state came to zero — the controller itself brings the cluster to the repository, without human involvement in the contour.
What to do if manual edits already exist in the cluster?
Manual edits in a live cluster are not a reason to rewrite everything at once. Start with reconciliation in observation mode: the controller compares object state with descriptions but does not apply changes itself. Output — a divergence map obtained without risk to running services.
The first step of the transition period is object inventory.
Mark what came from the repository and what was created manually, and sort findings into three groups: permanent improvements that need to be moved into descriptions; urgent fixes that are due to be rolled back; objects that intentionally live outside the repository. For each group, immediately record the owner and deadline.
Then configure exclusions from reconciliation. For individual fields, set an ignore annotation; for releases, set rules for ignoring divergences by specific specification paths; for some resources, temporarily suspend reconciliation entirely.
Each exception has an expiration date: otherwise, instead of a transition period, we get a second source of truth.
Consolidation of edits goes in waves — namespace by namespace, service by service. Move an edit into the description via merge request, remove the exception, and let the controller apply state. Apply is idempotent, so switching happens without downtime.
A separate topic is field ownership conflicts: we close them via server-side apply, explicitly handing the field to the controller. Flux CD shows the change plan before apply — this is how we verify that the description and live object have finally matched.
Technical concerns: secrets, drift, rollback, monitoring
Engineers ask the same set of questions before start: where secrets are stored, what happens when the cluster diverges from the repository, how to roll back a release, and who learns about failure first.
Below are short answers for each item, to remove doubts at the design stage rather than during an incident.
Are secrets stored in plain text in the repository?
Values are encrypted before commit, decryption happens inside the cluster by a separate controller. The key never leaves the cluster, so access to the repository does not equal access to production secrets.
How is configuration drift tracked?
The controller periodically reconciles the desired state with the actual state in the cluster. A manual edit via command line rolls back to the repository version; for critical objects, reconciliation is suspended so as not to overwrite an emergency change.
What happens if the controller loses connection to the repository?
It runs inside the cluster: repository unavailability does not stop running services. New versions are not applied, the cluster remains at the last consistent state, and missed reconciliations raise an alert.
How to roll back a failed release?
Rollback is reverting a commit: synchronization brings the cluster to the previous set of manifests. Helm release revision history allows rollback to the previous one, but the result is recorded in the repository, not only in the cluster.
How to understand that synchronization is broken?
Controller metrics: reconciliation success and duration, number of revisions, release readiness. Alerts for unsuccessful synchronization, sustained drift, and stuck updates, so the incident is visible before a user complaint.
Let's discuss your task: what is needed to assess the contour
To come back with an assessment and work plan, not general words, we need only to see a few things about your contour. Nothing needs to be prepared in advance — some answers are found already at the first meeting.
- Repository structure with manifests: where desired state is stored, how many environments there are, and how they differ.
- Access and boundaries: which cluster, cloud, or own servers, who owns permissions, and how roles for automation are issued.
- Environment scheme: test and production contours, restrictions on prod — approvals, release windows, access policies.
- Current change delivery method: manual commands, scripts, or build pipeline — and what annoys you most about it.
- Secrets: where keys, tokens, and certificates are stored, whether an external vault is needed, and who manages it.
- Observability: which metrics, logs, and notifications are already collected, where they go, and who looks at them.
- Rollback requirements: how quickly the previous state must be restored and who makes that decision.
Based on this set, we put together the contour scheme, scope of work, and order of stages: what we configure first, where we place checks, and at which step we connect monitoring.
Some items from the list may be missing — that is not an obstacle; such items will simply be included in the plan as separate steps.
If part of the contour is already described declaratively — say, on Flux CD — the review starts with a revision of existing settings, and this noticeably speeds up assessment. Describe the task — we will come back with an assessment, scheme, and work plan.