Why pull-based deployment breaks under manual edits
Pull-based deployment rests on one assumption: the repository fully describes the desired state of the environment, and an agent inside the cluster regularly compares reality against it and brings reality to the described form.
As long as the assumption holds, rollout is a change in the repository, and its result is predictable.
It all breaks on the first manual edit. A parameter is changed directly in the production environment — via command line or dashboard — to resolve an incident faster.
The environment diverges from the repository, and configuration drift does not manifest itself until the next synchronization.
Then comes a fork, and both paths lead to loss. With reconciliation enabled, the agent sees the live config as a deviation and overwrites it — the incident returns with the next rollout.
If reconciliation is paused, desynchronization between the environment and the repository accumulates: edits surface in a release that are in no change, and some of others' modifications are lost. Rollout stops being predictable — its outcome depends on who managed to change what by hand.
That is why manual edits in the cluster are either made impossible or automatically returned to the repository. Argo CD and related controllers show divergence, but the access policy, incident procedure, and method of transferring edits back are determined by people. Below is how this is closed in practice.
What managed change delivery to a cluster provides
We move to pull-based deployment: workload descriptions live in the repository, and a controller inside the cluster pulls changes itself and brings the state to the described one.
No one goes into production by hand, so the class of failures "applied the wrong thing in the wrong place" goes away along with direct access.
Transparency of changes is provided by commit history: the author, review, what exactly changed, and when it took effect are visible.
Rollback speed is measured in minutes: returning to the previous description is a commit revert, after which the controller itself returns the previous state. There is no need to recall what was changed on which server and repeat the steps in reverse order under pressure.
The share of manual operations drops to edits in the repository: rights in the production environment remain with the controller, not with a list of engineers with administrator access.
Environment reproducibility rests on the same descriptions for test and production environments — divergence between environments is caught as automatically detected drift rather than surfacing at release time.
Downtime is reduced because application follows a verified path, and cluster state is continuously compared with the reference.
The result: rollback — minutes, manual operations — a few, drift between environments — under control, and analysis of any incident relies on change history rather than participants' memory.
Pull-based deployment options: comparison of operating modes
Pull-based deployment schemes are built around one principle: the GitOps delivery pipeline stores the desired state, and the cluster side initiates application.
The difference is who notifies the agent about a new commit — network access and permission requirements depend on this.
In practice, the choice comes down to three modes: agent polling on a schedule, polling by external signal, and a hybrid rollout scheme. Below is which environment each suits and what limitations of the scheme will have to be accepted.
| Option | When it fits | Limitations |
|---|---|---|
| Agent polling on a schedule | One environment, steady release pace, minimal network exceptions | Delivery delay equals the polling interval, extra repository requests during idle time |
| Polling by external signal | Strict access requirements, fast reaction to a commit needed | A signal receiver with signature verification is needed; if a signal is lost, polling repeats |
| Hybrid rollout scheme | Phased migration, multiple environments, some workloads without an agent | Two delivery mechanisms, harder to track which commit was applied where |
Common to all modes: the rollout decision is made based on repository state, so rollback is a commit revert, not a manual edit on the server. If this rule is not fixed and agent permissions are not limited, the scheme loses its main advantage — reproducibility.
How we build the change delivery pipeline: work stages
The work sequence is arranged so that risks are removed before the pipeline starts being applied in production.
The main bottleneck when implementing pull-based deployment is not the agent but the opaque current rollout: changes come in by hand and via scripts, and no one can say exactly what is currently in the cluster.
- Rollout audit. We assemble a diagram of environments, access rights, and all points of manual change application. The output is a map of divergences between repositories and actual state.
- Pipeline design. We separate the code repository and the state repository, describe the environment model and the reconciliation rule.
- Agent setup. The agent runs inside the segment and pulls state from the repository itself — no inbound connections from outside are needed. We set minimal permissions and the procedure for working with secrets.
- Rollout migration. We move services one by one, starting with non-critical ones, and compare the result with previous behavior. We immediately configure divergence alerts and rollback by reverting a commit.
- Acceptance testing. We run scenarios: planned release, rollback, manual edit directly in the cluster, unavailability of external services. We measure change application and recovery time.
- Handover to operations. We provide documentation, an edit procedure, and instructions for the on-call engineer, and train the team. The pipeline is accepted when the on-call shift releases and rolls back a release without us.
Then — support under an agreed procedure: the set of edits changes, but the delivery method remains unchanged.
What remains with you after implementation: artifacts and procedures
Pull-based deployment is handed over not as a set of scripts but as a package by which the pipeline can be operated without the author of the changes.
Everything lies in the repository next to the configuration and is versioned with it: when the pipeline is edited, the procedure is updated too, not a chat message.
- Environment diagrams: test, pre-production, and production, indicating where the agent runs, what outbound connections it opens, and what remains inaccessible from outside.
- Access matrix: for each role — what can be read, changed, and applied, and which actions require a second confirmation.
- Agent access policies: what the agent is allowed to pull from the repository, which parts of the infrastructure it is tied to, and what happens if connection to it is lost.
- Rollback procedure: step-by-step return to the previous version, including checking dependency state and the criterion by which the on-call engineer decides to roll back.
- Operations documentation: rotation of keys and tokens, agent updates, analysis of typical alert types.
- Instructions for the on-call shift: short scenarios "alert → what to check → what to do" for 5-8 common failures.
- Access provisioning procedure: how to onboard a new engineer and how to revoke their access when they leave.
Each item is a working document, not a description of an idea: using it, an on-call engineer without the implementation author brings the pipeline up after a failure and confirms the legitimacy of changes during review.
Case: divergences between environment and repository eliminated
Baseline
Before switching to repository polling, production rollout consisted of fourteen manual steps: editing manifests directly on servers, restarting services one by one, checking parameters against a checklist.
Over a quarter, six incidents accumulated related specifically to rollout, and almost every one was a divergence between environment and repository.
The cost of an incident averaged forty minutes of downtime: while they found who changed what over the past hours, the service degraded.
A separate problem was that the environment could not be brought up from scratch from the repository: some settings lived only on the server, and there was nothing to reproduce the environment for analysis.
What was changed
An agent inside the cluster polls the repository itself and brings the environment to the described state — the build pipeline no longer has external access to production.
Manual edits on servers are closed: any change goes through a commit, and authorship and reason are visible in history. Divergence is recorded automatically and does not live longer than a polling cycle, and rollback is a return to the previous state, not a rebuild.
What resulted
The number of manual operations per release fell from fourteen to two — rollout confirmation and checking key metrics. Recovery time decreased from forty minutes to six: rollback is started with one command and does not depend on which engineer is on call.
Rollout incidents over two quarters — one, and it was from an external dependency, not from configuration divergence.
How does polling state from a repository differ from rollout?
The key difference is who initiates the change. With push, an external system connects to the environment itself and applies the update; with pull-based deployment, an agent inside the environment periodically polls the repository and decides itself what to apply.
The difference seems cosmetic right up to the moment you start counting the perimeter and permissions.
In a push deployment scheme, the deployer needs access inside: keys, network rules, an account with broad rights to change the environment. Compromise of this account or the source system means access to all environments at once.
With polling, access is inverted: the environment holds only an outbound connection to the repository, no inbound ports are open, and it has no rights to change anything from outside.
Loss of connection behaves differently too. An interrupted rollout leaves a partially applied state: some nodes updated, some not, and this has to be sorted out manually.
An agent that has lost connection continues working on the last agreed version and catches up itself when the channel returns.
Hence the trust model: only someone with write access to the repository can initiate an update. The change goes through approval and remains in history — who, what, and when changed is visible without polling the environment itself.
The scheme is chosen by environment: closed environments with strict access requirements lean toward polling, while the need for external starts and fast reaction tilts toward push.
In practice, schemes are combined, but the change window is managed in one way — otherwise environment state stops being predictable.
Technical concerns: access, rollbacks, configuration drift
Five questions most often asked during the design stage of pull-based deployment. We answer substantively: how it is arranged and what happens in an emergency.
What access rights does the agent inside the infrastructure need?
Minimal: read manifests and write them to its own environment. The connection is initiated by the agent itself, so inbound ports and external keys are not opened, and rights are limited to one environment — compromise of the agent does not open access to other environments.
How is a failed release rolled back?
Rollback is returning the manifest to the previous state in the repository. The agent itself brings the environment to the described form, manual access to servers is not needed, and recovery time does not depend on who is on call.
Where should secrets be stored if everything is described by configuration?
The repository contains only references and ciphertext; decryption keys remain inside the infrastructure. Rotation happens without editing manifests, and change history does not reveal values.
How do you separate multiple environments in one repository?
Directories and promotion rules separate test and production environments. A change passes along the chain, production is protected by separate approval — an edit on test does not automatically go to production.
What happens if the agent loses connection to the repository?
The last applied version remains in the environment; nothing rolls back by itself. After connection is restored, the agent compares desired state with actual state and catches up, and shows manual divergence as an alert rather than applying it silently.
Let's discuss your task: we will return with an estimate and work plan
So that the estimate and work plan rely on facts rather than assumptions, we need a view of your infrastructure. Describe the task in free form: what already exists, what access restrictions there are, and what is considered a successful result.
- which environments we are moving to pull-based deployment and in what order — from test to production;
- where manifests and environment settings currently live: repository, branches, change review rules;
- how many teams publish changes, how often, and how fast the rollout should be;
- how access is arranged: who issues credentials, how secret rotation works, and the agent in the cluster;
- what is critical to preserve: rollout windows, approval procedure, rollback to the previous version;
- which checks are mandatory before application: build, tests, image vulnerability scanning.
In response, we send an estimate and work plan: a change delivery scheme, stages of environment migration, and a list of artifacts with roles. We separately mark control points where it is possible to stop without losing operability.
- a scheme of interaction between repository, agent, and cluster;
- a change delivery implementation procedure with control points;
- a list of artifacts: configurations, policies, roles, documentation, and instructions for the on-call shift.
We are ready to discuss the task — we will return with an estimate and work plan and name the team composition for your scope.