ArgoCD Image Updater: why manual image updates break releases
The build completed successfully, the image with the new tag is in the registry, the reports are green. Yet the previous build keeps running in the cluster: the manifest in Git still points to the old tag, and the cluster with that manifest is fully synchronized.
The continuous delivery tool honestly reports the state as “healthy,” because it compares Git with the cluster, not the registry with Git. The gap appears exactly at this junction and remains unnoticed until someone looks into the registry manually.
The cost shows up in production. A bug that has already been fixed reproduces again; the test environment has a newer version than production; a rollback achieves nothing because there is nothing to roll back to — the tag in the manifest has not changed since last time.
A manual rollout comes down to editing the tag in the repository for each service separately, and each such commit becomes a separate point of failure: a service was forgotten, the tag was mistyped, the fix was made in the wrong branch. Version drift between environments accumulates over releases, not minutes.
In an environment with dozens of services and frequent releases, this becomes a systemic problem.
Image updates turn into a task that is postponed until an incident, and the outdated build in the cluster is discovered during an outage investigation, when you need to quickly understand which code was actually handling requests.
Reconstructing the picture from deployment logs is difficult — the history has no “version changed” event.
ArgoCD Image Updater closes exactly this gap: it watches the registry, compares the current tag with the one recorded in the manifest, and writes the new one to the same place from which the delivery tool gets the desired state.
Next, we break down how this is configured — update policies, tag filters, repository write permissions, and rollback behavior.
What image auto-updating provides: speed, stability, control
Manual image updating is the most common source of delay between a finished change and the working environment.
The build passes, the image lands in the registry, and then the waiting begins: the on-call engineer has to remember the service, insert the new tag into the desired state description, and trigger synchronization.
The window easily stretches to hours, and with a dense release flow the queue simply does not get cleared.
ArgoCD Image Updater closes this link: it monitors the registry, compares the tag and digest with what is deployed, and initiates the update itself. Change delivery time is measured in minutes and no longer depends on who is on shift today.
The second metric is the share of manual operations. Editing a tag by hand carries a whole class of failures: a typo in the name, updating the wrong environment, a forgotten synchronization after an incident.
The update policy sets the rule once — by tag mask, by version ascending, or strictly by digest — and after that the update proceeds without human involvement. The human factor remains only where a decision is truly needed.
Rollout stability rests on the fact that the update changes the declarative state, not the live cluster.
If the new image fails readiness checks, the rollout stops on its own, the on-call engineer sees the discrepancy and returns the previous revision with a single command — without manual editing and without searching for “what was there before.”
The result is predictability: at any moment you can see which image is running in each environment, how long delivery took, and how many operations remain manual. This is no longer one-off effort but a managed process with clear control points.
Image Updater operating modes: automatic update options
Image Updater supports two update modes — by mutable tag and by immutable digest. This choice determines what counts as a new image, how the synchronization policy behaves, and whether yesterday’s rollout can be repeated without rebuilding the artifact.
With tag-based updating, the tool compares the value in the manifest with what the registry returns, and when a new build appears it either edits the manifest in git or immediately updates the workload in the cluster — depending on how the synchronization policy is configured.
The digest pins a specific artifact: the tag is not overwritten, so an unambiguous reference to what is actually deployed remains.
| Option | When it fits | Limitations |
|---|---|---|
| Mutable tag | test environments, where the tag always points to the latest build of a branch | no reproducibility guarantee: the tag is overwritten, the old image is lost |
| Immutable digest | production environments and audit, where an exact reference to the artifact is needed | the manifest shows a hash, not a version; a registry that serves the digest is required |
| Manual rollout | infrequent releases and strict change control | delay between build and rollout, human factor |
| Hybrid: auto-update in test environments, digest in production | a typical case where the cost of error differs across environments | separate rules and discipline in tag naming are required |
Auto-updating has a common limit: the tool changes only the image reference.
Editing a configuration or secret will not restart the workload, and a failed start will not roll back on its own — without readiness probes and a well-thought-out synchronization policy, a broken version will reach production and remain in the commit history.
Step-by-step implementation: from repository audit to handover of procedures
ArgoCD Image Updater removes routine work, but an error here shows up not in the build, but in the working environment that has moved to an unverified tag. Therefore, the order of work is fixed: first inventory and an agreed scheme, only then enablement.
- Repository audit. We break down the services: where the manifests are, how tags are published, who owns them. The output is a service map with current strategies and a note on where automation is safe.
- Update scheme. For each service we determine what to subscribe to: a stable tag branch, an immutable digest, or a separate channel per environment. We agree on which environments update themselves and which do so by decision.
- Rule configuration. We write tag selection rules, write permissions for the configuration repository, and behavior when no suitable tag exists. Services where automation is unacceptable are explicitly restricted.
- Scenario testing. We run not only a successful update, but also what breaks in practice: an unavailable registry, a conflict with manual manifest editing, return to a previous image, disabling automation.
- Rollout to the working environment. We enable by service: first non-critical ones, then the rest, checking the application state after each update.
Handover of procedures
We hand over the service map, update scheme, rollback instructions, and instructions for connecting a new service, and conduct team training. After that, services are connected by the customer’s team, and updates remain manageable.
What we hand over: configurations, policies, and procedures
We assemble the artifacts so that image auto-updating does not depend on a single engineer: the setup can be handed over to an on-call shift or a new contractor without verbal explanations. Below is the minimum that ships together with a configured ArgoCD Image Updater.
- Configurations. The update rule manifest: target application, image list, tag filter (we ignore
latest, branches, and pre-release labels), registry polling interval, and how the new version is written to the repository. - Environment scheme. How a version moves dev → stage → prod: where auto-updating is enabled, where changes are only prepared, and which environment requires manual promotion.
- Access policies. Rights of service accounts: read the registry, write only to the directory with the application manifests, without unnecessary permissions in the cluster.
- Rollback procedure. By what signs a version is removed, how the previous tag or immutable digest is recorded, who confirms the return, and how long it takes.
- On-call instructions. Troubleshooting typical failures: the new tag was not picked up, the update was applied and the service did not come up, the repository and cluster state diverged.
- Observability. Which alerts are configured — version drift, failed synchronization, registry authorization failure — and by what signs the on-call engineer understands that the issue is in the update.
- Decision log. Why these filters, intervals, and environment order were chosen: the next edit should not roll back a deliberate compromise.
Case study: how a team eliminated manual releases in a cluster with a hundred services
A cluster with more than a hundred services lived in manual mode: the image tag was edited in a values file, then review, merge, synchronization.
Rollout stretched to hours, a critical defect fix waited in the common queue, and version drift between environments was discovered only after a failure occurred.
They connected Argo CD Image Updater and described rules by service types: tag masks and selection policy, writing the new tag back to git, a limit on concurrent updates so the registry and cluster are not overwhelmed.
For domain services, a two-step scheme was left: first a pre-release tag in the canary environment, then production. Anything that did not update within a given window goes into an alert as a lagging image.
What changed in numbers
| Metric | Before | After |
|---|---|---|
| Share of manual rollouts | about 9 out of 10 | less than 1 out of 10 |
| Time for a fix to reach production | hours, depended on the on-call engineer | 10–20 minutes from image build |
| Incidents due to version drift | regularly, several per month | isolated, caught by alert in advance |
| Return to a previous version | manual manifest edit | one commit revert |
We counted not by feelings but by synchronization history, tag change logs, and incident reviews. People stopped being a link in artifact transfer: the engineer is responsible for update rules and anomaly investigation, not for moving a tag from the registry to the manifest.
How does updating by tag differ from updating by digest?
A tag is a mutable label in the registry; a digest is a hash of the image contents. The update controller that watches a tag periodically polls the registry, compares the received digest with the one recorded in the manifest, and edits the manifest if they differ.
When updating by digest, the manifest immediately contains an immutable reference: overwriting the tag in the registry will not move the rollout.
The difference shows up precisely when the tag changes. A new build was made under the same latest — the previous image is overwritten, and there is nothing left to restore what ran in the previous release.
A rollback by such a tag is unreliable: you will return the label, but not the same set of layers. The digest lacks this drawback — it points to specific layers that live in the registry as long as the retention policy is in effect.
The second axis is application restart. A new digest changes the pod template, so a new set of replicas is rolled out, and the old one is shut down according to the rolling update strategy; restart here is expected and controlled.
If the digest has not changed, there is nothing to restart: the containers continue running on layers already pulled to the node — the update proceeds without a restart.
More dangerous is the reverse scenario: the registry returned a different digest under the same tag, and the application restarted at an unplanned moment.
The practical choice is obvious. For production environments, only immutable references: ArgoCD Image Updater resolves the tag into a digest and writes that into the manifest, automation is preserved, and the rollout becomes reproducible and rollback-capable.
A tag without resolution is appropriate where “the latest image from a branch” is needed: development stands, test runs, demo environments.
Common technical concerns: rollbacks, permissions, synchronization conflicts
Image auto-updating removes manual routine, but engineers immediately have questions about predictability: what happens if the registry is unavailable, how not to affect the production environment, who can stop the automation. Below are short answers to concerns that come up in almost every project with ArgoCD Image Updater.
What happens if the image registry is unavailable?
Deployed applications continue running on the current images, and manifest updates simply wait for connectivity to be restored — new rules are not applied to running services.
We separately limit the number of polling retries so the queue does not grow indefinitely, and set up a notification if the registry stays silent longer than a given threshold.
How do you separate multiple environments so a test build does not go to production?
Each environment has its own update rule: its own tag filter, its own polling frequency, its own set of applications.
In the production environment, automation is either disabled or triggers only on a separate marker tag set by the release environment after verification in the test environment.
Who can stop auto-updating, and how?
We distribute access rights through roles: an engineer enables and disables automation on a specific application without gaining access to the entire infrastructure.
Stopping is done by removing the auto-update label; manual deployment continues to work as usual, so the emergency path remains open.
How do you roll back a version, and what about synchronization conflicts?
Rolling back a version is a regular image edit in the repository: we return the previous tag, and synchronization picks it up without manual actions in the cluster.
If automation and a human edit the same manifest at the same time, we give priority to the version from the repository, and we investigate discrepancies from the log — the conflict does not remain in the working environment.
Let’s discuss your task: we’ll build an auto-update scheme for your environment
<cite> is not needed. We mention ArgoCD Image Updater once. --- ArgoCD Image Updater works on top of the image registry and your manifests, so the accuracy of the assessment depends not on the number of meetings, but on how fully the current environment is described.
We suggest starting with a review: you show how images are currently built and published, and we note where auto-updating will have an immediate effect and where it will require changes in the build process.
To come back with an assessment and plan, we need input data from the customer:
- environment scheme: how many clusters, where test, pre-production, and production are, and how they differ;
- image tagging rules: which tags are considered release tags, whether there are immutable ones, and how hotfix builds are marked;
- access: registry read access, permissions for the manifest repository, and the cluster role for the update service;
- update windows and rollback rules: whether production updates automatically or upon confirmation;
- constraints: stateful services, data schema migrations, components that must not be touched by automation;
- target result: what we consider success — speed of fix delivery, share of manual operations, predictability of rollback.
After the conversation, we assemble an auto-update scheme for your environment and break it down into implementation stages.
Usually this is a pilot on one service in the test environment, then rollout to the rest with separate tag-based policies, then an operations procedure and handover to the team. For each stage, we fix a readiness criterion, so you see progress rather than general assurances. Describe your task — we will come back with an assessment and a plan.