Why AWS ECR setup breaks in production
AWS ECR setup usually goes smoothly while a single engineer from a single machine is the only one touching the container image registry.
As soon as build agents, developer workstations, and contractors connect to it, drift begins: one person's image is pushed manually under the latest tag, another's is built from uncommitted code, and production ends up running an artifact that cannot be reproduced.
Formally, the registry works; in reality, no one can say which commit the current release was built from.
Manual image publishing breaks reproducibility first of all. While an image is built on a laptop and sent with a single command, there remains a window between “built” and “pushed” in which the environment, dependencies, and environment variables can drift apart.
Rollback to the previous release in such a scheme turns into a search — the required image may simply not be in the registry because it was overwritten.
Registry access permissions are the second source of failures. Policies granted broadly, “so as not to get in the way of work,” give write permission to everyone who should only read: tags get overwritten by others’ builds, and there is no way to sort it out after the fact.
The opposite extreme is access granted manually per person: a new agent or contractor waits a week for a key, and the team returns to manual publishing.
The third pain point is image storage overflow. Without lifecycle rules, every tag drags along layers, registry volume grows linearly, and at some point a build fails during push — the release stalls at the most inconvenient moment.
We explain how this is addressed: where permission boundaries are set, how publishing is automated, and what cleans the registry without losing the ability to roll back.
What a managed image registry in ECR provides
A managed registry is not an image warehouse but a perimeter with access and retention policies. IAM roles separate pull for runtime environments and push for the build pipeline, so the build agent physically cannot overwrite a tag that has already been deployed to production.
Additionally, we enable tag immutability: the artifact under a given tag remains the same, and an attempt to republish is rejected at the Amazon ECR level.
A lifecycle rule keeps the last few images per tag and purges the rest by age — the registry stops growing linearly, but rollback to the previous release remains available.
A separate rule for intermediate builds deletes them after a day so development branches do not take up space on par with releases. As a result, storage grows in proportion to the number of releases, not the number of commits.
Deployment automation removes manual steps: build, tagging by commit ID, publishing, updating the service description. Deployment time drops from tens of minutes — half of which is coordinating tags and switching by hand — to several minutes of a predictable process.
Rollback becomes switching to the previous immutable tag, so recovery time after a failure is measured in minutes and does not depend on which engineer is on call.
Build reproducibility closes a separate class of failures — artifact drift between environments.
The same image by digest passes checks in the staging environment and reaches production, rather than being rebuilt “from roughly the same code.” The share of manual operations falls, and along with it go incidents caused by human error during release.
Image publishing options: comparison of approaches
The publishing method determines what ends up in the staging environment on the next rollout and whether it will be possible to return to the previous state.
A tag is convenient as a human-readable reference, but when overwritten, two different states of the system point to the same name — the builds can no longer be compared with each other.
A reference to an immutable identifier ties the rollout to specific image content, so rollback is reproducible.
The second choice is a single registry or separation by environment. In a shared registry, access rights and retention rules are separated by name prefixes, and an error in one rule affects all perimeters at once.
Separate registries provide independent cleanup policies: in the production perimeter, tag overwriting is prohibited; in test ones, excess is aggressively trimmed.
| Option | When it fits | Limitations |
|---|---|---|
| Publishing by tag | Manual runs, quick checks | Tag overwriting breaks reproducibility and build comparison |
| Publishing by immutable identifier | Production rollout, audit, rollback | Not human-readable, requires discipline in the build pipeline |
| Single registry for all environments | One perimeter, small team | Shared rights and cleanup rules; tests affect the production perimeter |
| Registries separated by environment | Multiple perimeters, access isolation requirements | Image duplication, harder to synchronize rules |
| Automatic cleanup policy | Regular releases, many branches | Without precise configuration, an image needed for rollback gets deleted |
In practice, the options are combined: the production perimeter is published only by immutable identifier, registries are separated by environment, and the cleanup policy keeps the latest builds for each tag. The registry stops growing uncontrollably, and rollback to the previous version remains available.
How we structure implementation: stages of work
We start with an inventory rather than a clean slate: an AWS ECR registry usually already exists, but permissions are granted with excess, images accumulate without limits, and access keys sit in build variables.
We bring this to a scheme that the client’s team maintains on its own — the sequence of stages is the same for a new perimeter and for sorting out an inherited one.
-
Audit of access rights and the current scheme. We look at which repositories and tags actually exist, who publishes images, and where credentials come from. The output is a map of excessive privileges, duplicates, and places without rotation.
-
Designing the registry scheme. We split storage by environments and teams and fix image and tag naming rules. The scheme remains in the infrastructure description, not in manual console settings.
-
Configuring access policies. Roles with least-privilege rights: write — only from the build pipeline, read — selectively for staging environments. We replace long-lived keys with temporary credentials with rotation.
-
Image retention policies. Auto-deletion of old builds while preserving the latest release tags, vulnerability scanning, protection of release tags from overwriting. The registry stops growing, and rollback to the previous version remains available.
-
Verification in the test perimeter. We run publication, rollout, rollback, and reading an image from an adjacent environment. We make sure permissions are no broader than needed and that writes from another account are rejected.
-
Infrastructure handover. We deliver the description of the registry, roles, and policies as code plus a regulation for working with tags. We walk the team through typical operations: a new repository, contractor access, release rollback.
Documentation package, policies, and registry schemes
After implementation, the team is left not with a set of commands from chat history but with a working package — configurations, policies, and documents by which a new engineer can repeat the procedure without the author. All artifacts are versioned in the repository and go through review just like code.
- IAM roles and access policies — permissions based on least privilege: publishing from the pipeline, reading from runtime environments, and administrative operations are separated into different roles, plus explicit boundaries between projects.
- Image naming scheme — a repository per service, an immutable tag per release, separate tags for verification environments; the name immediately shows what is deployed and from where.
- Cleanup regulation — lifecycle rules with thresholds: how many latest images to keep per tag, when to delete intermediate builds, and how often to review the thresholds.
- Registry configuration as code — repositories, upload checks, and encryption are described by files and applied automatically, without manual edits in the console.
- Instructions for engineers — how to publish an image, how to roll back to the previous release, what to do if publication fails, and how to request extended permissions.
- Diagram of links with the pipeline and environments — where the image is built, where it is published, who reads it, and what checks run at each step.
- Review procedure — who reviews policies and how often, so permissions do not accumulate and retention periods meet requirements.
We hand over the package to your repository with a short walkthrough — the on-call engineer will understand the registry without contacting us.
Case study: reduced deployment time and removed manual steps
The client is a product team that built images and published them to the AWS ECR registry manually: an engineer tagged the build, pushed it, and edited the environment configuration before every release.
At 12–14 publications per week, this resulted in up to 14 manual actions in the release cycle — and almost every second deviation in production was related to the wrong tag being sent to the environment.
What it was before
Publication took 25–30 minutes: build, waiting for upload, manual check that the image actually made it into the registry.
The registry grew without rules — images accumulated under all tags, including intermediate branch builds, and finding the current artifact turned into trial and error. Rollback to the previous build took up to an hour because the required image was not at hand.
What we changed and what the result was
We separated permissions: the build agent got only publishing to specific repositories, the engineer — read and rollback, production — only pulling the image.
Lifecycle rules were tied to the tag scheme: immutable release tags are stored for a long time, intermediate builds are cleaned up by count and age. Publishing was built into the pipeline — the tag is assigned automatically, and a human only confirms the release.
Publication was reduced to 5–7 minutes, and two manual operations remained in the release: confirmation and metric checks. Registry volume decreased threefold while fully preserving release history, and rollback is performed by returning the previous tag in a couple of minutes. The share of releases rolled back due to an incorrect artifact went to zero.
How publishing by tag differs from publishing by hash
A tag is a mutable reference to a manifest: app:release points to one build today, and tomorrow the pipeline will reassign it to another.
A tag by itself does not guarantee that the same code as yesterday will be deployed from it — this is where drift between the staging environment and the production perimeter comes from.
The hash (digest) of a manifest is calculated from the image content, so it cannot be rewritten unnoticed: the same identifier always contains the same layers and the same configuration.
Artifact immutability is ensured precisely by content addressing, not by a promise not to reuse a tag.
The difference becomes clear during release rollback. Deployment by hash is a deterministic operation: we return the previously pinned identifier and get exactly the image that was running before the incident, in one action and without guesswork.
With tags, rollback turns into an investigation: you need to remember which tag pointed to what and hope it was not reassigned or deleted during cleanup.
The working scheme is this: the build happens once, the image gets a human-readable tag for navigation, and the hash is pinned in the deployment manifest. The hash goes to the production perimeter and staging environments; the tag remains a label for people.
If tag immutability is enabled in the repository, overwriting a published identifier is blocked at the registry level, and accidental version substitution is cut off technically.
Thanks to this, the environment is reproducible: from the deployment manifest, the same version can be brought up at any time and in any staging environment, and the old image for testing a hypothesis is found by hash even if its tag moved long ago.
FAQ on access and image lifecycle
Access and registry cleanup are feared for “breaking a working release.” In practice, both operations are introduced without downtime: a lifecycle rule is first dry-run in preview — you can see which tags will be deleted — and roles are granted separately for read and write.
The key fork is by what criterion to clean and how to grant permissions without exposing anything extra.
We mark release tags as immutable, so only intermediate builds and untagged images older than a specified period are deleted.
Separate repositories per service prevent one team from affecting another team’s image, and the cap on the number of images in a repository stops being a problem: new build writes go through because old ones are freed automatically.
If we enable cleanup, will it delete an image needed for rollback?
No, if the rule is based on tags rather than date. The “keep the last N images per each tag” scheme preserves previous releases, and only intermediate builds and untagged images that no deployment references are deleted.
How do we grant access to external contractors without handing over keys to the entire registry?
A role for reading images plus permission to pull them from a specific repository — without rights to delete and modify.
For access from another account, we add a repository-level policy: the contractor connects with their own account, no keys need to be handed over, and access is revoked with one line.
What will happen to an already running deployment?
Nothing: running services continue to operate from the local cache on the nodes. Cleanup and permission changes affect only the registry, so they are introduced as a separate step, not together with a release.
Let’s discuss the task and come back with an estimate and plan
To come back with an estimate and plan rather than generalities, the first conversation needs a minimum of inputs: how the build is structured now, who publishes images, and which environments are supported. It is enough to describe the task in your own words — we will pull out the rest with questions.
Before estimating timelines, we clarify:
- how many services and repositories are in the registry, who publishes images, and how often;
- how access is currently granted: personal accounts, shared keys, or roles with environment-scoped permissions;
- how the build pipeline is structured: what you build with, when the image enters the registry, and whether there are intermediate staging environments;
- retention requirements: how many image generations to keep, whether immutable tags are needed for releases, and how often to clean;
- security policy: vulnerability scanning, prohibition of publishing from local machines, audit log, and separation of rights between teams;
- rollback scenarios: how quickly the previous version needs to be restored and who does it;
- constraints: internal regulations, maintenance windows, and who accepts the result on your side.
Based on the answers, we assemble the scope of work for the registry, break it into stages, and state implementation timelines — separately what our engineers do and separately what remains on your side. That way, no “unexpected” items appear in the middle of the project.
Describe the task — we will come back with an estimate and plan.