Why AWS deployment breaks as load grows
Manual deployment on AWS holds up exactly until a certain system size.
While there is one service and one engineer does releases, the sequence “built an image, deployed a new task revision, deregistered instances from the load balancer, checked” lives in one’s head and in chat history.
When there are several services and two or more environments, manual release risks cease to be hypothetical: a forgotten step, an environment variable set only in prod, a diverged service health check.
Environment drift between staging and prod is the most common source of “it works for us, but crashes for the client.” Staging is assembled with fewer EC2 instances, different memory and disk limits, the image tag in the registry is not pinned, and schema migrations are run manually.
The error manifests not at deployment time, but an hour later under load, when the cause is sought in application code rather than in configuration differences.
Manual version rollback is the most expensive step of the process. If the previous build is not tagged and described, rollback turns into a random search among dozens of images, and recovery time is no longer measured in minutes.
A separate topic is downtime during deployment: when instances are taken out of the load balancer one by one and updated manually, the unavailability window grows with their number, and some connections break mid-request.
All these failures are the consequence of one thing: release remains a manual operation without a fixed state. Let’s break down how this is closed.
What a managed release delivers: effect in measurable numbers
Time to release a change
Build, checks, and version publication run in a single pipeline: the developer sees the run result a few minutes after submitting code, rather than waiting for a manual deployment “window.” Exactly the artifact that passed checks goes to prod, so the cycle from change to a working version on the site shrinks from days to an hour, and release frequency grows 3–4x — small fixes stop accumulating until a big release.
Share of successful releases on the first try
Settings are moved into environment parameters, and data schema changes run as a separate backward-compatible step, so release does not depend on who tweaked what manually on the server.
The share of deployments after which no urgent follow-up fixes are needed grows from 60–70% to 90% and above — the team spends time on tasks, not on dealing with release consequences.
Time to return to a working version
A new version first receives a small share of traffic: we compare error rate and response time, and only then switch visitors over entirely. If metrics decline, rollback takes minutes — traffic switches to the previous artifact without rebuilding and server-side edits.
Recovery time drops from hours to minutes, and the site again serves working pages while the cause is investigated.
Eliminated class of failures
An entire class of failures disappears: configuration drift between environments, a forgotten step during manual deployment, “it worked in test but not in prod.” Health checks after switching automatically remove unhealthy servers from load balancing in AWS, so deployment predictability stops depending on the attentiveness of a specific engineer.
AWS deployment methods: comparison of modes and limitations
Deployment mode is chosen not by trendiness, but by load profile: how often releases happen, whether there are long-lived connections and background tasks, and how sensitive the user is to cold start. Below are four modes that are realistically chosen among on AWS.
| Option | When it fits | Limitations |
|---|---|---|
| Virtual machines with load balancing | Long-lived processes, custom agents and daemons, steady 24/7 load | Fleet is updated manually, configurations drift between nodes, state control is required |
| Container deployment | Multiple services with a shared network, frequent releases, reproducible environment | Requires an image registry and service description, an extra layer for a single monolith |
| Serverless execution | Event-driven and infrequent tasks, traffic bursts, queue and file processing | Execution time limit, cold start, dependence on the provider's environment |
| Static site and content delivery network | Storefronts, documentation, interfaces with an external API | No server-side logic, cache requires explicit invalidation on release |
How we choose the mode
We start from the nature of requests and how the team rolls back a release.
If a service holds a connection to the database and takes minutes to warm up — virtual machines or containers; if requests are short and infrequent — serverless execution is cheaper in downtime and requires no on-call duty.
For static content, deployment comes down to uploading artifacts and serving them through a content delivery network.
Why modes are often combined
A single project can live in a hybrid scheme: interface as static content, API in containers, background tasks serverless. We describe infrastructure as configuration, so switching between modes does not turn into rewriting the project from scratch.
Work stages: audit, environment scheme, handover
We start work by capturing the actual state: without this, any deployment scheme is built on guesses. Then we proceed sequentially, and each stage ends with an artifact — a file, a diagram, or a measurement that can be verified.
-
Infrastructure audit. We examine what is already deployed in the account: which services are used, where deployment is performed manually, which dependencies between components are not described anywhere. The output is a list of bottlenecks that make releases risky today.
-
Environment scheme. We separate the test environment and production, define the resource composition in each and the rules for promoting a build between them. We separately discuss which data must not enter test environments.
-
Infrastructure description. We move settings into declarative files in the repository: the environment is deployed from a description rather than assembled manually. Configuration is versioned, and drift between environments becomes visible.
-
Deployment setup. We build the artifact, deliver it to target servers, switch traffic to the new version without downtime, and distribute load between instances. The readiness criterion is a repeatable deployment with a single command.
-
Rollback verification. We practice returning to the previous version in advance, not at the moment of an incident: we measure rollback time, check data schema compatibility, and verify user session preservation.
-
Handover to operations. We provide the infrastructure description, environment scheme, deployment and rollback procedures, access, and alerting setup. The customer's team performs a release themselves under the supervision of our engineers — this is the acceptance.
What we hand over: configurations, schemes, and procedures
We hand over not an archive of files, but a working system: using these materials, the team deploys environments from scratch, ships a release, and rolls it back without our participation. Everything is in a single repository and goes through review.
- Infrastructure as code — descriptions of networks, load balancing, nodes, and storage in declarative files. The staging environment comes up with a single command, and drift between environments disappears.
- Environment scheme — which environments exist (development, pre-production, production), how they differ in permissions, data, and number of nodes, and how configuration is injected at startup.
- Access policies and roles — what is allowed to the build service, what is allowed to the application in operation, what is allowed to the on-call engineer. Keys are not stored in code but kept with rotation.
- Release procedure — steps from build to traffic switching, mandatory checks, and criteria under which a release is considered successful.
- Rollback procedure — how to return to the previous version, how long it takes, and what to do with already-applied data schema migrations.
- Infrastructure diagram on one page — traffic flows, service connections, failure points, and external dependencies.
- Documentation for the team — instructions for typical tasks and incidents, service map, owners, and escalation procedure.
After that, the team works independently — we stay in touch, but not as the only ones who understand how your AWS deployment is structured.
Case: zero-downtime releases on a high-load service
A service with tens of thousands of concurrent sessions lived on two instances in AWS: release meant connecting to nodes, restarting processes on live ones, migrations by hand.
A release took about 40 minutes and ran only in a night window once every two weeks; rollback took another 25 minutes according to the instructions.
Every second release caused an error spike: some nodes served the old version, some the new one, and cache and database schema drift persisted until the next restart.
What we changed
We packaged the build into immutable artifacts, and moved deployment out of the instances into a separate environment. The load balancer switches traffic to the new target group only after health checks pass; the old one remains hot until metrics are confirmed.
Rollback is reverse traffic switching, not a rebuild. Database schema changes run as a separate step and must be compatible with the previous code version.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| One release | ~40 minutes in a night window | ~8 minutes during working hours |
| Rollback | 25 minutes manually | less than a minute by switching |
| Release frequency | 2 per month | 6–8 per week |
| Deployments without manual intervention | 0% | all, including rollbacks |
Configuration drift between nodes, config edits in production, and restart downtime have left the failure classes. Errors in the code itself are now caught by a canary group on 5% of traffic — before full deployment.
How containers differ from serverless execution
A container keeps a process running: the environment initializes once, then a warmed-up application serves requests. A function lives only for the duration of a call — the platform starts it for a specific event and shuts it down after processing. Hence the differences in cold start, predictability, and behavior under load.
A function experiences cold start after idle time or a sharp increase in concurrent calls: time goes into bringing up the environment, loading dependencies, and establishing database connections. The heavier the environment, the more noticeable the pause on the first request.
With a container, this effect appears only when a new version is released — the rest of the time latency is steady.
Predictability and debugging also diverge. A container is an ordinary process: you connect to it, take a profile, reproduce a scenario locally, and see the same picture as on the server.
A function does not preserve state between calls, instances are isolated, and analysis is done through logs — some problems are caught only under real load.
Scaling boundaries differ between the approaches. Functions scale out for each call on their own and almost immediately, but hit a ceiling of concurrent executions and a processing duration limit.
Containers grow by specified rules and require capacity planning, but they comfortably handle a constant request flow, long connections, and background tasks.
The choice is simple: steady constant load, long-lived connections, latency sensitivity — containers; infrequent peaks, event processing, scheduled tasks — functions.
The Amazon Web Services platform supports both modes, and on a single project they often coexist: the core in containers, the surrounding tooling on functions. We look at the load profile and only then fix the scheme.
Common technical questions about releases and environments
We pin the application version in an immutable artifact, and keep environment parameters separate from code. Then staging and prod differ only in settings, and deployment and rollback become a predictable operation rather than a one-off task for an engineer.
How to roll out an update without downtime?
We bring up new instances alongside the old ones and switch traffic to them only after health checks. If a check fails, traffic remains on the previous version, and the user notices nothing.
Where to store secrets and access keys?
We keep them in a managed secrets store with role-based access. The application retrieves them at startup; they do not enter the repository or build parameters, and rotation does not require rebuilding the image.
Staging and prod — separate or keep as one environment?
We build both from the same artifact, but data and networks are separate for each. Configuration drift is the main reason why “it worked in staging but not in prod,” so configurations are versioned together with code.
How do we know a release went as planned?
Immediately after rollout, we look at error rate, response time, and application logs, and run key scenarios in the live environment. Alerts arrive earlier than user reports.
How quickly can we return to the previous version?
Artifacts are versioned and immutable, so rollback is switching traffic to the previous image, not rebuilding. We write schema migrations compatible with both versions so data is not lost.
Let's discuss the task and come back with an estimate and plan
To estimate AWS deployment, we need not a retelling of the task, but the actual state of infrastructure and processes.
Below is the minimum set of inputs from which we assemble the scope of work, implementation timeline, and handover plan, rather than coming back with clarifying questions a week later.
- Environment scheme: what already runs on EC2, ECS, or Lambda, what remains outside the cloud, how services are connected to each other — network, load balancers, databases, queues.
- Current release: how the build reaches prod, which steps are done manually, where the process most often breaks, and how long rollout takes.
- Environments: how many environments are needed and how they differ in data and access, whether separate staging environments are needed for task verification.
- Access and secrets: who has access to production, where keys and credentials are stored, how they are issued and revoked.
- Reliability requirements: acceptable downtime during release, rollback depth, what is considered an unacceptable failure.
- Constraints: security regulations, release windows, data retention rules, and external dependencies.
- Who remains with the system: composition of your team, whether documentation handover and post-implementation support are needed.
Based on these inputs, we prepare an estimate and work plan broken down by stages — the first changes in the release pipeline are visible long before the final migration. Describe the task — we will come back with an estimate, scope of work, and timeline.
