Deploying LLM on AWS: where production breaks
A pilot on managed model inference in AWS sustains a flow of several test scenarios, but once you open access to real users, the first bottleneck in request servicing appears not at the model but at the ingress.
Requests pile up in queues, autoscaling based on average load cannot keep up with stepwise traffic growth, and the service falls under load before resources run out.
Then model response degradation kicks in: as the number of parallel generations grows, time to first token increases, client retries double the load, and the system enters a storm.
Average response time still looks acceptable — the tails fail, and it is they that decide whether the user returns.
This is where the cost of system downtime becomes visible: it is measured not in machine downtime hours but in the share of failed requests and the team time spent on manual triage.
The third failure point is a shared environment: test, preview, and prod live on the same cluster, in the same quota, and with the same set of access permissions.
A prompt edit or model change affects prod, and there is nothing to roll back to: the previous configuration is not recorded anywhere.
Deploying LLM on AWS is primarily about working with these failure modes, not about choosing a service. Below we break down how this is addressed: concurrency limits, autoscaling by queue length, separate environments, and predictable rollback.
What changes after moving models to a managed environment
A managed model service removes the most expensive layer from the team: operating inference environments. Instead of a fleet of machines with manual assembly of dependencies, drivers, and a scheduler, there remains a single inference service that scales under load on its own.
Configuration is described declaratively and reproduced identically in staging and prod — engineers no longer depend on a specific runtime build.
Model response time stops being a surprise. Autoscaling by queue depth levels latency at peak: with a tenfold increase in requests, p95 grows by tens of percent, and cold start is mitigated by scheduled instance warm-up.
This also changes product decisions — synchronous scenarios where the user waits for a response start to make sense.
Model version control becomes as routine as code version control. Each set of weights is assigned an immutable identifier, inference parameters are stored alongside it, and rollout is switching traffic between variants while preserving the previous one. Rollback takes minutes instead of hours of searching for the right image.
A unified access layer closes a separate class of failures: secrets, data permissions, and network rules are described in one place and inherited by all staging environments.
The “it works on my machine” discrepancy stops being a source of incidents, and recovery after a failure is measured in minutes — environment state is reproduced from the description rather than assembled from memory.
Hosting LLM on AWS: options and limitations
The platform for a model is chosen not by “power” but by load profile: a steady request flow, rare peaks, or strict data isolation requirements. It also matters who is responsible for updates and scaling — the provider or your team. Below are four options that are realistically chosen between, and what limits each one.
| Option | When it fits | Limitations |
|---|---|---|
| Managed model service | Need control over the image, weights, and inference parameters; load is predictable | Capacity sits idle between peaks, autoscaling is configured manually, tied to a chosen machine class |
| Serverless access to models | Rare or uneven calls; need a fast start without your own infrastructure | No access to weights or fine-tuning, strict limits on request size and number of calls |
| Your own virtual machines | Non-standard models, custom builds, full control over the stack and data | All administration is on you: drivers, updates, queues, fault tolerance, warm-up |
| Hybrid environment | Some data cannot leave the perimeter, but elasticity for peaks is needed | More complex request routing and model version synchronization between segments |
Limitations usually appear not at the start but in the second or third week of operation: either the call quota hits a ceiling, or expensive capacity sits idle, or there are insufficient rights for fine-tuning.
A hybrid environment removes some of these constraints but requires mature routing and a single model version in both segments. Therefore we finalize the choice after measuring the request profile and data requirements, not by formal criteria.
How implementation proceeds: from load audit to handover
The sequence of work is the same for managed services and for deployment on your own instances — only the set of settings changes.
First we capture the request profile, then design the architecture, and only then deploy: without measurements, configuration is selected by eye, and the discrepancy with reality surfaces already in production operation.
- Load audit. We count requests per hour, peak windows, context length, and response time requirements. The output is a capacity calculation with headroom for peak and the boundaries beyond which answer quality drops.
- Environment design. We choose the placement architecture — managed service, dedicated instances, or hybrid: where model weights are stored, how roles are divided between nodes and environments.
- Environments and access. We separate development, acceptance, and production environments, configure roles and keys, network restrictions, data encryption, and logging of model calls.
- Load testing. We run a flow close to peak: response time under load, behavior when a node fails, cold start speed, resource consumption.
- Optimization. We enable caching of repeated requests and batch processing, and select the model size for the task: a smaller one gives the same accuracy with more predictable latency.
- Production rollout. We switch traffic in stages, monitor metrics, and keep rollback to the previous configuration ready — a single command returns the system to a working state.
- Handover to operations. We provide regulations, environment diagrams, monitoring dashboards, and instructions for the on-call shift, train the team, and stay in touch during the first period of operation.
What we hand over: diagrams, policies, runbooks
We consider LLM deployment on AWS delivered not when the model answers a test request, but when the environment lives without us: it can be raised from scratch, the model updated, a release rolled back, and an incident triaged at night according to instructions.
Therefore, along with the working environment, the customer receives a set of artifacts that their team already uses to operate.
- Environment diagram — networks and private subnets, egress points, load balancing, storage, and the request queue; separately marked are where model weights reside, where logs are, and which nodes are critical in a failure.
- Environment description — test, pre-production, and production environments, their differences in configuration, limits, and quotas, as well as the path a change takes to production.
- Access policies — roles and permissions: who changes configuration, who sees logs and request content, how keys are issued and revoked, and what happens when an employee leaves.
- Model update procedure — how a new version is rolled out, what share of requests goes to it first, by what metrics it is accepted, and how to return to the previous one in a single step.
- On-call shift instructions — typical failures and their signs, what to check first, what actions are safe without a developer, and at what point to engage the second line.
- Solution documentation — architectural decisions made with rationale, dependencies, data schema, known model limitations, and load profile.
- Recovery plan — backups of configurations and storage, target recovery time, and the procedure for checking the environment after an incident.
The documents live alongside the infrastructure description: when the environment changes, they are edited in the same change, not as a separate “someday” task. We hand them over with a walkthrough on a call and leave the right of edits with your team.
Case study: moving model load without service downtime
The service ran on a single deployed model copy: traffic grew, response time at peaks tripled, and releasing a new version meant stopping for manual switching.
A planned replacement took about an hour of nighttime work — and any failure at that moment turned into a complete outage because there was no spare copy.
What we rebuilt
Inference was moved to a separate layer: several replicas behind a load balancer instead of a single point, model versions as separate entities in a registry with labels for production and trial environments.
Traffic migration was made canary: first a small share of requests goes to the new version, then the entire load.
Rollback was tied to metrics: if the error rate or response time goes beyond the threshold, the production label automatically returns to the previous version, without an on-call engineer.
We deploy on a managed Amazon Web Services environment — the platform takes on the nodes, while configuration, traffic routing, and quality control remain with us.
What changed in numbers
Response time under peak was reduced by about three times, and at the average level it stopped depending on influx of requests.
Recovery time after a failed release went from an hour of manual operations to a few minutes: rollback is now automatic and does not require a night shift.
A whole class of failures disappeared: releasing a new version no longer stops the service, and a single replica failure does not reach the user — load is redistributed among the others. Peak hours stopped being a risk, and the team releases improvements during working hours.
How does a managed service differ from your own instances?
The question is not who brings up a model faster, but who is responsible for the layer beneath it. On AWS this is a choice between Bedrock, SageMaker, and your own environment on EC2.
A managed service is rented responsibility for the execution environment and fault tolerance; your own instances are full control with a full set of duties.
A managed service provides a ready access point with autoscaling and replacement of failed nodes — the team focuses on prompts, answer quality, and access.
On your own instances, you maintain drivers, quantization, and environment versions yourself, but you can deploy a non-standard model and your own inference optimizations.
A separate fork is data isolation. In the provider's shared environment, you are limited by its policies: data is not used for training, but it resides next to others' workloads. Your own instances in a private subnet give control over network paths, logs, and encryption.
| Criterion | Managed service | Your own instances |
|---|---|---|
| Responsibility zone | Provider — hardware, environment, updates; you — data, access, quality | You — the entire stack, including patches and node replacement |
| Configuration flexibility | Limited by service parameters | Any model, quantization, your own inference optimizations |
| Data isolation | Shared environment under provider policies | Private subnet, your own network paths and logs |
| Team effort | A few hours per week for quality control | Ongoing load for operations and capacity |
In practice, the choice is not “one or the other.” Sensitive scenarios are moved to your own instances, mass traffic to a managed service: team effort stays within reasonable bounds, and flexibility is preserved where it is needed.
Common technical doubts before launch
Before the project starts, customers raise almost the same set of questions — and the architecture depends on the answers, not the wording of the contract.
Below are five technical doubts that we resolve before the first line of configuration appears: behavior under load, the fate of an already running environment, responsibility for model versions, data access, and the migration procedure.
We address them substantively, because each answer changes the work plan: in some places a parallel staging environment is needed, in others fixing the model version in the scenario configuration, and in others it is enough to separate roles and enable request logging.
Will the response withstand load growth?
First we capture the profile of real requests: input and output length, share of repeated calls, peak hours. Then we run this profile on a load-testing staging environment and see where degradation begins.
Caching of repeated prompts, an application-side queue, and worker autoscaling usually remove the peak without revising the architecture, but the decision is made based on measurement numbers, not a “by eye” margin.
What to do with an already deployed environment — rewrite everything?
No. We start with an inventory: which scenarios already work, where requests go, what integrations exist, and how data is stored.
We bring up the new environment in parallel, switch traffic scenario by scenario, and leave a fast path back — returning to the previous call is possible with a single configuration edit, not by rolling back the entire release.
Who is responsible for model versions and their updates?
The model version is fixed in the scenario configuration and comes under change control — like any other artifact.
An update is run against a set of reference requests, the answers are compared with the current ones, and only then is the version changed in the production environment. The provider may retire a model — so a replacement is prepared in advance and checked against the same set.
How is data access restricted?
Roles are separated by scenario and environment, model calls are logged, and sensitive fields are masked before sending. Data is not used to further train external models — this is enforced by settings and contract, not a verbal agreement. Environment isolation prevents test requests from entering production storage.
How is an existing solution migrated without downtime?
The procedure is: inventory of scenarios and integrations, transfer of settings and sets of reference requests, testing on shadow traffic, then gradual switching.
Downtime is not required — the old and new environments run simultaneously, and discrepancies in answers are visible in the comparison before the switch.
Let's discuss the task and come back with a plan and estimate
The conversation starts not with a presentation but with your task: what the model should do, what data it answers on, what environment is acceptable, and what already works in the infrastructure.
We do not start with choosing a service — first we fix the load profile, requirements for answer quality, and the data handling regime, and only then decide whether the scenario fits into managed services or requires a separate environment.
After discussing the task, we send:
- phased implementation plan — from a pilot on a narrow scenario to industrial load, with the boundary of each phase;
- timeline estimate and the point after which work can be stopped without losing the result;
- placement architecture: what goes to a managed service, what remains on your own nodes, where the data boundary runs;
- requirements for access, network, and storage — what will be needed from your team;
- resource profile for peak load and scaling rules;
- list of risks: answer quality on your data, behavior under peak, rollback to the previous model version.
The next steps are a walkthrough with the engineer who will lead the project. Describe the task in an email or on a call, and we will come back with an LLM deployment plan on AWS in your environment and a timeline estimate.
