Deploying LLMs on Azure: OpenAI or Machine Learning Endpoints

Deploying LLM on Azure: bottlenecks and risks

While model inference lives in a demo environment, the bottlenecks of an LLM service are not visible.

Deploying LLM on Azure changes the picture: quotas are issued in tokens and requests per minute separately for each deployment, and when they are exceeded, the service does not slow down smoothly but starts returning failures.

Externally this looks like "the model broke," although the model responds correctly — the request limit has been exhausted.

Degradation under peak load develops nonlinearly. The queue on the application side grows, retries multiply the number of calls, and a single request with a long context can consume the entire per-minute quota of a deployment.

Response time increases not proportionally to the number of users: only yesterday the scenario responded in fractions of a second, and at peak half of the requests end in failure — and the user sees not an error but a frozen interface.

The model inference architecture determines what exactly the service will pay with at peak.

Without load distribution between deployments, without limiting response length and without a cache of repeated requests, a failure takes down the entire scenario, not just a separate part of it.

Add several models for different tasks — and the quota begins to be divided between them instead of working. What breaks first and in what order — we analyze how this is mitigated.

What changes after moving the load to a managed environment

A managed environment changes not the model itself, but the way its compute part is handled. Previously, nodes were reserved for load with headroom for peak — they sat idle most of the day, and during a burst of requests the service still hit the ceiling.

Now the load profile is described by scaling rules based on queue length and number of active requests: replicas are added at peak and scaled down during quiet hours.

Time to put the model into operation is reduced from weeks of infrastructure approvals to hours.

Deployment in Azure Machine Learning is registering a model version, selecting a compute node class, configuring an endpoint and running a validation set; network, secret storage, logs and metrics come with the environment.

Fault tolerance of the model service rests on replicas across multiple availability zones and health probes.

A faulty instance is automatically removed from rotation, and request rate limits at the deployment level prevent one consumer from exhausting the quota for others — a cascading failure does not spread to the entire service.

Service downtime during updates goes into a planned window: the new version is brought up as a second deployment, traffic is shifted by weights, rollback is by switching back.

We track time to put into operation, share of requests without failures, share meeting target response time, and scaling trigger frequency — this is the measurable result of moving the load.

Deployment modes: managed service and custom endpoints

Choosing the deployment mode for LLM on Azure determines not only the model response speed, but also how the service survives peaks.

A managed service removes infrastructure from the team, but operates within shared quotas per region: during a load burst, some requests are rejected, and the application must be able to queue and retry them.

Dedicated endpoints provide predictable throughput, but only if capacity is reserved in advance and for a specific request profile.

What we compare

In practice, the scheme is almost always hybrid: conversational scenarios with bursty traffic go to the managed service, while batch processing and fine-tuned models go to dedicated endpoints.

We calculate the load profile, quota headroom and isolation requirements, then fix the mode for each scenario and the rules for switching between them — so that limits do not turn into user-facing failures.

Implementation order: from load audit to environment handover

Deploying LLM on Azure starts with a load audit, not with creating resources: we capture input context length, share of streaming responses, peak requests per second. These numbers determine the choice between Azure OpenAI and Azure Machine Learning Endpoints and quota headroom. Then work proceeds in stages.

1. Load audit: measure real scenarios on a pilot — peak requests, input and output length, share of repeated questions. Output: throughput calculation and list of bottlenecks.
2. Environment design: fix regions, private endpoints, roles and key storage in a secure vault. We agree on the design before configuration, not after an incident.
3. Configuration and protection: rate limiters, request size limits, retries with increasing delay, cache of typical responses, logs without personal data.
4. Fault tolerance testing: simulate region failure and quota exhaustion. The environment switches to a backup deployment or a lighter model, and the user gets a clear response instead of a timeout.
5. Load run: drive traffic according to the profile from the audit — stepwise growth and long peaks. We look at time to first token, 95th percentile latency, error rate and quota consumption.
6. Handover to operations: deliver the on-call procedure, dashboards with alerts, scaling rules, rollback plan to the previous configuration and training for the customer's team.

Engineering artifacts: diagrams, access policies, procedures

Deploying LLM on Azure is handed over not as a working service, but as a kit by which it can be reproduced, extended and transferred to another team without the author.

Everything listed is stored in the customer's repository with change history, not in an archive of correspondence.

• Deployment configurations — declarative files with parameters of the selected option: endpoint type, model placement, quotas, request length limits, timeouts and scaling rules.
• Access policies and roles — a "who can do what" matrix: managed service identities instead of keys in code, roles at the resource and group level, access rules for the secret vault and logs.
• Integration diagrams — the request path from application to model: routing, response cache, error and retry handling, connections to the corporate catalog and accounting systems.
• Model update procedure — where quality checks are run before releasing a new version, who approves, how traffic is switched and how to return to the previous version.
• Observability and rollback — latency and failure rate metrics, alert thresholds, monitoring panel for the on-call engineer and a step-by-step recovery scenario during degradation.
• Environment documentation — architecture diagram, environment variables, analysis of typical incidents, registry of dependencies and external services.

The kit reduces onboarding of a new engineer into the environment from weeks to days: decisions on access, versions and rollback are already fixed, and they do not have to be reconstructed from code.

Case: eliminated drops under peak model load

What was before the changes

The client is a support assistant service where response generation goes through a model deployed in Azure.

Both interactive operator requests and nightly batch processing of tickets went through a single deployment with a shared quota: while the load was steady, response time stayed within 8–10 seconds, but during peak hours requests queued up in one line.

At peak, the median grew to 25 seconds, and the upper percentile to 46. Some requests dropped with an overload error, and the share of such failures reached 7% during evening hours. Operators restarted requests manually, and the nightly batch was not completed by morning.

What was changed and what was achieved

The load was separated by modes: interactive traffic moved to a deployment with dedicated throughput and peak headroom, batch tasks — to an on-demand deployment that scales itself and does not compete for quota.

Routing selects the mode by task type, retries with increasing pause and a context length limit were added, so that one heavy request does not consume the window for others.

The failure class "queue on shared limit" stopped appearing: overload of one mode no longer blocks another. The application code was not rewritten — the deployment scheme and routing rules between modes changed.

How is a managed service different from a dedicated environment?

The fork runs along one line: who owns the model weights and who controls scaling under load. In a managed service on Azure, the model, its updates and compute resources remain on the platform side — you connect to a ready endpoint and rent throughput.

In a dedicated environment, the weights are deployed in your subscription, and you determine the model version, pool size and update timing.

Comparison of modes comes down to four axes: control over model version, network isolation requirements, behavior at peaks and responsibility for inference. A managed service wins where a quick start is needed — you do not have to select configuration for the model.

Limitations appear later: the shared pool distributes requests by its own rules, priority for your profile cannot be configured, and access from a private network is limited by platform boundaries.

A dedicated environment is deployment in a private network with its own endpoint, autoscaling by your metrics and rollback to the previous version in one step.

The price of control is measurable: idle replicas, independent image maintenance, full responsibility for capacity at peaks.

In practice, what wins more often is not one mode but a hybrid scheme with load separation: scenarios with sensitive data go to the dedicated environment, public and bursty ones — to the shared service.

Separation is done at the gateway: unified routing, shared access policies and end-to-end logging — otherwise observability and access rules diverge between environments.

Technical doubts and platform limitations

Questions surface not on the prototype, but when the solution is brought into production operation: will the environment survive a migration, what happens when the model version changes, who is responsible for access. Below are short answers for scenarios that most often slow down approval.

What happens when the model is updated?

The new version is deployed as a separate deployment next to the old one. We run the reference set of prompts, compare quality, latency and token consumption, then switch traffic in increments; rollback is instant.

How available is the service and what about limits?

We place the deployment in two regions and reserve throughput in advance. When the limit is exhausted, requests queue with degradation instead of failure, and liveness probes move traffic to the backup region.

How to separate access to the model?

Managed identities instead of keys in configuration, roles at the resource and individual deployment level, private connection endpoints. Secrets are in a secure vault, and model calls are logged with binding to the source service.

Is it realistic to change the platform later?

We keep call logic in a separate adapter layer, prompts and evaluation sets — outside the platform. Migration comes down to replacing the adapter and rechecking quality.

Let's discuss the task: what is needed for assessment and a work plan

So that project assessment is based on the scope of work, not on assumptions, we need a minimum of inputs along three axes: model use case, data and access, and constraints on load and timelines.

The more specific the input, the more accurate the range by stages and the fewer edits in the implementation plan midway.

• Model use case: what task we solve (document parsing, dialogue with a knowledge base, ticket classification), who the user is and what we consider a correct answer.
• Data: where the sources are located, volume, formats (scans, texts, tables), whether there is labeling and requirements for personal data processing.
• Load: how many requests per hour at peak, acceptable response time, whether streaming output and offline mode are needed.
• Access requirements: Azure subscription, owner or contributor roles, quotas per region, private endpoints and access to storage.
• What already exists: prototypes, prompt templates, test sets, restrictions on data placement in a specific region.
• Acceptance criteria and timelines: how we measure quality on the test set, who accepts the result and by what date the working environment is needed.

Describe the task in free form — we will return with an assessment, environment design and staged implementation plan.