Protecting AI APIs from Overload: Rate Limiting and Token Quotas for LLMs

What breaks without rate limiting in AI services

Model response generation keeps a worker busy for the entire streaming duration — from the first token to the last.

When a client starts batch processing or a burst of requests from a web client hits the service, concurrent calls line up in a queue, and each subsequent one waits for a slot longer than the previous.

The worker pool is the first to hit its limit: it is sized for average load, not peak load. Then response time under load grows — first to seconds, then to timeouts on the caller side.

The queue is not infinite: when it overflows, requests are dropped, but part of the work has already been done and wasted.

Next, cascading failures begin. After receiving a timeout, the client retries the request, and retries finish off the remaining capacity.

Errors spread to neighboring services — the ingress gateway, usage metering, index search, cache: they all receive a barrage of identical retries at the same time and degrade along with the primary service.

A separate problem is the lack of capacity isolation.

One integrator with an aggressive polling loop takes the entire pool and degrades service quality for everyone else, formally without violating anything: service overload looks like a general failure, not like the behavior of a specific client. We explain how this is addressed: per-client limits, token quotas, and queue priorities.

What limit management delivers: load relief and rejections

The effect of limits is visible not in the configuration but in the service metrics. Below is what changes when thresholds for requests and tokens are set for each key, and the queue depth is bounded.

The rejection rate stops depending on someone else's traffic

Without thresholds, a burst from one consumer goes into the shared pool and brings everyone down: the service responds with errors, and retries add load. A separate queue per key breaks this chain — the rejection rate stays in the single digits even during peak hours.

We keep the admission counter in Redis with token accumulation: the check does not lengthen the model response.

The peak is handled through degradation, not rejection

During a tenfold spike, some requests enter a bounded-depth queue, while the rest receive a clear rejection indicating when to retry. Instead of an avalanche of errors for everyone, there is predictable latency for those who pass the threshold. Resilience rests on excess demand being shifted to the client rather than to the service.

Response latency stops fluctuating

A limit on context length and the number of concurrent generations removes contention for compute resources. Interactive requests get priority over batch requests, so the 95th percentile response time stays within the agreed corridor even at full load.

Capacity is distributed evenly

Quotas per team and project plus a separate pool for background tasks: one large consumer does not crowd out the rest. A guaranteed share of capacity is reserved for each client and does not go to whoever simply sent a request first.

Throttling modes: request limits, token quotas, queues

Request limits, token quotas, concurrent sessions, buffering, and key priorities solve one problem — preventing one consumer from eating up the model's capacity.

The granularity differs: rate limiting counts calls but not their weight; a token quota does not distinguish ten small calls from one heavy request with a large context.

That is why the modes are not chosen as “one of” but arranged in order: what is checked first, what goes into the queue, and who gets the remaining capacity.

At the entry point are key priorities; we keep the budget on a token quota; the request limit insures against a barrage of small calls; we enable concurrent sessions for interactive use and buffering for background tasks.

We fix the order of checks in configuration so that the rules are reproducible and do not depend on the code of an individual service.

Implementation stages: from load measurement to operating procedures

Before setting limits, we capture the real profile: hourly request distribution, context length, retry rate, and behavior of individual clients.

Eyeballed thresholds are equally bad in both directions — an inflated limit does not protect against overload, while an understated one cuts off legitimate scenarios and looks to the user like service instability. That is why the work order is strict.

1. Load profile. We collect statistics over several weeks, including at least one peak period. We separately mark which calls are critical for the business process and which can be deferred to a queue.
2. Threshold calculation. We define limits layer by layer — per key, per user, per service — with headroom up to model saturation, so that the limiter triggers before response time starts to grow.
3. Limiter configuration. We choose an algorithm for the traffic pattern (sliding window or token bucket) and introduce separate quotas for the number of requests and token volume: a short call and a long context consume resources incomparably.
4. Rejection handling. The limit must be predictable: a clear response code, an indication of when to retry, and a priority queue for critical operations.
5. Peak testing. We run a profile that replicates the real peak with headroom and record at what values rejections appear and whether they affect legitimate clients.
6. Operations procedure. We hand over a document: how to change quotas, where to view metrics, at what deviations to raise the limit, and how to issue a temporary exception.

We separately agree on reviewing thresholds — when scenarios change or the number of clients grows, the numbers become outdated faster than the configuration.

What we hand over to the customer: configurations, diagrams, procedures

We deliver not a description of “how it should be” but a working set that an on-call engineer and developer can use without us: files applied directly in the environment and procedures for specific incidents.

• Limiter configuration — rules per key, user, and model: window, threshold, request weight. Stored in a repository, applied on the fly, without restarting the service.
• Request flow diagram — the path from client through the gateway to the model: where token usage is counted, at what point a rejection triggers, and what is returned to the caller.
• Quota matrix — a table by keys and customer groups: per-minute and daily limits, a separate ceiling for expensive models, and headroom for internal tasks.
• On-call shift instructions — how to distinguish a load spike from abuse, which thresholds the shift can raise on its own, and which only with approval.
• Threshold change log — who moved a limit, when, and for what reason, with the ability to revert to the previous value.
• Metrics and alerts — rejection rate, remaining quota, queue wait time; the signal arrives before customers notice degradation.
• Key issuance and rotation procedure — who gets which access level, how a key is revoked, and what happens to already-started requests.

We update the set together with the service; otherwise the procedure diverges from the environment, and during an incident no one knows where the truth is.

Case study: peak user influx without failures

The peak for the response generation service occurred in the evening hours: 40 requests per second against the provider's contractual limit of 20 with OpenAI. Requests went directly — without a queue, without tracking token usage per user, and without separation by plan. Every excess request returned an error, the client retried it, and the avalanche only intensified.

Retries on timeout also contributed their share: the user pressed the button again, and the application submitted a second task to the same bottleneck.

The result was 7.8% error responses and a p95 latency of 12 seconds, and during the peak twenty minutes the rejection rate reached 15%: people left without waiting for a response.

What was set at the entry point

Rate limiting was made two-tier: a token bucket per user with the right to a short burst and a 60-second sliding window for smoothing. On top — a daily token quota for each plan and a priority queue for paid requests.

A shared provider circuit breaker opens when errors rise and prevents finishing it off with retries.

Retries were switched to exponential backoff with jitter, and identical prompts within a minute began to be served from cache — this relieved part of the load that users themselves had previously created.

Metrics after

• Rejections: 7.8% → 0.4% of all requests.
• p95 latency: 12 seconds → 2.6 seconds; a burst of up to 60 requests per second passes without degradation.
• Share of responses from cache at peak: 22%.
• Draining the accumulated queue after a provider failure — 90 seconds instead of manual resubmission.

Quotas no longer silently drop a request: when the limit is exhausted, the system shows the estimated wait time. Token usage became plannable by plan, and the load on the provider became predictable.

What to do if the AI gateway is already hitting the ceiling?

First, real-time measurement, not analysis of week-old logs. On the running service, we enable metric collection: rejection and retry rate, request distribution by key and model, 95th percentile response time, queue depth.

We do not enable request rate limiting and token quotas yet — first we need to see who exactly creates the load and where the bottleneck is.

Then we calculate shadow thresholds. The rule is applied to each request in a simulated way: the log records a verdict about what would have been rejected under such a quota, but the request itself passes.

The threshold is selected based on real traffic, and the cost of an error is immediately visible — how many legitimate clients the limit would affect.

We enable it in stages. The rule first applies to one key or a small share of traffic, then the share grows — each step has its own transition criterion. Before anything else, we cover rare but most expensive calls, and only then the general flow.

Rollback of settings lives in configuration, not in code: the threshold profile is changed with a single edit and reverts to the previous one in seconds, without a restart or rolling out a new version.

Plus automatic rollback — if the rejection rate rises above baseline, the rule is removed automatically.

The service does not stop at any of the steps. The customer is left with a log: which threshold was active, how many requests it affected, when it was changed — the next iteration is built on this data.

Common technical concerns about limits and quotas

Request rate limiting and token quotas are not introduced for the sake of prohibitions: their job is to prevent a burst from one client or a looped call from eating up capacity for everyone else.

The rules are described declaratively, next to the gateway, and the business logic of applications remains untouched. Below are the objections most often raised when approving the scheme: they concern the behavior of real users and the scope of code changes.

Can different rules be set for teams and products?

The rules are attached to an access key, not to the service as a whole: internal teams, partner integrations, and the external application have different thresholds. The token quota is calculated separately from the request rate.

Are limits easy to bypass?

Counters live on the service side, not in the client, so editing application code does not affect them. Keys are issued and revoked centrally, and anomalies on a single key are visible in the metrics.

Will application code need to be rewritten?

There is no mandatory rewrite: the application calls the same address; only handling of the limit-exceeded response changes. With standard retries, the change comes down to a few lines, and we include it in the scope of work.

Let's discuss your load — we'll come back with an estimate and a plan

To set limits by calculation rather than by eye, a few initial data points are enough — they are usually already in the logs and in the model provider's statistics. We will build the picture and come back with numbers for quotas, timelines, and scope boundaries.

• Load profile: requests per second at peak and on average, distribution by API route and by client.
• Token usage: median and 95th percentile per request and per response, share of long contexts.
• Keys and access: what the quota is calculated by — key, account, team; how they are issued and revoked.
• Behavior on failures: whether there are retries, the pause between them, how much traffic returns after a 429 code.
• What is already at the entry point: gateway, proxy, load balancer, where counters are stored — in the memory of an individual node or in shared storage.
• Priorities: which scenarios must not fail, which may be delayed in a queue.

Then we prepare a plan: request and token limits, window sizes and burst handling, separate pools for interactive and background tasks, behavior when the quota is exhausted — a rejection with an estimated wait time, a queue, or soft degradation.

We return gateway configurations, a map of limits by key, a rollout procedure with rollback, an estimate of timelines by stage, and responsibility zones: what we do and what remains on your side.

Describe the task and send what you have on load — we will come back with an estimate and a plan.