What breaks without caching LLM responses as load grows
While the load is low, repetitions go unnoticed: no one counts how many requests go to the model twice. On real traffic, the share of repeating requests reaches half — greetings, standard phrasings, the same question about request status in different words.
Each such request goes through the entire path again: prompt assembly, model call, generation. The most resource-intensive part of the pipeline is spent on an answer that was already received a minute ago and is lying idle.
Then model response time grows, and the reason is not the model itself but the queue. There are more simultaneous requests than available processing slots, so requests wait rather than being computed. The latency tail lengthens for everyone — including those who came with a unique question.
At peaks, a second effect kicks in — overload under peaks. The queue grows faster than throughput, and timeouts appear at the upper boundary under peak load. The user sees an error and sends the request again, adding work for the service: the failure begins to reinforce itself.
Caching LLM responses removes exactly this mechanism: a repeating request gets a ready answer without reaching the model. Below we break down how this is addressed — from exact match to semantic.
What LLM caching provides: response speed and model load
The cache works at two levels: exact match of the normalized request and semantic similarity of vectors. The first level catches verbatim repetitions — typical support questions, template documentation requests, repeated calls from the same scenario.
The second compares embeddings and returns an answer when similarity is above a set threshold, so paraphrases (“how to process a return” and “returning goods — how to do it”) fall into one entry.
The hit rate in enterprise assistants usually stays in the 30–60% range and depends on the diversity of phrasings and the length of the system prompt.
It is this share that determines the reduction in response time: a hit is returned from storage in tens of milliseconds, while generation takes seconds. The user sees the difference not in the average time but in the tail — heavy requests stop being the norm.
The number of model calls drops proportionally to the hit rate.
The freed capacity becomes capacity headroom: the assistant withstands growth in users without expanding the infrastructure, and peaks — the morning influx, a mass mailing, a campaign launch — pass without rate-limit failures. This class of failures disappears entirely rather than being masked by client-side retries.
The similarity threshold is selected by measurement, not by eye: too low yields false hits and answers to the wrong question, too high nullifies the gain.
Industry measurements, including Anthropic on prompt reuse, show a multiple reduction in latency on repeated requests — provided that hit rate and answer accuracy are measured continuously, not once at launch.
Two request matching modes: exact and semantic
Before enabling the cache, you need to decide by what criterion two requests are considered identical. Exact match compares normalized strings, semantic search compares vector representation similarity.
This choice affects both the hit rate and the risk of giving the user an answer to someone else’s question.
Request normalization is a common layer for both modes: case, extra spaces, service fields, parameter order. Without it, exact match loses hits for no reason: “How much does delivery cost?” and “how much does delivery cost” go to the model twice.
| Option | When it fits | Limitations |
|---|---|---|
| Exact match | Template repetitions: forms, autocomplete, repeated call with the same parameters | Does not catch paraphrases and typos, requires normalization |
| Semantic search | Free-form questions where meaning matters, not wording | Similarity threshold: too low yields false hits |
| Tiered hybrid | Heterogeneous traffic: exact level first, then semantic | Harder debugging and selective cache reset when the model changes |
In practice, a tiered hybrid matching scheme works more often: the first exact level is fast and error-free, the second semantic level catches paraphrases.
The similarity threshold is selected on a labeled sample of real dialogues: too high lowers the hit rate, too low grows the class of false positives.
A separate limitation of the semantic level is reset: when the model is updated, old vectors stop corresponding to answers, and entries are invalidated selectively rather than entirely.
Implementation order: from request profile to handover to support
The order of work is fixed: first we measure what actually goes to the model, and only then enable the cache. If you start with configuration, the request key is assembled from guesses — false hits appear, and the user gets an answer to someone else’s question.
That is why the stages proceed sequentially, with a point of no return after testing on shadow traffic.
- Request flow audit. We take an export of requests for 2-4 weeks: share of repetitions, spread of phrasings for the same question, context length, set of generation parameters, share of requests with personal data. The output is a map of scenarios with caching potential.
- Request key scheme. We determine what goes into the exact-match key (normalized text, system instruction, generation parameters, model version identifier) and what we consider similarity for the semantic layer. We separately mark scenarios that are not cached at all.
- Storage policies. We set record time to live by scenario type, invalidation rules when the prompt or model changes, index size, and storage limits.
- Cache layer configuration. We place it before the model, bring up the similarity index, configure warm-up on historical requests, and log hits.
- Shadow traffic testing. Production traffic proceeds as usual, while cache decisions are computed in parallel, without affecting the answer. We look at the hit rate, answer discrepancies, and behavior on rare phrasings.
- Post-enablement measurements. We compare with the baseline: share of requests that reached the model, response time, generation load, share of answers with signs of quality degradation.
- Handover to support. Invalidation procedure, metrics dashboard, alert thresholds, and a procedure for quickly disabling the layer — so operations do not depend on the authors of the solution.
Implementation artifacts: configurations, diagrams, policies, procedures
Implementation ends not with the fact of enabling the cache, but with the handover of a package of artifacts by which the operations service can maintain the solution without the author.
Below is the composition of this package; each item addresses a specific class of failures, not just for a checkmark.
- Service configurations — working files with the semantic match threshold, record time to live by request class, storage size limits, and key space separation rules. Each value is provided with a comment: the range within which it can be changed, and what degrades when going beyond the boundaries.
- Request flow diagram — from normalization of incoming text through exact-match search and vector search to the model call and answer write. The diagram marks logging points and the place where the answer is marked as obtained from cache.
- Exclusion and privacy policies — a list of requests that are not cached at all: with personal data, with current session context, with customer credentials. Plus key isolation rules by tenant, so that one user's answer does not surface for another.
- Invalidation rules — automatic reset when the model version or system request changes, manual clearing by tag and by key space, emergency cache shutdown with one switch. It describes what happens to answers already returned.
- Metrics dashboard — hit rate, share of false semantic matches, response time, number of model calls. These indicators show where the threshold is set too softly.
- Documentation for the on-call shift — step-by-step instructions for typical situations: growth in false matches, storage overflow, vector index unavailability. For each — symptom, action, rollback procedure to the mode without cache.
- Validation procedure — a set of regression requests with reference answers and the run order before changing configuration, as well as access and roles for the customer team.
We hand over the package with a walkthrough on a call and a short session for the on-call shift — so that a month later the solution can be changed without us.
Case study: how latency and model load were reduced in a month
Before: request flow without cache
The customer has a first-line support conversational assistant. Requests repeat massively, but every request still went to the model: average response time stayed around 3.4 s, and at peaks reached 9 s.
During load hours, some requests were cut off due to the model provider's limit, and users went to the ticket form instead of getting an answer in the dialogue.
After: two cache levels
We set up two levels. The exact one compares the normalized request together with the prompt version and generation parameters — a match is returned immediately, storage on Redis with a limited record time to live, so it does not grow uncontrollably.
The semantic one searches for semantically close phrasings in the vector index and answers only above the match threshold; everything else goes to the model.
On a comparable profile, the hit rate came out to 41%, of which 27% were semantic. Average response time fell from 3.4 to 0.6 s, the upper percentile — from 9 to 1.4 s, and the number of model calls decreased by 43%.
The class of failures “provider limit at peak” disappeared: model load no longer grows from repetitions.
Measurements were taken on the same set of requests — two weeks before implementation and two weeks after, broken down by request categories and separately by warm-up in the first minutes after restart.
Invalidation by prompt version prevented answers from diverging after instruction edits. The full breakdown of measurements is below: request profile, hit rate, and latency distribution for each category.
What to do about false hits in an already running system
A false hit almost never means that matching is configured incorrectly. In a running system, it usually means that the similarity threshold is set the same for all request types, and decisions are not recorded anywhere: the failure is visible, the cause is not.
The first step is not to adjust the threshold but to log every hit: the source request, the found “neighbor,” the similarity value, and the flag of whether the answer was accepted.
The similarity threshold almost always needs to be separated by groups. Short and general questions produce high similarity with anything, so the threshold is raised for them, while for long requests with specifics it is left lower.
A single number for the whole system is a compromise that both lets noise through and cuts useful matches.
Next is review of disputed pairs: we select those where similarity fell into the borderline zone and review them manually.
A few dozen reviewed pairs usually show that some hits are caused not by the threshold but by service noise in the vector — a greeting, a signature, a technical tail of the conversation that pulls the request toward someone else’s topic.
Targeted exclusions work more precisely than raising the threshold for everyone indiscriminately.
Requests with numbers, dates, names, and negations go into the exclusion list: there, similarity deceives most often, and substituting someone else’s answer costs more than an extra model call.
Answer quality control closes the loop. You need a set of reference pairs that is run on every change to the threshold or rules, and a metric for the share of irrelevant hits in live traffic. If after a change the metric has not improved, the change is rolled back.
Common technical questions: invalidation, privacy, measurements
An answer cache breaks not during writes but during maintenance: records outlive a model change, customer keys overlap, and the effect is measured by average response time.
What time to live should be set for a record?
We set it in two layers: expiration by time and a threshold by number of accesses. Reference answers live longer, factual ones shorter, so the time to live is tied to the request type rather than set as one number for the entire cache.
How are data of different customers and tenants separated?
The key is built as “tenant → customer → model version → request”: identical questions from different customers produce different records, and overlap is excluded. Storage (for example, Redis) is kept in separate sets with separate eviction policies.
What happens when the model version changes?
The model version is part of the key: the new version does not read the old cache but writes its own; previous records live out their time to live in parallel and do not interfere. Rolling back to the previous version returns its cache entirely.
How to measure the effect honestly?
We count the share of requests closed from cache and the time to first token on misses. Measurement runs on live traffic with a shadow branch: some requests always go directly to the model, and the difference between branches shows the real gain.
Let’s discuss the task: align requirements and come back with a plan
So that the caching estimate relies on your data rather than average numbers, it is enough to attach a few inputs to the discussion.
They do not require preparing a separate document — an export of metrics, a flow diagram, and a description of where the bottleneck currently is will suffice. We will break down how it is set up for you and come back with a work plan.
- Input data on load: how many requests per day, what the peak profile looks like, what share of phrasings repeats verbatim and what share is paraphrased.
- Where exactly time and cost are lost: response latency, share of model requests, amount of overpayment for repeated generations.
- Freshness requirements: after how long following a source update the answer must stop being served from cache, and whether there are requests that cannot be cached in principle.
- Infrastructure and constraints: where the services live, what data isolation requirements exist, whether there are rules for storing personal data and access audit.
- Timeline priorities: what must work first, what can be postponed to the second stage, by what date the result is needed.
- Who on the customer side makes decisions on disputed points — for example, on the semantic similarity threshold and invalidation rules.
As a result, you get a proposal format by which contractors can be compared: a cache scheme broken down into exact and semantic levels, work stages with effort estimates, a set of acceptance metrics (hit rate, response latency, share of correct returns after invalidation), and a list of risks with a rollback plan.
Describe the task in a convenient form — we will come back with an estimate and work plan within one business day.
