LLM Acceleration: Setting Up KV Cache and Semantic Cache

LLM inference caching: where speed is lost

Every new request is processed by the model from scratch: it recalculates attention over the entire context, although 80–90% of the prompt has not changed since the previous call.

On long system instructions and dialogue history, the share of recomputation reaches ninety percent — and it is paid for with time to first token, not with product logic.

While there are few simultaneous sessions, this goes unnoticed. When there are dozens of them, memory for KV state runs out, the server starts queuing requests and cutting batch size.

High model response latency looks like “the model is slow,” although the bottleneck during generation is repeated prefill and an overflowing cache.

Throughput drops faster than the number of requests grows, and degradation as requests grow turns into gateway timeouts, stream interruptions, and client retries, which again load the same inference server.

Adding accelerators is the most expensive and weakest solution: there will be no linear throughput growth while every request recalculates the same thing.

We break down how this is solved: LLM inference caching relies on two mechanisms — reuse of an already computed prefix (as in vLLM) and a cache at the level of request meaning.

What reuse of responses and context gives

In a dialogue service, requests are almost never independent: the system instruction, domain description, and previous turns repeat in every call.

Without context reuse, the compute engine recalculates the same prefix, and that is exactly where most of the time to first response goes. The attention cache removes this work: computed vectors remain in memory and are substituted into subsequent requests of the same session.

In current-generation serving engines, such as in vLLM, the cache is maintained in blocks, and shared prefixes are reused between sessions — growth in concurrent dialogues stops linearly consuming compute memory.

Time to first response on a long system prefix is reduced by 2–4 times, and total output time falls in proportion to the reused context share.

The semantic cache works one level higher: questions like “how to arrange a return” come in dozens of formulations, vector search finds a semantically close request, and returns the ready answer without starting generation. Such requests are served in time indistinguishable to the user.

Both mechanisms add up to service throughput: with a hit rate of 30–50% on a typical flow, load on compute engines decreases almost proportionally, the peak is handled on the same fleet, and the latency tail stops spreading.

Caching modes: exact, semantic, contextual

Three caching modes solve different tasks, and confusing them is a common cause of disappointment.

Exact response cache catches request repetition character by character, semantic cache catches paraphrases like “how to arrange a return” and “how to make a product return,” and contextual cache holds a cache of attention states so as not to recalculate the shared prompt prefix on every call.

The choice is dictated by the request profile: share of repetitions, length of the shared prompt part, and cost of error on a false hit.

For semantic cache, the similarity threshold is critical — a lowered one gives a confident answer to the wrong question, a raised one turns the cache into decoration. We select the threshold on a real request log, separately analyzing false hits.

In practice modes are layered: exact first, semantic behind it, attention states cache at the inference level itself. Thus a paraphrase does not miss the cheap layer, and a model update affects the lower level and does not break the upper ones.

How we implement: audit, scheme, measurement, handover

Work proceeds in stages, and the first of them is not configuration but measurement. Implementing caching without baseline measurements turns into an argument about tastes: it is unclear what gave the gain and what will break on the next model update.

Therefore we first record the current state, then change the configuration — and repeat measurements on the same load.

1. Load audit. Analyze real traffic: share of verbatim repeated and paraphrased requests, context length distribution, share of system prompts.
2. Request path scheme. Draw the path from client to model and back, mark where the cache check occurs and what happens on a miss.
3. KV cache tuning. Select buffer size, eviction policy, and per-session separation so memory does not grow with every dialogue.
4. Semantic cache. Set similarity threshold, request normalization rules, and tenant isolation — one user's answer must not go to another.
5. Metric measurement. Compare state before and after on a comparable sample: time to first token, total response time, hit rate, accelerator load.
6. Edge-mode testing. Cold start, traffic spike, cache storage unavailability, model change — the system must degrade, not fail.
7. Handover to operations. Maintenance regulations, monitoring dashboard with key metrics, rollback plan, and training for your team.

What remains with you: configurations, schemes, regulations

The project ends not with a demo but with handover: your team must retain artifacts by which the system can be developed and maintained without us. Here is what is included in the LLM inference caching package.

• Cache configurations — storage sizes, request similarity thresholds, record lifetimes, and eviction rules, with an explanation for each chosen value.
• Request path scheme — where it hits the cache and where it misses, what measurements are taken at each path segment.
• Invalidation regulations — what is reset on model update, system instruction change, or reference data edit, who initiates it, and how the result is verified.
• Access policies — separation by keys and consumers so a cached response does not go to the wrong party, plus rules for storing sensitive data and logging requests.
• Metrics and alerts — hit rate, response time, error classes; with a description of how to distinguish load growth from cache degradation.
• Cache documentation — architecture, allowable parameter ranges, deployment procedure, and rollback to the previous configuration.

We hand over everything as configuration files and descriptions, without binding to our infrastructure.

A walkthrough with your engineers and test scenarios on real requests are part of acceptance, so that maintenance moves to your side without losing knowledge of how caching is arranged.

Case: response time and hit rate before and after

The task came from a support service: an assistant answers from a knowledge base, about 12 thousand dialogues per day pass through, more than half in two peak windows.

Before caching, inference recalculated the context entirely on each request — the system instruction, history, and retrieved documentation fragments went into the model again, even when the question repeated verbatim.

Measurements before implementation: median response held at 7 seconds, at peak reached 14, cache hit rate was zero. The main failure class — timeout interruption: the request queue grew, and some requests ended with an error instead of an answer.

What we changed

We separated the immutable prompt prefix and the variable part: attention blocks for the shared instruction are no longer recalculated, and the key-value cache is reused between dialogues.

On top we placed a semantic cache — the question is reduced to a vector, and when the match is above threshold 0.92, the ready answer is returned without calling the model.

Records are split by access rights, lifetime is tied to the knowledge base version, so when documentation is updated, related answers are invalidated automatically.

Metrics after implementation: median response — 1.8 seconds, on repeated questions — 0.6; cache hit rate — 47% average and 63% in peak windows.

Timeouts as a failure class disappeared, inference load dropped about fourfold, and the freed capacity went to growth in concurrent dialogues.

Before and after measurements were taken on the same set of scenarios, so the figures are comparable. With the customer's consent, we discussed the case at an industry conference — LLM Inference in Production.

How is context reuse different from response replay?

The difference is in the level at which work is saved. Context reuse lives inside one request: the model does not recalculate attention states for the already read part of the request, but continues from the saved position.

This speeds up first token output on long system instructions and in dialogue, but the request to the model still occurs.

Response reuse works higher — at the level of the entire request. If a close request has already been processed, the system returns the ready result, and the model does not start at all.

Hence two different mechanisms: exact cache compares normalized request text, semantic cache compares meaning and allows paraphrases.

The choice of mechanism is not either-or. Context reuse is enabled always: it does not change the answer and reduces load on long inputs.

Semantic cache is placed where the request flow noticeably repeats, and necessarily with a similarity threshold, hit log, and exceptions for personal data and transactional questions.

A poorly chosen threshold gives the worst failure class: the system confidently returns a similar but incorrect answer. Therefore hit rate and accuracy are verified on a set of reference “query — expected answer” pairs, and disputed topics are marked as not subject to caching.

Limitations and typical configuration mistakes

Cache speeds up responses but introduces an error class that did not exist without it: the user gets a confidently worded but outdated answer.

KV cache is tied to a specific prompt prefix and falls apart when the system instruction changes, while semantic cache searches by meaning and therefore tends to false matches. We break down the doubts that come up at acceptance.

Most failures live not in the cache itself, but in the hit rules and record lifetime. A key from raw request text gives zero hits when words are reordered, and a key without binding to source version replicates outdated data. We check both boundaries before enabling in production.

Where is the boundary of false matches in semantic search?

We select the similarity threshold by annotation: pairs “query — expected hit” and pairs of adjacent but semantically different questions. A lowered threshold mixes queries about different entities, a raised one zeroes out hits. Queries with a number, date, or name are marked as unsuitable for cache.

What definitely should not be cached?

Personalized answers, requests with access rights checks, and one-off operations. We calculate hit rate from the log: if a scenario repeats rarely, the record consumes memory and adds latency without gain.

Let's discuss the task: what we need from you to start

To discuss the task substantively, not “by feel,” we need a minimal input data set. Ordering cache configuration makes sense when the traffic profile and target metrics are clear — that is exactly what we review in the first conversation.

• which model is used and how it is run: self-hosted environment, cloud provider, or mixed scheme, whether there are restrictions on outbound requests;
• traffic profile: share of repeated requests, typical context length, distribution by sessions and users;
• current metrics: time to first token, total response latency, accelerator load at peak hours;
• privacy requirements: whether request and response texts can be stored, exactly where and for how long;
• target indicators: by how many times latency needs to be reduced and what cache hit rate you consider workable;
• change window: whether there is a test environment, whether a brief dip in production during rollout is acceptable, how rollback is arranged;
• responsible persons and accesses: who on the customer side approves configuration changes and production launch.

In reply we send an assessment and work plan: inventory of the current inference, KV-cache and semantic layer scheme with invalidation policy, rollout order, and a set of measurements by which we will verify the effect. We agree on stages — and project start takes days, not weeks.