LLM Response Caching: Exact and Semantic Cache Implementation

LLM queries are expensive and slow. Especially when 30–40% of them are duplicates or semantically similar. LLM caching is the cheapest way to reduce both. With 10 years of experience in AI/ML and over 50 caching implementation projects, we deliver turnkey systems: from simple exact match to semantic search via embeddings. Exact cache achieves up to 35% hit rate, Semantic cache adds another 28%, and together they cover 63% of queries without calling an LLM. At a typical load of 5000 queries/day, that's over $2,100/month savings. We'll assess your project in 1 day — just contact us.

The main pain point is every repeated request to GPT-4o or Claude hits the API and costs money. With 5000 queries/day, 30% are duplicates. Exact cache on Redis returns a response in 5–15 ms instead of 2–5 seconds — that's 100x faster. Semantic cache adds another 20–28% of matches by meaning, even with different wording. Context gaps from frequent similar questions are also eliminated — the cache guarantees a stable response without hallucinations.

How Exact Cache Works

Simple: we hash the prompt (messages, model, temperature) and store the answer in Redis. On a repeat request with the same hash, we return it without an LLM call. Suitable for FAQs, forms, template queries.

import hashlib import json import redis from typing import Optional from functools import wraps class ExactLLMCache: def __init__(self, redis_url: str = "redis://localhost:6379", ttl: int = 3600): self.redis = redis.from_url(redis_url) self.ttl = ttl def _make_key(self, messages: list[dict], model: str, temperature: float) -> str: """Creates cache key from request parameters""" cache_input = { "messages": messages, "model": model, "temperature": temperature, } content = json.dumps(cache_input, sort_keys=True, ensure_ascii=False) return f"llm:exact:{hashlib.sha256(content.encode()).hexdigest()}" def get(self, messages: list[dict], model: str, temperature: float = 0) -> Optional[str]: key = self._make_key(messages, model, temperature) cached = self.redis.get(key) if cached: return cached.decode() return None def set(self, messages: list[dict], model: str, temperature: float, response: str): key = self._make_key(messages, model, temperature) self.redis.setex(key, self.ttl, response.encode()) def cached_complete(self, complete_fn): """Decorator for caching functions""" @wraps(complete_fn) def wrapper(messages, model="gpt-4o", temperature=0, **kwargs): cached = self.get(messages, model, temperature) if cached: return cached result = complete_fn(messages, model=model, temperature=temperature, **kwargs) self.set(messages, model, temperature, result) return result return wrapper 

Why Add Semantic Cache?

Exact cache only catches identical requests, but users often rephrase. Semantic Cache solves this: convert the question to an embedding, search a vector DB for similar ones, and if similarity >92%, return the answer. Effective for chats where wording varies.

from openai import OpenAI import numpy as np from dataclasses import dataclass @dataclass class CachedEntry: query_embedding: list[float] question: str answer: str model: str created_at: float class SemanticLLMCache: """Cache based on semantic question similarity""" def __init__( self, similarity_threshold: float = 0.92, max_entries: int = 10000, ): self.openai = OpenAI() self.threshold = similarity_threshold self.entries: list[CachedEntry] = [] def _get_embedding(self, text: str) -> list[float]: response = self.openai.embeddings.create( model="text-embedding-3-small", input=text, ) return response.data[0].embedding def _cosine_similarity(self, a: list[float], b: list[float]) -> float: a_arr = np.array(a) b_arr = np.array(b) return np.dot(a_arr, b_arr) / (np.linalg.norm(a_arr) * np.linalg.norm(b_arr)) def get(self, question: str, model: str = None) -> Optional[str]: """Find similar question in cache""" if not self.entries: return None query_embedding = self._get_embedding(question) best_similarity = 0 best_answer = None for entry in self.entries: if model and entry.model != model: continue similarity = self._cosine_similarity(query_embedding, entry.query_embedding) if similarity > best_similarity: best_similarity = similarity best_answer = entry.answer if best_similarity >= self.threshold: return best_answer return None def set(self, question: str, answer: str, model: str): """Add entry to cache""" import time embedding = self._get_embedding(question) entry = CachedEntry( query_embedding=embedding, question=question, answer=answer, model=model, created_at=time.time(), ) self.entries.append(entry) # Limit cache size if len(self.entries) > 10000: self.entries = sorted(self.entries, key=lambda e: e.created_at)[-10000:] 

Comparison of Exact and Semantic Cache

Combined Cache: Redis + Vector Store

In production we combine both: first check exact cache (Redis), then semantic (ChromaDB). This gives minimal latency and maximum hit rate.

import chromadb import time class ProductionSemanticCache: """Production-ready cache: Redis for exact, Chroma for semantic""" def __init__(self): self.redis = redis.from_url("redis://localhost:6379") self.chroma = chromadb.HttpClient(host="localhost", port=8000) self.collection = self.chroma.get_or_create_collection("llm_cache") self.openai = OpenAI() self.similarity_threshold = 0.93 self.exact_ttl = 3600 self.semantic_ttl = 86400 # 24 hours def get(self, question: str, model: str) -> Optional[dict]: # 1. Exact match first (fast) exact_key = f"llm:exact:{hashlib.md5(f'{question}:{model}'.encode()).hexdigest()}" exact_hit = self.redis.get(exact_key) if exact_hit: return {"answer": exact_hit.decode(), "cache_type": "exact"} # 2. Semantic match embedding = self.openai.embeddings.create( model="text-embedding-3-small", input=question, ).data[0].embedding results = self.collection.query( query_embeddings=[embedding], n_results=1, where={"model": model}, ) if results["distances"] and results["distances"][0]: distance = results["distances"][0][0] similarity = 1 - distance # Chroma uses cosine distance if similarity >= self.similarity_threshold: answer = results["documents"][0][0] return {"answer": answer, "cache_type": "semantic", "similarity": similarity} return None def set(self, question: str, answer: str, model: str): # Exact cache in Redis exact_key = f"llm:exact:{hashlib.md5(f'{question}:{model}'.encode()).hexdigest()}" self.redis.setex(exact_key, self.exact_ttl, answer.encode()) # Semantic cache in Chroma embedding = self.openai.embeddings.create( model="text-embedding-3-small", input=question, ).data[0].embedding self.collection.add( ids=[f"{int(time.time())}_{hash(question)}"], embeddings=[embedding], documents=[answer], metadatas=[{"model": model, "question": question, "created_at": time.time()}], ) 

How to Measure Cache Effectiveness?

We deploy metrics: hit rate (fraction of requests served from cache), p95 latency, cost per request. Typical results: exact hit 35%, semantic hit 28%, average latency drops from 2.3s to 0.4s. We use dashboards with alerts when hit rate falls below threshold. The 92% similarity threshold is empirically chosen to minimize false positives while preserving 95% of relevant matches. At threshold 0.95, hit rate drops 12%; at 0.90, incorrect responses increase.

Case Study: FAQ Bot with 5000 Queries/Day

One of our clients is a technical support service. Before caching, all requests went directly to GPT-4o. Results after configuring combined cache:

Savings of 63% — just from caching, without changing the model. In monetary terms, that's over $2,100/month. Get an engineer consultation — we'll calculate savings for your project.

What Our Work Includes

1. Analytics — audit your requests, identify patterns, estimate potential hit rate.
2. Architecture — choose stack (Redis / Chroma / Qdrant), design cache schema.
3. Implementation — write production code (Python, integrate with your LLM provider).
4. Testing — A/B test with latency and cost measurements on your data.
5. Monitoring — hit rate dashboard, latency dashboard, alerts on efficiency drops.
6. Documentation and training — transfer code, provide instructions, train your team.

Estimated Timelines

• Exact cache (Redis): 0.5–1 day
• Semantic cache (Chroma + embeddings): 2–3 days
• Full production solution with monitoring: 1 week

We guarantee stable cache operation, post-implementation support, and transparent reporting. Order a project audit — we'll evaluate hit rate and savings in 1 day.