LLM Response Caching: Exact and Semantic Cache Implementation

LLM queries are expensive and slow. Especially when 30–40% of them are duplicates or semantically similar. LLM caching is the cheapest way to reduce both. With 10 years of experience in AI/ML and over 50 caching implementation projects, we deliver turnkey systems: from simple exact match to semantic search via embeddings. Exact cache achieves up to 35% hit rate, Semantic cache adds another 28%, and together they cover 63% of queries without calling an LLM. At a typical load of 5000 queries/day, that's over $2,100/month savings. We'll assess your project in 1 day — just contact us.

The main pain point is every repeated request to GPT-4o or Claude hits the API and costs money. With 5000 queries/day, 30% are duplicates. Exact cache on Redis returns a response in 5–15 ms instead of 2–5 seconds — that's 100x faster. Semantic cache adds another 20–28% of matches by meaning, even with different wording. Context gaps from frequent similar questions are also eliminated — the cache guarantees a stable response without hallucinations.

How Exact Cache Works

Simple: we hash the prompt (messages, model, temperature) and store the answer in Redis. On a repeat request with the same hash, we return it without an LLM call. Suitable for FAQs, forms, template queries.

import hashlib
import json
import redis
from typing import Optional
from functools import wraps

class ExactLLMCache:
    def __init__(self, redis_url: str = "redis://localhost:6379", ttl: int = 3600):
        self.redis = redis.from_url(redis_url)
        self.ttl = ttl

    def _make_key(self, messages: list[dict], model: str, temperature: float) -> str:
        """Creates cache key from request parameters"""
        cache_input = {
            "messages": messages,
            "model": model,
            "temperature": temperature,
        }
        content = json.dumps(cache_input, sort_keys=True, ensure_ascii=False)
        return f"llm:exact:{hashlib.sha256(content.encode()).hexdigest()}"

    def get(self, messages: list[dict], model: str, temperature: float = 0) -> Optional[str]:
        key = self._make_key(messages, model, temperature)
        cached = self.redis.get(key)
        if cached:
            return cached.decode()
        return None

    def set(self, messages: list[dict], model: str, temperature: float, response: str):
        key = self._make_key(messages, model, temperature)
        self.redis.setex(key, self.ttl, response.encode())

    def cached_complete(self, complete_fn):
        """Decorator for caching functions"""
        @wraps(complete_fn)
        def wrapper(messages, model="gpt-4o", temperature=0, **kwargs):
            cached = self.get(messages, model, temperature)
            if cached:
                return cached
            result = complete_fn(messages, model=model, temperature=temperature, **kwargs)
            self.set(messages, model, temperature, result)
            return result
        return wrapper

Why Add Semantic Cache?

Exact cache only catches identical requests, but users often rephrase. Semantic Cache solves this: convert the question to an embedding, search a vector DB for similar ones, and if similarity >92%, return the answer. Effective for chats where wording varies.

from openai import OpenAI
import numpy as np
from dataclasses import dataclass

@dataclass
class CachedEntry:
    query_embedding: list[float]
    question: str
    answer: str
    model: str
    created_at: float

class SemanticLLMCache:
    """Cache based on semantic question similarity"""
    def __init__(
        self,
        similarity_threshold: float = 0.92,
        max_entries: int = 10000,
    ):
        self.openai = OpenAI()
        self.threshold = similarity_threshold
        self.entries: list[CachedEntry] = []

    def _get_embedding(self, text: str) -> list[float]:
        response = self.openai.embeddings.create(
            model="text-embedding-3-small",
            input=text,
        )
        return response.data[0].embedding

    def _cosine_similarity(self, a: list[float], b: list[float]) -> float:
        a_arr = np.array(a)
        b_arr = np.array(b)
        return np.dot(a_arr, b_arr) / (np.linalg.norm(a_arr) * np.linalg.norm(b_arr))

    def get(self, question: str, model: str = None) -> Optional[str]:
        """Find similar question in cache"""
        if not self.entries:
            return None
        query_embedding = self._get_embedding(question)
        best_similarity = 0
        best_answer = None
        for entry in self.entries:
            if model and entry.model != model:
                continue
            similarity = self._cosine_similarity(query_embedding, entry.query_embedding)
            if similarity > best_similarity:
                best_similarity = similarity
                best_answer = entry.answer
        if best_similarity >= self.threshold:
            return best_answer
        return None

    def set(self, question: str, answer: str, model: str):
        """Add entry to cache"""
        import time
        embedding = self._get_embedding(question)
        entry = CachedEntry(
            query_embedding=embedding,
            question=question,
            answer=answer,
            model=model,
            created_at=time.time(),
        )
        self.entries.append(entry)
        # Limit cache size
        if len(self.entries) > 10000:
            self.entries = sorted(self.entries, key=lambda e: e.created_at)[-10000:] 

Comparison of Exact and Semantic Cache

Combined Cache: Redis + Vector Store

In production we combine both: first check exact cache (Redis), then semantic (ChromaDB). This gives minimal latency and maximum hit rate.

import chromadb
import time

class ProductionSemanticCache:
    """Production-ready cache: Redis for exact, Chroma for semantic"""

    def __init__(self):
        self.redis = redis.from_url("redis://localhost:6379")
        self.chroma = chromadb.HttpClient(host="localhost", port=8000)
        self.collection = self.chroma.get_or_create_collection("llm_cache")
        self.openai = OpenAI()
        self.similarity_threshold = 0.93
        self.exact_ttl = 3600
        self.semantic_ttl = 86400  # 24 hours

    def get(self, question: str, model: str) -> Optional[dict]:
        # 1. Exact match first (fast)
        exact_key = f"llm:exact:{hashlib.md5(f'{question}:{model}'.encode()).hexdigest()}"
        exact_hit = self.redis.get(exact_key)
        if exact_hit:
            return {"answer": exact_hit.decode(), "cache_type": "exact"}

        # 2. Semantic match
        embedding = self.openai.embeddings.create(
            model="text-embedding-3-small",
            input=question,
        ).data[0].embedding

        results = self.collection.query(
            query_embeddings=[embedding],
            n_results=1,
            where={"model": model},
        )

        if results["distances"] and results["distances"][0]:
            distance = results["distances"][0][0]
            similarity = 1 - distance  # Chroma uses cosine distance
            if similarity >= self.similarity_threshold:
                answer = results["documents"][0][0]
                return {"answer": answer, "cache_type": "semantic", "similarity": similarity}

        return None

    def set(self, question: str, answer: str, model: str):
        # Exact cache in Redis
        exact_key = f"llm:exact:{hashlib.md5(f'{question}:{model}'.encode()).hexdigest()}"
        self.redis.setex(exact_key, self.exact_ttl, answer.encode())

        # Semantic cache in Chroma
        embedding = self.openai.embeddings.create(
            model="text-embedding-3-small",
            input=question,
        ).data[0].embedding

        self.collection.add(
            ids=[f"{int(time.time())}_{hash(question)}"],
            embeddings=[embedding],
            documents=[answer],
            metadatas=[{"model": model, "question": question, "created_at": time.time()}],
        )

How to Measure Cache Effectiveness?

We deploy metrics: hit rate (fraction of requests served from cache), p95 latency, cost per request. Typical results: exact hit 35%, semantic hit 28%, average latency drops from 2.3s to 0.4s. We use dashboards with alerts when hit rate falls below threshold. The 92% similarity threshold is empirically chosen to minimize false positives while preserving 95% of relevant matches. At threshold 0.95, hit rate drops 12%; at 0.90, incorrect responses increase.

Case Study: FAQ Bot with 5000 Queries/Day

One of our clients is a technical support service. Before caching, all requests went directly to GPT-4o. Results after configuring combined cache:

Savings of 63% — just from caching, without changing the model. In monetary terms, that's over $2,100/month. Get an engineer consultation — we'll calculate savings for your project.

What Our Work Includes

1. Analytics — audit your requests, identify patterns, estimate potential hit rate.
2. Architecture — choose stack (Redis / Chroma / Qdrant), design cache schema.
3. Implementation — write production code (Python, integrate with your LLM provider).
4. Testing — A/B test with latency and cost measurements on your data.
5. Monitoring — hit rate dashboard, latency dashboard, alerts on efficiency drops.
6. Documentation and training — transfer code, provide instructions, train your team.

Estimated Timelines

• Exact cache (Redis): 0.5–1 day
• Semantic cache (Chroma + embeddings): 2–3 days
• Full production solution with monitoring: 1 week

We guarantee stable cache operation, post-implementation support, and transparent reporting. Order a project audit — we'll evaluate hit rate and savings in 1 day.