We often see LLMs confidently generate fabricated facts: "Drug X is FDA-approved" — yet the drug doesn't exist, or a RAG citation points to a non-existent page. This is not random; it's a consequence of the autoregressive nature of models: the next token is predicted from a distribution, not from truth. For business-critical systems, this is unacceptable. Our team has developed a multi-level system for detecting false claims—hallucinations—proven in production. With over 5 years in NLP and MLOps, we have deployed more than 20 projects involving RAG and error detection.
Why Standard Methods Fail
The issue isn't that the model "doesn't know" — it's that GPT-4, Claude, Llama, and their peers lack an internal verification mechanism. The model doesn't know what it doesn't know. The confidence score from logprobs weakly correlates with factual accuracy: you can get a logprob near zero for a hallucinated fact. There are three main sources of hallucinations. First, mismatch between retrieval and generation: chunk_size=512 with no overlap, FAISS with L2 metric, weak embedding model. Second, temporal drift: the model was trained on data up to a certain date. Third, the usefulness-accuracy trade-off during RLHF. Our experience shows that 70% of cases stem from the first source.
How to Build a Hallucination Detection System
Hallucination detection cannot be solved with a single method. In practice, we use a multi-level architecture:
Self-Consistency Check
We generate N responses to the same question with temperature > 0 (typically N=5–10, temperature=0.7). We compare responses semantically using sentence-transformers (paraphrase-multilingual-mpnet-base-v2). High variability indicates unreliable facts. Self-consistency provides three times more accurate reliability estimation than logprob analysis.
Grounding Score
For RAG systems: we verify whether each claim in the response is supported by retrieved chunks. We use an NLI model (cross-encoder/nli-deberta-v3-base) to evaluate entailment between the response and context. Claims with an entailment score < 0.6 are flagged as unverified. The grounding score is more accurate than simple keyword checking. NLI verification is 40% more effective at detecting hallucinations.
Retrieval Faithfulness
RAGAS metrics (RAGAS: Automated Evaluation of Retrieval Augmented Generation) Es et al., 2023: faithfulness, answer_relevancy, context_precision. Faithfulness < 0.7 with context_precision > 0.8 means the context was present but the model ignored it.
External Fact-Checking
For critical domains (medicine, law, finance): verification via search (Tavily, Bing Search API) or specialized knowledge bases (Wikidata SPARQL, PubMed API). Claims containing named entities are processed through NER (spaCy + custom model) and each entity is verified separately.
Step-by-Step Implementation Guide
- Audit your current RAG pipeline: analyze chunk strategy, embedding model, and prompts. Collect a ground truth dataset of 100–200 real queries.
- Baseline measurements: overall hallucination rate, faithfulness, and latency p99.
- Select methods: for simple scenarios, self-consistency is sufficient; for critical ones, combine grounding score and external verification.
- Integrate the detector as middleware with logging to Grafana.
- Monitor and calibrate thresholds on a dataset of 100–200 queries.
Detailed Audit Checklist
- Evaluate retriever quality: precision@k, recall@k
- Analyze chunk strategy: size, overlap
- Check embedding model: dimensionality, cosine similarity
- Audit prompts: presence of accuracy instructions
- Manually label 100–200 queries for ground truth
Practical Case Study
A client — a law firm with an internal assistant for case law. Model: GPT-4-turbo with RAG on 50k documents (pgvector + LangChain). Problem: 18% of responses contained references to non-existent cases or incorrect decision dates (identified through manual audit of 200 queries).
Solution: We added two-level verification. At the retrieval level, a reranker cross-encoder/ms-marco-MiniLM-L-6-v2 raised context_precision from 0.61 to 0.84. At the generation level, NLI verification of each legal claim plus regex extraction of case numbers followed by verification against a database of arbitration decisions via API. The hallucination rate dropped to 3.2% within two weeks of iterations. Savings on manual verification exceeded $9k–13k per year.
Metrics for Detection Quality
| Metric | Tool | Target Value |
|---|---|---|
| Hallucination rate | Manual audit + NLI | < 5% for production |
| Faithfulness (RAGAS) | ragas library | > 0.80 |
| Grounding score | NLI deberta | > 0.65 per claim |
| Self-consistency | sentence-transformers | cosine sim > 0.75 |
| Latency overhead | — | < 500ms per detection |
Comparison of Detection Methods
| Method | Accuracy | Latency | Application Domains |
|---|---|---|---|
| Self-consistency | Medium | +200ms | Any |
| Grounding score | High | +100ms | RAG |
| External fact-checking | Very high | +1–3s | Medicine, Law |
What the Work Includes
- Audit of the current pipeline: retriever quality, chunk strategy, embedding model, prompts.
- Baseline measurement: hallucination rate, faithfulness, latency.
- Selection and configuration of detection methods specific to the domain.
- Integration of the detector as middleware into production.
- Monitoring: Grafana dashboard, alerts on metric drift.
- Documentation and team training.
Implementation cost is determined after an initial audit of your system — contact us for a tailored proposal.
Implementation Process
Current State Audit — analyze the existing pipeline: retriever quality, chunk strategy, embedding model, prompts. Collect a dataset of 100–200 real queries verified against ground truth.
Baseline Measurement — obtain numbers: hallucination rate, faithfulness, latency. Without baseline, it's unclear what to improve.
Multi-Level Detection — select methods per domain specifics. Medicine requires external verification; internal company knowledge may only need grounding score.
Pipeline Integration — embed the detector as middleware. Responses with low grounding are flagged with warnings or sent for human review.
Production Monitoring — log all scores, build a Grafana dashboard. Drift in metrics signals a need for reindexing or prompt strategy changes.
Timeline: from 2 weeks for adding detection to an existing RAG pipeline to 2 months for a full verification system with external sources in a complex domain. Eliminating hallucinations in production reduces manual answer verification costs.
Contact us to assess your project and receive a tailored proposal.