A client complains: "We have 10,000 documents in Confluence, but finding an answer is a lottery." A typical situation: employees spend up to 30% of their working time searching for information.
We solve this with a question-answering system based on RAG (Retrieval-Augmented Generation) — an approach combining retrieval and generation. It allows asking questions in natural language and getting precise answers with source citations.
Over 5+ years we have delivered more than 50 NLP projects, and RAG is the primary tool for corporate knowledge bases and intelligent search. According to Question Answering, the QA task consists of extracting or synthesizing an answer from a set of documents.
RAG outperforms extractive QA in summarization tasks — accuracy is 15% higher, and hallucinations are halved. For a legal company with 5,000+ contracts, we achieved an F1 of 82% and a search time of 30 seconds.
Savings on information search amount to around cost calculated after audit per month for a company with 500+ employees. For a company with 2,000+ employees, savings reach cost calculated after audit per month. RAG is also 10 times cheaper than a Long-context LLM at comparable quality.
Why is RAG the best approach for Question Answering?
Extractive QA (models deepset/roberta-base-squad2, sberbank-ai/rubert-base-cased-qa) is good when the answer is an exact quote. But if a question requires summarization or information from multiple documents, extractive QA can't handle it.
Long-context LLM (Claude 3.5, 200K tokens) is simpler, but expensive and doesn't scale beyond 500 pages. RAG is the golden mean: cheap search over vector indexes + LLM answer synthesis. We use it in 90% of projects.
| Approach | Accuracy | Hallucinations | Cost | Scalability |
|---|---|---|---|---|
| Extractive | High (EM ~80%) | Minimal | Low | High |
| RAG | Medium (F1 ~75%) | Moderate | Medium | Very high |
| Long-context | High | Present | High | Low |
What problems do we solve?
On a typical project, clients face three problems:
- Data in different systems. Confluence, SharePoint, Google Drive, 1C — documents are scattered. We build a unified indexer via Airbyte or custom ETL pipelines.
- Tables and scans. LLMs poorly understand complex tables. We use Text2SQL or serialization to Markdown. For scans — Tesseract + layout-parser, which ensures high-quality document recognition.
- High latency. Users don't want to wait >5 seconds. We optimize: embedding caching, batch inference, vLLM for GPU.
How we do it: a legal company case study
For a legal company (5,000+ PDF contracts), we launched a RAG system in 3 weeks. Stack: LangChain + Qdrant + GPT-4o-mini. Result: search time dropped from 15 minutes to 30 seconds, answer accuracy — 82% (F1).
The key thing — we added a Faithfulness check: a separate prompt verifies that every fact in the answer is supported by at least one document. If not — the system writes "The documents do not contain the information."
We selected chunking parameters experimentally:
| Chunk size | Overlap | F1 on the test set |
|---|---|---|
| 256 tokens | 32 | 78% |
| 512 tokens | 64 | 82% |
| 1024 tokens | 128 | 80% |
To improve accuracy, we use hybrid search: dense embeddings (OpenAI text-embedding-3-small) are combined with BM25, and then re-ranking via Cohere rerank v3. This yields an additional F1 gain of 3-5 percentage points.
How do we evaluate response quality?
We create a test dataset of 100+ questions covering typical scenarios. Metrics: F1, EM (exact match), faithfulness (share of responses without hallucinations), latency p95. The target threshold is F1 ≥ 75% and faithfulness ≥ 95%.
If necessary, we fine-tune the retriever (tuning k, choosing the embedding model) or the LLM (few-shot prompts).
Example indexer configuration
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Qdrant
from langchain.chains import RetrievalQA
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Qdrant.from_existing_collection(
embeddings=embeddings,
url="http://localhost:6333",
collection_name="docs"
)
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
return_source_documents=True,
)
result = qa_chain.invoke({"query": "What is the procedure for terminating a contract?"})
How much does QA system implementation cost?
A basic project (up to 10,000 pages, 1 source) — from 3 to 5 weeks. A complex one (multiple sources, tables, scans) — from 6 to 10 weeks. We provide the exact cost after a data audit — write to us, and we will evaluate your case. A typical project pays for itself within 3-6 months thanks to reduced search time.
Work Process
- Analytics: audit of data sources, document types, volumes, request frequency.
- Design: selection of architecture (RAG / hybrid / multi-agent), definition of the chunking, embedding, retrieval pipeline.
- Implementation: data indexing, LLM configuration, integration with corporate systems (Confluence, SharePoint, Telegram bot).
- Testing: creation of a test dataset of 100+ questions, evaluation of metrics (F1, EM, faithfulness, latency).
- Deployment and monitoring: deployment on Kubernetes or Managed ML (SageMaker, Vertex AI), response logging, A/B testing.
What is included in the deliverables
- Document indexer with incremental update support
- REST API for questions (Swagger documentation)
- Web interface (simple chat UI)
- Integration with messengers (Telegram, Slack) — optional
- Metrics dashboard (number of requests, latency p95, failure rate)
- Operations documentation
- Team training (2-3 hours)
- 3-month bug warranty
Our engineers are certified in AWS and GCP, and we guarantee accuracy of at least 75% F1 on your data. Order an audit — we will assess the scope, document types, and implementation timeline. Get a consultation and a commercial proposal within 1-2 days.
