RAG Weaviate: why search does not find the needed answer
While the corporate knowledge base is small, search over it relies on luck. Then a typical breakdown begins: an employee asks about shipping regulations, and gets receiving instructions — words match, meaning does not.
After two or three such answers, the assistant is shut down and people return to manually browsing folders on a network drive.
The cause is almost always in data preparation, not the model. Documents are cut into fixed-length chunks, boundaries fall in the middle of a table, vector representations are computed on "raw" text without metadata — department, dates, document type.
The vector database honestly returns the nearest vectors, but nearest does not mean correct: relevance of results is determined by the storage schema and filters, not index size.
The second layer — selection and reranking. Without a filter by validity period and department, canceled versions get into the answer; without reranking — chunks where the needed word is mentioned in passing.
In Weaviate this is closed by collection schema, hybrid search and selection rules, but the schema is designed for real employee questions, not by default.
RAG development therefore begins not with model choice, but with inventory of the corpus and metadata labeling. Losses here are measured not in licenses, but in trust: an assistant that errs in a third of queries stops being used, and the knowledge base becomes an archive again.
We break down how this is closed — from document chunking to evaluating relevance of results on real formulations.
What managed search gives: response speed and result completeness
The first effect — time to answer. Hybrid search in Weaviate combines term matching and semantic proximity, so results do not depend on whether the user guessed the wording.
On a warmed index, the answer is assembled in seconds instead of minutes of manual search through folders, chats and email.
The second — result completeness. Document chunks are cut with overlap, and each is tied to metadata: department, date, access level.
This removes a whole class of failures "document exists but search does not see it" — the regulation is found by meaning even if the question contains none of the words from its title.
The third — support load. When an answer has a link to the primary source, typical questions are closed without an operator, and repeat requests on the same topic decrease: the person gets an answer immediately, not after correspondence. The share of requests without manual intervention is a metric we put into a regular report.
The fourth — answer quality becomes measurable. We record which chunks made it into the context and whether the answer relies on them, and run knowledge base changes against a set of control questions. Thus edits to documentation do not silently break results.
RAG Weaviate modes: vector, hybrid, with reranking
Search mode is chosen by corpus composition and by how users phrase the query, not by default. Vector search matches meaning: it will find the section on contract termination by the phrase "how to opt out of cooperation", but will miss an article number or error code.
Hybrid search adds lexical matching by exact terms and weighs the contribution of both channels — in Weaviate this is configurable channel weight.
| Search mode | When suitable | Limitations |
|---|---|---|
| Vector | Free formulations, paraphrasing, synonyms, conversational questions | Does not find article numbers, codes, and rare abbreviations; depends on embedding quality |
| Hybrid (vector + lexical) | Mixed vocabulary: regulations, catalogs, technical documentation with codes and model names | Channel weights tuned to corpus; ranking harder to explain |
| Hybrid + reranking | High cost of error: legal and technical answers, long multi-part queries | Second stage adds latency and resource consumption; target chunk must be in candidate pool |
Modes do not exclude each other: hybrid search is reasonable to keep as baseline, and reranking enabled for individual scenarios or only for long queries.
We fix the choice by measurements on a set of real questions — calculate the share of queries where the target chunk appeared in the top three positions, and only then lock the setting in production.
Implementation stages: from corpus audit to handover to operations
Search implementation breaks not on the model, but on data: duplicates, outdated revisions and scans without a text layer produce the most expensive errors in answers.
Therefore the order of work is fixed — first audit of the document corpus and reference questions, then knowledge base indexing, and only then search implementation in the interface. The team gets a measurable acceptance criterion instead of "seems to answer correctly".
- Document corpus audit — inventory of sources: formats, owners, validity period, access rights. Separately mark duplicates and outdated revisions.
- Labeling control sample of questions — 100–200 real questions with correct answers and indication of the source document. It also becomes the acceptance test.
- Storage schema — index structure in Weaviate, chunk size and overlap, fields for filtering by departments, dates, and access rights.
- Setting up knowledge base indexing — scan recognition, chunking into fragments, vector computation, loading only changed documents on schedule.
- Validation on control sample — share of answers with correct source link and share of correct refusals when data is absent from the corpus. We agree the threshold in advance.
- Implementing search into operational loop — fragment-level access, request and source log, handover of regulations and training the team to update the corpus without us.
Handover package: configurations, access schemas, regulations
We close work not with a running loop in the working environment, but with a handover package for operations. Using this documentation, your team can change the corpus, add scenarios and investigate incidents itself — without contacting us for every clarification.
- Configuration descriptions. Collection schema, document chunking rules, vectorization model, relevance threshold and number of returned chunks — with explanation of why these values were chosen and what will happen if they change.
- Request processing schema. Path from user question to finished answer: where the vector is built, how rights restrictions are applied, how context is assembled and where it is truncated by length.
- Role-based access policies. Which collections and fields each role sees and how restrictions are applied at search stage so they cannot be bypassed by wording of question.
- Corpus update regulation. When vectors are recalculated, what happens when a document is deleted or replaced, and how it is checked that outdated versions did not get into search.
- Control set of questions. Reference pairs "question — expected source": hit rate on them is measured after every corpus update or model change, so quality drop is visible before user complaints.
- Rollback procedure. How to return the previous state of the search index and settings if answers got worse after update.
- Request investigation. Where to look when there is a complaint about an answer: search logs, saved contexts, response time on typical queries.
The package remains with you as files in a repository, not in correspondence. Describe the task — we will return with an estimate and plan, including composition of such a package for your environment.
Case: how answer accuracy grew in the knowledge base
Starting point
Support service knowledge base — several thousand documents: regulations, instructions, incident analyses. The operator searched by keywords and manually paged through results, because client wording almost never matched wording in the regulation.
On a labeled sample of real requests, the share of correct hits into the target document stayed around 40%, and finding an answer took 4–6 minutes — some requests by that time had already gone to the second line.
What changed
Documents were split into fragments by semantic boundaries, their representations recalculated and stored in Weaviate.
Search was made hybrid: vector proximity plus lexical match by terms and article numbers, with filters by access rights and product line, so the operator did not receive other departments' sections.
Reranking was placed on top — top results include fragments from which the answer is actually assembled, not just similar in topic.
A month after rollout, accuracy of hitting the required document grew from ~40% to 85%, answer search time dropped to 40–50 seconds including reading the found fragment.
The share of requests that the operator closed independently, without transfer to the second line, rose from 55% to 80%.
We checked not on showcase examples, but on a holdout sample of closed requests with a pre-labeled reference document: measured hit in the first and third result, separately analyzed cases where the document was outdated or contradicted a neighboring one.
This set remained in the project as a regression one — before any change to search settings it is visible whether it got better or worse, and rolling back parameters does not become guesswork.
What to do if search already works on another solution?
If search already works, migration is not rewriting from scratch, but transfer with contract preservation: same queries, same metadata fields in response, same access rights.
The first step — inventory of corpus and settings: how documents are chunked, which filters participate in queries, what the request log shows. Often already here it is visible that part of the corpus was not indexed at all, and access check followed one scenario.
Next — index transfer. Weaviate is stood up alongside the old system, indexing runs on a copy of the corpus, existing search serves users in previous mode.
Key risk — vector space shift: if vector construction models do not match, they cannot be transferred directly, full reindexing is needed. Therefore model version and chunking rule are fixed in advance, otherwise results will drift at fragment boundaries.
Parallel validation runs on a control set — several hundred real questions with known source answers. We compare recall and precision, share of answers without document grounding, response time on long queries. Separately we run access filters and rare formulations: discrepancies surface exactly there.
Switchover is done without downtime: first a small share of traffic goes to new search, metrics are compared on live queries, then the share grows. Rollback — one switch, the old system remains available until quality is confirmed. This way search migration stops being an event with downtime risk and becomes a managed process.
Technical doubts: chunking, access, index updates
Five questions a technical lead asks before start — and the answers determine whether knowledge base search will be accurate or turn into a generator of confident errors.
There are no universal values: chunk size and update rules depend on the corpus. Therefore we first look at real documents and user questions, and then fix parameters.
What chunk size to choose?
We start from 300–800 tokens with 10–15% overlap: small chunks lose context, large ones blur relevance. We tune the value on 50–100 real questions.
What to do with tables and scans in documents?
We store a table as a separate chunk together with row and column headers — otherwise numbers lose meaning. We recognize scans and check them on control pages: without this part of the corpus is not found.
How often to update the index?
Incrementally: compare document hash and reindex only changed, full rebuild — when parameters change. We remove deleted document from the index, otherwise the answer follows an outdated version.
How to delimit access rights to data?
Rights check — at search stage, before generation: each chunk stores metadata about role and department. Only permitted content enters the model context, leak through the answer is excluded.
How to understand that the solution got better or worse?
Control set of questions with reference answers and sources is run on every parameter change. We look at hit rate into the required chunk and answer completeness — regression is visible before deployment.
Let's discuss the task: what to prepare for the first conversation
The first conversation is needed not for introductions, but to fix source data. Analysis of the corpus, query classes and access constraints — without which any estimate turns into a range with double margin. Below is what to prepare to discuss the task substantively and immediately get to business.
- Corpus examples: several hundred real documents in the form in which they live now — with scans, tables and attachments, not an exported ideal set.
- 20–50 typical user questions and answers considered correct — from them a validation set is assembled to measure relevance.
- Current search method and its weaknesses: where required fragments are found today, and where results go off track.
- Access constraints: which roles have which rights, whether fragment output needs to be separated between departments and clients.
- Freshness requirements: how often the corpus is updated and what lag of index from source is acceptable.
- Languages and formats: mixed documents, handwritten edits, attachments without text layer.
- Acceptance criteria: share of questions whose answer is fully contained in found fragments, and threshold below which the solution is not accepted.
- Deployment environment: isolated environment, infrastructure owner, who maintains deployed search after handover.
With this set we return with an estimate and plan: work stages, corpus composition for vectorization, validation set schema, acceptance points. If part of source data is not yet collected — we will discuss what we cover on our side, and note what will be needed from you.
