Why HyDE for RAG Is Bottlenecked by Query Quality
Knowledge base search compares the query vector with the vectors of pre-chunked fragments. The user arrives with a short phrase — two or three words, internal jargon, an abbreviation from a ticket.
There is too little meaning in such a query: the vector comes out blurred and equally “close” to dozens of topics at once.
The second reason is deeper — the query and the document are written differently.
The base contains statements: “log retention period is 30 days when audit is enabled.” The query contains a fragment: “logs how long.” The similarity between them is low even when the meaning matches completely, and rare formulations — industry terms, names of internal systems, typos — fail even worse.
Then the top-of-results effect kicks in. Search is required to return a fixed number of fragments, and when there are no relevant ones, it pulls in formally similar ones: the same words, a different context.
These are false matches in the results — the generator takes them as a source and confidently builds an answer that is not in the base. The loss of relevance here is not cosmetic: the answer looks substantiated but relies on someone else's section.
HyDE for RAG breaks exactly this link. What goes into search is not a short query, but a generated hypothetical document — in form and vocabulary the same as real fragments of the base.
The comparison is “document to document,” and the quality of knowledge base search stops depending on how the user formulated the question. We break down how this works and where it breaks.
Effect of Hypothetical Document Generation
The first measurable shift is retrieval recall. A hypothetical document generated for a specific wording of the question lands in the same vector space as the real fragments of the base, so pieces that direct search on a short query did not find rise into the top of the results.
The share of questions where no fragment crossed the similarity threshold falls — this is exactly the search failure we are removing.
The second metric is the share of answers grounded in a source. When retrieval became more complete, the model more often cites the found fragment rather than constructing the answer from general knowledge.
We measure it by labeling: an answer counts only when its statements match the source text, so the figure reflects the real situation, not the model's self-assessment.
The third is stability on rare formulations. A one-word query, with a typo, in mixed language, or in colloquial form previously gave different results from run to run.
The query expansion effect smooths this out: the hypothetical document adds context and synonyms, and the answer stops depending on exactly how the user formulated the question.
It is worth looking at the breakdown by question types, not the average value. For frequent queries, the gain is modest — direct search was already coping there.
The main dynamics come from the long tail: narrow terms, internal names, rare combinations of conditions, where retrieval used to be empty.
We measure on a fixed set of questions with reference fragments, with the same answer generation model and the same cutoff threshold — otherwise a before-and-after comparison proves nothing.
HyDE RAG Modes: What to Apply and When
The modes differ in the point at which the hypothesis appears and in how many queries go into the index. The choice is influenced not by “model strength,” but by the query type: a colloquial formulation and a part-number search require opposite logic. Below is the fork by which we route the query stream.
| Mode | When it fits | Limitations |
|---|---|---|
| Hypothesis before search | Short colloquial questions without terminology | Adds noise on identifiers and codes |
| Hypothesis after the first pass | Multi-part questions, rare vocabulary | More expensive in response time, convergence control needed |
| Hybrid with reranking | Mixed stream, heterogeneous documents | Requires tuning weights and candidate selection depth |
| Without expansion | Exact queries: numbers, part numbers, verbatim quotes | Does not help when the wording diverges from the document text |
How to Choose a Mode
Expansion under HyDE pays off where the user's words and the document's words do not overlap: the hypothesis transfers the meaning of the query into the vocabulary of the corpus, and retrieval recall grows.
On exact queries, the effect is the opposite — synthetic text pulls the vector away from a literal match, so for identifiers we disable expansion and work by direct indexing.
We keep the hybrid mode as the baseline: reranking insures both branches, raising relevant items from both expanded and exact retrieval. The price is the latency of the second pass and weight tuning, so candidate selection depth is limited to the target response time.
The limitation of any mode is the lack of quality control over hypotheses: weak generation confidently sends search in the wrong direction. Therefore, we test the fork on the customer's real queries and fix the switching rule before production operation.
Implementation Stages: Audit, Design, Validation, Handover
Implementation starts not with code, but with measurement: until there is a figure for current relevance, any tuning of the hypothesis generator is a matter of taste.
Each stage yields a verifiable result and input for the next, and the output leaves a working solution and regulations for your team.
- Search audit. We analyze the corpus and real queries, looking for failures: short formulations, internal jargon, questions with no word matches. We take baseline metrics on a holdout set — recall at top positions and the share of answers without a supporting fragment.
- Query processing design. We determine HyDE's place in the pipeline: rewriting the query into a hypothetical fragment, merging with the original query, metadata filters, cutoff threshold. We agree on the design before implementation — reworking the storage at this step is the most expensive.
- Hypothesis generator tuning. We select the length and style of the hypothetical document, the number of variants per query, and the wording of the instruction. We change parameters one at a time: lengthening the hypothesis often raises recall and lowers precision.
- Validation on a holdout set. We run the variants on queries that were neither in tuning nor in examples and compare with the baseline version. We keep the configuration that wins on diverse formulations, not on the average.
- Regression loop. We lock in a set of control queries so that future corpus edits do not break quality unnoticed, and preserve the ability to return to the previous configuration.
- Solution handover. We deliver the design, configurations, control queries, and document update regulations, train the team, and stay in touch during independent operation.
Delivery Package: Designs, Configurations, Regulations
We hand over not only a working service, but a package with which your team can develop the solution on its own. Each artifact is tied to a query processing step: when the corpus or model changes, it is clear exactly what to revisit.
- Query processing design. The path from question to answer: hypothetical document generation (HyDE), index search, fragment selection, context assembly, model call. With failure points and system behavior when no relevant fragments are found.
- Search configurations. Chunking parameters, number of candidates before and after reranking, cutoff thresholds, field weights, hypothetical document length — with an explanation of the chosen values.
- Source selection rules. Which collections go into the index, what is excluded, how duplicates and outdated revisions are handled.
- Index update design. Schedule for full and incremental reindexing, handling of changes and deletions, requirements for source availability.
- Solution documentation. Architecture, service composition, environment requirements, startup procedure, known limitations, and applicability boundaries.
- Versioning and rollback. How configuration versions are stored, how to return to the previous set of settings and compare quality before and after a change.
- Quality re-evaluation regulations. Reference question set, labeling methodology, metrics: share of answers grounded in a source, citation completeness, run frequency, and the threshold after which we record degradation.
The package sits in your repository and is updated together with the solution. Without it, search remains a black box: quality can be improved only by a full rebuild.
Case Study: Accuracy Gains on Rare Queries
Internal knowledge base of a service company: regulations, ticket histories, technical notes. Vector search returned top chunks by the question embedding, and on frequent formulations this worked.
On rare ones — a narrow term, a specific combination of conditions — the question and document were lexically far apart, the relevant fragment did not make the top, and the model answered in general terms or admitted there was no data.
We implemented HyDE: before search, the model generates a hypothetical answer, and the vector of this draft goes into the index.
The construction was rewritten for real employee formulations, the hypothesis was truncated to one paragraph so noise would not derail retrieval, and a filter by base section and a similarity threshold were added — below it, search honestly returns an empty result instead of random chunks. The method's foundation is Precise Zero-Shot Dense Retrieval without Relevance Labels.
What Shifted
Answer accuracy on rare queries grew from 54% to 81%, and source confirmation appeared in 9 out of 10 answers. The share of “no data in the base” refusals fell by a factor of three, and the share of answers marked by employees as useful went from 61% to 88%.
Answer preparation became more expensive by fractions of a second, so the hypothesis is built by a lightweight model with caching by normalized question.
We tested on a labeled set of 400 real requests that the model had not seen: the gain was specifically on long rare formulations; on short ones, the method matched ordinary vector search — therefore both modes coexist in operation.
How HyDE Differs from Query Reformulation
Query reformulation and expansion work within the original meaning: we add synonyms, expand abbreviations, and remove extra clarifications before sending to search.
Synonym search is the same technique, only through a domain dictionary: it covers known spelling variants, but is powerless when the user describes the task in their own words and the document is written in the language of a regulation or instruction.
HyDE changes not the formulation, but the object of comparison.
The model generates a short hypothetical answer fragment, we compute its vector representation, and search the index for the nearest real documents — the matching goes along the line “answer ↔ document,” not “question ↔ document.” This is a different shift: paraphrasing remains in the vocabulary of the query, while hypothesis generation transfers us into the vocabulary of the corpus, so it finds things that lexically barely overlap with the query.
The choice of strategy is a matter of cost and risk, not fashion. Expansion and synonyms cost one dictionary lookup, work in milliseconds, and are predictable, so they almost always remain the base layer.
Hypothesis generation adds a model call before search: it is justified where queries are short and colloquial, and the corpus is large and written in formal language.
The risk of HyDE is that the hypothesis pulls the vector aside if the model confidently invents a non-existent detail.
Therefore, it is usually placed next to paraphrasing, not instead of it: we compare the retrieval of both variants, and if the hypothesis gives low similarity to documents, we fall back to the original query. We break down how this is tuned for a specific corpus.
Limitations of the Approach and Technical Concerns
The hypothesis does not improve search by itself — it substitutes the query. The generator completes a plausible answer fragment, and the vector is built from it.
If the model imagines a fact that is not in the base, retrieval goes off course: the answer comes back confident but not grounded in a source, and this is not visible in the logs — the query itself looks normal.
The second limitation is latency. Every question first passes through the generator, and only then does search begin, so response time grows by one generation.
In a stream, this is treated with hypothesis caching by normalized query, length limitation, and pre-filtering: short exact queries — document number, part number, surname — go directly into the index.
There are also cases where expansion hurts: narrow factual questions, identifier search, bases of short structured records. Therefore, we enable the method not globally, but via a router by query type, with a switch to ordinary search without rebuilding the index.
The generator will invent a fact — will it end up in the answer?
The hypothesis participates only in constructing the query vector; the answer receives the texts of the found documents.
We set a similarity threshold and output a link to the source, so an invented fragment cannot become the answer — it can only worsen retrieval, and this is visible in the measurement.
How can we tell that expansion hurts on our data?
We assemble a labeled set of questions and compare the share of top hits and answer completeness with and without expansion. If the gain stays within noise or drops, we do not enable the method on this class of queries.
How much does the load on the generator increase?
Response time grows by one generation per query, and the model itself works with the same stream as its other tasks. We smooth it with hypothesis caching, batch processing, and a queue with degradation to ordinary search if the generator is unavailable.
Let’s Discuss the Task and Return with a Work Plan
For the assessment to be substantive rather than “based on feelings,” we need a little: the composition and volume of the corpus, the languages and formats of the sources, how search is currently arranged, and which queries users consider failures.
That is enough to understand whether hypothetical document generation will yield a gain specifically on your data — it helps where the wording of the question is far from the wording in the texts.
At the start, we propose the following order of steps:
- Task review. We look at real queries and reference answers: without agreement on what counts as a correct answer, any quality measurements are meaningless.
- Validation slice. We run current search on your documents and record the baseline — the share of queries where the needed fragment made the top of the results.
- Hypothesis validation. We compare retrieval with and without hypothetical document generation, and separately calculate the cost: how much time it adds to the answer.
- Work plan. Stages, control points, areas of responsibility on your side and ours; we close risks around data and access before the start.
- Team composition. Search engineer, model specialist, data and infrastructure engineer — one person is responsible for the result.
- Implementation start. Pilot on one segment of the knowledge base, effect measurement, then rollout — so as not to rewrite the solution after the first release.
Describe the task: what the knowledge base is, who uses it, and what is getting in the way now. We will return with an estimate, work plan, and team composition for your scope.
