Parent Document Retriever for RAG: improving context recall by 19%

Parent Document Retriever: Where Answer Context Is Lost

A 40-page document is cut into fragments of 300–500 tokens with overlap — otherwise search does not find exact matches.

The top 3–5 fragments go to the generator, and the answer is assembled from scraps: the condition in one piece, the exception to it in another, and the definition of the term left outside the selection boundary. Formally, each fragment is relevant to the query; in fact, the output is incorrect.

Retrieval completeness falls not because of the model, but because of chunking.

The relevance of a fragment is not equal to the completeness of the source section: a paragraph with a keyword makes it into the selection, while the neighboring one, without which it is read incorrectly, does not.

On long documents — regulations, contracts, instructions — up to 19% of needed information is lost this way, and this is no longer a rare case but a systemic property of the scheme.

The cost of failure does not manifest immediately. The first answers look convincing, nobody goes to check them, and later a lawyer or engineer compares the output with the source and finds a missed exception.

After two or three such findings, the system stops being used — and implementation rolls back to manual document search.

This is solved by changing the unit of search. Parent Document Retriever searches by small fragments, but returns the parent section as a whole into context: match precision is preserved, coherence is not broken, and long documents stop falling apart into semantic scraps. We analyze how this works and which chunking parameters will have to be rebuilt.

What Hierarchical Search Provides: Effect and Applicability Boundaries

More Relevant Context in the Answer

The Parent Document Retriever scheme separates search and delivery: comparison is by small fragments, while the parent block goes to the model as a whole — with heading, table and caveats.

The share of relevant fragments in the output grows by 15–19% on technical documentation corpora. A small fragment accurately catches a semantic match; a large one does not lose conditions without which the answer is formally correct and essentially empty.

Response Latency Remains in the Same Class

The vector index is built on small fragments, so the number of comparisons does not change. The additional step — reading the parent block from a key-value store — drowns against the model call: the difference is visible in measurements, but not in user experience.

Storage Volume Grows Only in Text

The vector part is calculated once over small fragments and remains the same. Only the text part grows: parent blocks are stored in their original form, the total increase usually fits within 1.5–2 times and then scales with the corpus, not faster.

Where Hierarchical Retrieval Does Not Win

Short homogeneous documents without structure — a one-page regulation, a product card — have nothing to group: the parent block coincides with the fragment itself.

The technique reveals itself on long texts with an explicit heading hierarchy, where context is scattered across sections.

Standard Retriever vs Hierarchy: Comparison of Approaches

Single-level search cuts the document into chunks and returns the top-k nearest by embedding. The scheme is predictable while the answer lies within one block — enough for FAQ and short articles.

When the answer is assembled from several sections, chunking limitations are visible immediately: tables and lists break at boundaries, neighboring blocks lose connection.

Document hierarchy changes the search level: we find small blocks, but return the parent fragment as a whole into context. Hybrid schemes supplement the vector with lexical search and a reranker. Below — how these options differ in applicability and what each outputs.

The choice is determined by measurement, not by a description of the approach: we collect questions whose answers are scattered across the document, and calculate context recall before and after the scheme change.

In practice, hierarchy and hybrid yield gains where single-level search consistently returns neighboring but not needed fragments.

Implementation Stages: From Corpus Audit to Handover

The order of work here is set not by convenience but by result quality: Parent Document Retriever stores two levels — small fragments by which search runs, and parent documents that go into the answer.

If you start with tuning instead of corpus analysis, parents will be heterogeneous, and context completeness will drop on the most frequent query types.

1. Corpus audit. We count volume and languages, the share of scans without a text layer, duplicates and outdated editions, and mark document types. Output — a map: what to split, what to clean, what to exclude from search.

2. Level scheme. We fix the hierarchy “fragment → section → document”, metadata inheritance and the level supplied to the answer. We agree before tuning — otherwise splitting will have to be redone together with reindexing.

3. Search tuning. We select fragment size and overlap, hybrid search weights, the number of candidates before and after reranking. We put parameters into configuration, not code.

4. Validation on holdout set. We prepare questions not involved in tuning and run them on final parameters. This shows where the answer relies on a retrieved document and where the model answers from memory.

5. Failure analysis. We attribute each error to a cause: parent not found, found partially, metadata interfered, question outside corpus. Next — fix splitting, replenish corpus, or explicit refusal to answer.

6. Solution handover and team training. We hand over configurations, level scheme, reindexing procedure, validation question set and corpus update regulation. We review everything with your engineers so work continues without us.

What the Team Retains: Schemes, Policies, Regulations

Implementation ends not with a “working search” but with a set of artifacts by which the loop can be changed, rolled back and handed to another team.

Below — what exactly is fixed in the repository and alongside it so output quality does not rest on one engineer who remembers why parameters are set exactly this way.

• Search loop configuration — in versioned files: chunk size and overlap window, parent block assembly rule, how many candidates are sent to reranking, weights when mixing lexical and vector search. Any edit passes mandatory review and enters change history.
• Block relationship scheme — how a chunk references the parent document, what metadata is stored alongside (source, section, actuality date), which fields participate in filters. With example queries so relationships do not have to be reconstructed from logs.
• Access policies — role-based differentiation at chunk and parent-document level: who sees restricted sections, who only public ones. The check is built into the index query itself, not post-processing of ready output.
• Index update regulation — reindexing frequency, behavior when a source document is deleted or edited, rollback point to the previous version. Separately — divergence metric between source and index.
• Validation question set — a fixed list with reference answers and labels of needed fragments. On it, after each configuration edit, output completeness is measured, not “seems better”.
• Error analysis regulation — which queries are considered search failure, how they enter the validation set, who and how often rebuilds it. Plus the owner responsible for the decision on each failure category.
• Documentation and handover — description of the loop scheme, extension points, local run instructions, decision log with rationale for chosen parameters. A new engineer reaches working launch without the author's support.

Case: Context Retrieval Completeness Grew by 19%

Corpus — normative documentation: regulations, methodologies, orders with appendices and cross-references. Before implementation, search ran over fixed-length fragments: the needed clause was found, but caveats, scope and exceptions from neighboring paragraphs were lost. Retrieval completeness held at 0.68 — roughly every third query required manual tweaking.

We moved to two-level retrieval: search runs over small fragments, and the parent section as a whole is inserted into the model context — the Parent Document Retriever approach.

The corpus was rebuilt into the hierarchy “clause → section → document” with metadata on edition and scope: filters by them cut off revoked editions, so they no longer get mixed into output.

Child fragment size was selected to match text structure, not a fixed character count: a clause with subclauses remains whole. The upper context bound was limited so response time does not grow linearly with document length.

Metrics Before and After

Retrieval completeness grew from 0.68 to 0.81 — plus 19% to baseline on the same query set. The share of answers with a reference to a revoked or nonexistent clause fell almost threefold.

User behavior changed more noticeably than the metrics themselves: clarifying questions within a session decreased by a third, and the share of cases where an employee opened the primary source after an answer halved.

People check a link when they are in doubt — the drop in this share is a sign of trust in the output.

On short reference queries, the gain is almost unnoticeable — ordinary search is enough there. The gain is visible on compound questions: the answer is assembled from several sections, and every missed clause matters.

What to Do if the Corpus Is Already Split into Small Fragments

While search works, full reindexing is not mandatory. Vectors of small fragments remain valid — only the metadata layer changes: to each fragment we append a parent identifier (document path, section heading, block ordinal). Such a pass recalculates relationships, not vectors, and takes hours instead of days.

Parent text is pulled in at retrieval time: the query hits the exact small fragment, and the entire section from which it was cut goes into the model context. This is Parent Document Retriever in applied form — search by small, answer by large.

If the parent field is not yet in the schema, it is added by metadata migration without stopping search.

Zero-downtime migration is built on two parallel indexes. The old one serves traffic, the new one fills with relationships in the background, and switchover happens by matching output on a control query set. Rollback — return the pointer: the previous index lives until validation ends.

A hybrid scheme is needed when the corpus is heterogeneous. New documents are immediately indexed as “fragment — parent” pairs, the archive is built up by a scheduled crawl, so fresh arrivals do not wait for processing of the entire array.

Full reindexing is justified only with broken fragment splitting: pieces tear sentences, there is no overlap, metadata carry no structure.

Result control — context recall on a holdout set of real queries and the share of answers where the parent block fully fit into the context window.

Common Engineer Questions: What Breaks in Practice

The method closes a specific class of failures: a short fragment enters the index, and the answer loses details that were in the document.

Parent Document Retriever searches by small fragments, but supplies the parent document as a whole into context — in our measurements, retrieved context completeness grew by 19%. The scheme also has a cost, and engineers ask about it first.

How much will storage volume grow?

It grows because the parent lies alongside the fragment. Relative to a scheme with fragments only, this is a one-to-twofold increase; the exact amount depends on parent length and overlap during chunking.

It is controlled by configuration: the parent is stored as one instance, fragments reference it, and rare documents are moved to cold storage and do not occupy the fast index.

How to measure quality if there is no reference labeling?

A full set of questions with answers is expensive and quickly outdated. We take an indirect metric: generate questions from the documents themselves and look at the share of cases where the needed fragment reached the top of the output and its parent arrived in context.

Additionally, we check on small manual labeling — a few dozen pairs are enough to catch a systemic failure, but too few to claim the method holds on rare queries.

Let's Discuss the Task: How We Arrive at an Implementation Plan

The implementation plan is built not on general words but on your facts: without them, any timeline estimate becomes guesswork. Therefore, we start the task discussion with a short list of what is needed from your side, and immediately fix what we return in exchange.

• A sample of real user queries for the recent period and documents on which these answers should have relied — on it we check whether search returns the parent fragment as a whole, not a phrase snippet.
• A list of places where it currently goes wrong: support requests, manual operator edits, repeated clarifying questions.
• Rules of access to database sections — who sees what and what cannot be taken outside your loop.
• An expert on your side who confirms the correctness of answers on the validation set and resolves disputed cases.
• Numerical acceptance criteria: the share of queries where the needed fragment made the selection, and the share of answers with a source reference.
• Environment constraints: deployment loop, database volume, its update frequency and synchronization method.

In response you receive a timeline estimate by stages, an implementation plan and a pilot launch procedure: first a search prototype on your own sample, then a pilot in one department with metric measurement, and only after that — expansion to other teams.

Each stage ends with a verifiable result and a rollback point to the previous index state, so the solution can be stopped without losing already working search.

Describe the task — we'll come back with an estimate and plan.