RAG document indexing: where answer quality is lost
Most often, the assistant answers confidently and incorrectly not because of the model, but because of what is in the search index.
RAG document indexing is the stage where quality is lost unnoticed: answers look coherent, source links are present, but the facts in them have already been outdated for a month. Below are four failures that we find in most implementations.
The first is text extraction from PDF. Scans without a text layer produce an empty result, multi-layer files return duplicate paragraphs, tables fall apart into unreadable mush, and headers and footers repeat in every fragment and clog the output.
It is fixed by recognition that preserves structure: headings, lists, and tables are transferred into markup, not into a continuous string.
The second is loss of context during retrieval. If a document is cut into fragments by a fixed number of characters, it breaks in the middle of a table or sentence, and the section heading remains in the previous chunk.
Search finds a fragment, the model completes the answer by meaning—and the result is a plausible fabrication. That is why fragment boundaries are tied to semantic blocks and the section name is added to each chunk so that the fragment is understandable without its neighbors.
The third is outdated file versions in the knowledge base and scattered sources. While files live on a network drive, in a wiki, and in email, copies diverge: an edit is made in one, search returns another, and duplicates compete for a place in the results.
The index must be updated incrementally when a file changes, mark the previous revision as invalid, and store a link to the current source.
Finally, the lack of quality checks: no one measures how many questions the assistant answers based on a document and how many by guesswork. Below we break down how this is addressed.
What managed indexing delivers: the effect in measurable metrics
The completeness of retrieval across a knowledge base is determined not by the model, but by how the corpus is broken down into fragments.
During RAG document indexing, each source is cut by semantic blocks—a clause of a regulation, a table, a section of an instruction—rather than by a fixed number of characters, with overlap at the boundary of neighboring blocks.
A set of metadata travels inside the fragment: source, section, revision, effective date. That is why the needed paragraph ends up in the answer instead of dissolving between two adjacent pieces of text.
Index freshness relies on incremental reindexing: when a file is updated, checksums are compared, only changed fragments are rebuilt, and deleted ones are removed from storage.
Knowledge base update time is measured in minutes from document publication to its appearance in answers, not by a nightly full-corpus rebuild window—storage on Qdrant accepts point edits without stopping search.
The third effect is the elimination of the failure class “answer not based on the document.” Each answer carries a link to the source fragment, and when similarity is low, the system honestly reports that there is no data instead of completing the wording by guesswork. This removes the most expensive scenario: a confident answer based on a revoked revision.
Together, this yields a predictable knowledge base—you can see what it contains, when it was updated, and which document each answer relies on.
Comparison of preparation modes: PDF, DOCX, HTML, and Markdown
The preparation mode determines what ends up in the RAG index: coherent text with preserved hierarchy or a flat canvas without headings. For some sources, text is extracted directly; for others, an additional pass is needed, and some limitations are inherent in the format itself.
| Source | When it fits | Where it hits a wall |
|---|---|---|
| PDF with a text layer | Contracts, regulations, formatted reports | Columns and headers/footers merge into a single stream, tables lose cell association |
| Scanned PDFs without a text layer | Archives, scanned acts, attachments | Recognition and manual review of problematic pages are required |
| DOCX with tables | Instructions, price lists, documents with headings | Styles are arbitrary—heading levels are mapped manually |
| HTML | Documentation, knowledge base, help | Menus and navigation create noise and duplicates before cleanup |
| Markdown | Technical documentation, regulations in a repository | Hierarchy is explicit; relationships between sections are marked up manually |
The main cost is scans. Recognition yields text but does not guarantee reading order on multi-column pages, so problematic pages are reviewed manually and marked. Otherwise, a merged fragment goes into the index, and retrieval based on it answers inaccurately.
In DOCX and HTML, the difficulty is not extraction but cleanup: template navigation and menus create duplicates.
Markdown wins because the hierarchy is predefined—parsing comes down to reading the structure without guesswork. In practice, a corpus is almost always mixed, so modes are combined: one document goes through text extraction, a neighboring one through recognition with review.
We assess the source composition by the share of each format and only then choose the scheme—this determines how many manual reviews to budget for.
Implementation order: from corpus audit to handover
Indexing is not “upload a folder to search” but a chain of decisions, each of which is expensive to change later: a wrong chunking scheme produces ragged answers, and the lack of access rights leads to restricted documents leaking into general results.
That is why the work order is fixed: first we understand the corpus, and only then do we build the index.
- Document corpus audit. We count formats and volumes, look for duplicates and version discrepancies, and separately flag scans without a text layer, nested tables, and multilingual documents. The result is a corpus map with a list of places where text extraction will predictably lose meaning.
- Chunking scheme. We determine fragment size and overlap between neighbors, and cut by structure—headings and sections—rather than by a fixed number of characters. The goal is for the fragment to be self-contained and not to split a table or list in the middle.
- Metadata and access rights. For each fragment, we attach source, date, version, department, and access classification. Rights are inherited from the source storage system, so an employee sees only what they are allowed to open in the results.
- Format-specific extraction setup. We take PDFs via the text layer, scans via recognition, parse DOCX with tables and styles in mind, and HTML and Markdown by heading structure. For each format, we configure its own processing and test it on a sample.
- Search index assembly. We combine semantic search with full-text search by keywords, part numbers, and numbers—otherwise queries with exact designations fail. Reindexing is triggered on a schedule and on document change events.
- Testing on reference questions. We assemble a set of questions with known answers and look at the share of sources found, completeness and precision of results, and answer time. Based on the results, we adjust the chunking scheme and metadata, rather than “turning settings at random.”
- Handover. We deliver the indexing configuration, reindexing policy, a set of reference questions, and a quality dashboard—so the client’s team can check the index after every corpus update.
Configurations, parsing schemes, and update policies
We hand over not a “working search” but a set of artifacts that your team can use to stand up and evolve the knowledge base themselves: parsing configurations, a source map, access rules, and a regular update policy. Below is what specifically remains in the repository and wiki after delivery.
- Working indexing configurations. Settings files for each source type: which document fields go into the index, how long sections are chunked, how much context is attached to the fragment. Configurations are versioned—changing parsing does not require rebuilding the entire base.
- Source and data flow scheme. A map: where a document comes from, which parsing it goes through, where the result ends up, and where the original is stored. With the map, you can see what will break if a network folder is disabled or the export structure changes.
- Format-specific parsing schemes. Separate rules for PDF, DOCX, HTML, and Markdown: how headings and tables are extracted, what to do with scans and attached files, and which documents go to manual review.
- Knowledge base update policy. Reindexing frequency for each source, conditions for full rebuild versus point addition, and behavior when a document is deleted from the source—outdated fragments leave the results along with it.
- Access and segregation rules. Which employees see which sections, how source rights are transferred into the index, and what happens when a user’s role changes.
- Quality check log. A set of control questions used to check results after every update: the share of correct hits, cases of a document being returned without rights, and losses during table parsing. The run is reproducible with a button.
- Documentation and handover. Instructions for reindexing, the procedure for rolling back to a previous database snapshot, analysis of typical failures, and the points to look at in the logs.
With this set, updating the knowledge base stops being a separate project: a change to the policy or configuration takes minutes and does not require our involvement.
Case study: support knowledge base before and after rebuild
What it was like before the rebuild
Support had grown into a base of several tens of thousands of documents: regulations in PDF, instructions in DOCX, portal articles in HTML and Markdown.
The indexer cut them with a fixed character window, regardless of structure: the heading went into one fragment, the conditions table into another, and attachments did not get into the index at all.
The assistant’s answers relied on outdated revisions and pieces of neighboring sections: the share of answers based on the current section hovered around 60%, retrieval of the needed fragment took up to 10 seconds, and the operator closed every fifth request manually—opening the document and searching by eye.
What was changed
Parsing was rewritten around document structure: markup of headings and tables with preserved hierarchy, version and expiration metadata on each fragment, deduplication of identical sections, and separate processing of attachments.
The vector registry was moved to OpenSearch—hybrid semantic search together with exact terms and a filter by current revision directly in the query. Rebuilds became incremental: only the changed document is updated, and outdated fragments do not accumulate.
After a month of measurements following the rebuild:
- share of answers based on the current section—from 60 to 94%;
- retrieval time for the needed fragment—from 10 to 1.5 seconds;
- requests with manual handling—from 20 to 4%;
- documents with correctly parsed tables—from 55 to 97%.
The policy is updated—a new fragment appears in results within a minute, the previous one leaves the filter, so rollback to an old revision remains possible via metadata.
Operators no longer check the version manually: the assistant answers with a link to the current section or reports that there is no confirmation in the base.
How does structure-based chunking differ from semantic chunking?
Both approaches solve one task—turning a stream of text into fragments for the index. The difference is where the cut boundary runs: by document markup or by topic change. An error at this step cannot be fixed by the retrieval model or by reranking.
Structure-based chunking relies on existing markup: headings, sections, paragraphs, lists, individual table cells. For regulations, instructions, and contracts, this is a winning strategy—the fragment remains self-contained and does not mix two topics in one vector. The weak spot is a long section without subheadings: it goes into the index as a whole.
Semantic chunking calculates the similarity of neighboring sentences and cuts where the topic changes—this works on texts without markup: transcripts, articles, correspondence.
The downside is that the boundary may run through the middle of a table or step-by-step instruction, and half a step goes into the index.
Fragment overlap smooths the seams: the tail of the previous chunk is duplicated at the start of the next, usually by 10–20%. It saves answers that lie on the boundary but bloats the index and adds nearly identical results—they have to be filtered out by similarity.
In practice, chunking is assembled in a hybrid way: structural boundaries as the basis, semantic splitting inside large sections, with the parent heading carried into each fragment.
The impact of chunking on accuracy is more noticeable than changing the retrieval model, so it is tested on a set of real questions before launch.
Common technical doubts before starting a project
Before starting, we do not give a presentation but analyze the document corpus: what formats there are, where rights come from, and who owns each source.
The five questions below recur in almost every RAG document indexing project—the answers determine the indexing scheme, the scope of work, and what actually ends up in search results.
We record the answers in writing in the technical solution description. This is cheaper than discovering at handover that half the archive is scans and some employees should see only their own department.
What to do with scans without a text layer?
We run text recognition and evaluate quality on a control sample. If a document contains tables or stamps, it goes into a separate queue and is marked as requiring review—such fragments should not silently enter answers as reliable.
Can the base be updated without a full rebuild?
Yes. Indexing is versioned: only changed documents are rebuilt, and the rest is taken from the previous index. The previous version is retained, so rolling back to a prior search state takes minutes, not a day.
How do access rights get into search results?
Each fragment is assigned access labels from the corporate system, and the filter is applied at the query level. A user receives only documents they are permitted to see, even if search runs across the shared index. When a role changes, rights are recalculated without reindexing the corpus.
What to do with contradictory documents?
We do not average out contradictions. We preserve the source, date, and version status, and on conflict the system shows both fragments indicating where each came from. Source priority is set by the document owner—this rule is configurable, not hardcoded.
How do you make sure search finds the right thing?
We assemble a set of control questions with reference fragments and measure the hit rate before and after every schema change. Without such a set, any indexing change is a change made at random.
Let’s discuss the task: what is needed to estimate scope
To come back with an estimate of corpus scope and an implementation plan, access to the entire infrastructure is not needed—it is enough to describe the task in terms of documents, questions, and rights.
The more specific the input data, the fewer assumptions in the estimate and the sooner a pilot on one section can start: one topic, real employee questions, and answer quality checks before scaling to the entire array.
- Corpus composition: how many files and sections there are, where they are stored—file storage, internal portal, email archive, knowledge base.
- Formats and their quality: PDFs with scans without a text layer, DOCX with tracked changes, HTML, Markdown in a repository. Scans change the scope of work—they need a separate recognition step.
- Dynamics: how many documents are added per month, and whether there are outdated revisions that must not appear in answers.
- Common user questions: 20–30 real phrasings. From them we assemble a validation set and see before implementation which data is missing from the corpus.
- Current access rights scheme: which groups see what. Fragment-level segregation and how restricted documents are indexed depend on it.
- Answer delivery points: internal portal, messenger, operator workstation—the format of results and source links depends on this.
- Constraints: where external services are prohibited, whether request logs and query history storage are required.
Describe the task and send what you already have—we will come back with a scope estimate, an implementation plan, and a proposal for a pilot section.
