Document Chunking for RAG (Recursive, Semantic, Sentence-Level)

Why chunking strategies determine answer quality

Fragment boundaries during chunking are often set not by meaning but by a character counter — a sentence breaks off in the middle, a condition from the first paragraph goes into one fragment, the conclusion into another.

The model receives half of an idea and completes the rest on its own: the answer sounds confident but diverges from the document. At the same time, search reports that the needed page has been found.

Heterogeneous document layout makes it worse: a section heading goes into one fragment, its table into the next, a diagram caption is torn away from the diagram.

This is context loss at the junction of blocks — it is unclear which row a number relates to and which period is meant. In regulations and contracts, where meaning rests on the link "clause — subclause — note", such a break is the most costly.

Overlap of adjacent windows solves one problem and creates another — duplicate fragments in the output. One paragraph ends up in several chunks at once; they take up space in the context, while facts from other sections do not reach the model. The answer either repeats the same thing or relies on a single source.

And separately — an answer without source confirmation: if a fragment does not retain a link to the document, section, and page, the statement cannot be verified, and such an answer is useless in work.

That is why the chunking strategy is selected to match the document structure rather than taken by default: we break down how this is addressed.

The effect of managed chunking: what changes

Managed chunking manifests not in an implementation report but in daily work with the knowledge base. Below are checkpoints by which we compare the result before and after: we measure each on your document corpus rather than estimate by eye.

• Share of answers with the correct source. A cut in the middle of a regulation clause or table produces an answer based on a fragment — it is impossible to cite a specific section. Boundaries by semantic blocks restore an exact reference, and handling a request takes minutes.
• Eliminated class of failures "not enough context". If a question requires two adjacent paragraphs but only one made it into the index, the system stays silent or makes things up. Fragment overlap and structure-based merging remove this class entirely.
• Knowledge base assembly time. Chunking by known blocks instead of fully recalculating vector representations reduces initial assembly from hours to tens of minutes and makes it predictable as the corpus grows.
• Document update without full reindexing. Editing one order recalculates only its fragments; we do not touch the rest of the index. The new revision appears in answers the same day.
• Search quality across regulations. The metric is the share of queries where the needed clause ended up at the top of the results. On annotated regulations it grows noticeably because the fragment contains both the condition and the consequence of the rule.

We test the strategy for each document type on annotations: the recursive splitter from LangChain maintains the separator hierarchy, while semantic boundaries are enabled where the structure is not annotated.

Chunking strategies: comparison of approaches

The chunking strategy determines what exactly enters the model's context during retrieval.

An error here is not visible in retrieval metrics: the needed document is found, but the fragment starts in the middle of a table or loses the section heading — and the model answers confidently but incorrectly.

That is why the strategy is selected to match the corpus structure rather than taken "by default".

Below are four working options and their real limitations. In practice, recursive chunking most often serves as the base, while semantic boundary checking is enabled where document markup does not help.

We select fragment size and overlap by measurement, not by eye: we assemble a set of questions with known correct fragments and look at the hit rate and answer completeness. The guideline is a fragment that fully covers one question; 10–20% overlap removes breaks at boundaries.

Too-small fragments give precise retrieval and a fragmentary answer; too-large ones give noise in the context and extra processing costs.

Stages of work: from document audit to handover

The work proceeds in a fixed order: output quality is determined not by the model but by how the corpus is chunked. Each stage ends with an artifact that we verify on a control sample — therefore the result does not depend on impressions from a demo.

1. Document corpus audit. We collect everything that search should find: regulations, instructions, contracts, support correspondence. We note formats, duplicates, outdated revisions, and the owner of each source.
2. Chunking scheme. We decide what counts as a semantic unit — a regulation clause, an instruction section, or a paragraph — and choose a chunking strategy: recursive by headings, semantic by topic shift, or sentence-based. We set overlap so that the answer does not break at fragment boundaries.
3. Control set of queries. We take several dozen real questions with known correct answers and a source reference. This is the benchmark for comparing chunking options.
4. Configuration and run. We index the corpus and calculate the measurable: the share of questions where the needed fragment reached the top of the results, answer completeness, and the share of "I don't know" refusals.
5. Output quality check. We analyze errors: broken tables, lost context, merging of document revisions, an answer from an outdated version. Changes to the scheme — reindexing and a repeat run.
6. Handover and support training. We provide documentation on the chunking scheme and the update procedure: how to add documents and when to reindex. We train the team that responds to users.

Artifacts: chunking configuration, schemes, and procedures

After launch, the client retains not a "turnkey service" but a kit by which chunking can be changed in-house. Below is what we hand over with the solution and what makes it possible not to call a contractor for every new document format.

• Chunking configuration — an algorithm for each class of documents: recursive division by separators, semantic by topic shift, or chunking by sentence boundaries.
• Fragment overlap rules — how much context is carried into the adjacent fragment so that the answer does not break off at the junction of sections, and how this rule differs for tables and regulations.
• Document processing schemes — the route from file upload to writing to the Qdrant index: parsing, cleaning, metadata tagging, branching by source types.
• Document access rules — which sections go into the shared index, which into restricted collections, and how permissions are inherited from the source system.
• Knowledge base update procedure — what and when is reindexed when a document is edited, who starts the run, and by what criteria the result is accepted.
• Support guide — typical failures: broken parsing, empty fragments, duplicates; what is checked at each step and how to recover.
• Repeat run checklists — comparison of the number of fragments before and after, control questions for search, and the condition for rolling back to the previous chunking version.

Artifacts are versioned together with code and go through review rather than living in chat. A repeat run on the same documents builds the same index — a discrepancy is visible in the comparison immediately, not in the assistant's answers a month later.

Case study: chunking a knowledge base with tables and attachments

The task was brought by the support service: the internal knowledge base assistant answered fluently but cited the wrong document.

What was in the original base

The base had been assembled over years in Confluence: regulations, instructions, and commercial proposals — text with parameter tables and nested lists several levels deep.

Splitting was done by a fixed number of characters, and tables regularly fell apart between adjacent fragments. The assistant returned a "Total" row without a header or a third-level item without its parent heading: formally similar, but semantically off. On a control set of 400 questions, source hit accuracy stayed around 64%.

What we changed and what happened

The foundation became recursive chunking: first by structural headings, then by paragraphs, and only as a last resort by sentences. Tables were moved into separate fragments with a duplicated header, and nested lists were merged with the parent item to preserve hierarchy.

For dense texts without headings, we added semantic division by topic shift — adjacent paragraphs about different subjects no longer end up in one block.

Source hit accuracy grew from 64% to 93% on the same set of questions. The share of answers where the model cited the right document but the wrong table fell severalfold, and cases of "fragment cut off mid-row" disappeared.

Attachments that previously fell out of the index are now processed with source indication. Full corpus reindexing takes about 40 minutes, so updates go out the same business day.

What to do if the base is already chunked the old way?

Rebuilding an existing base is needed less often than it seems. Part of the corpus can be left as is — there the old chunking does not interfere: documents are short, the structure is flat, and answers come in whole.

It is worth rebuilding those groups where the cut breaks meaning: tables, regulations with nested clauses, appendices with lists. It is convenient to keep the fork as three options with different entry costs.

Zero-downtime migration is arranged as follows: the new index is built alongside the old one; during the build, documents enter both streams, and switching is done with a single toggle.

Until the new set confirms quality on control questions, keeping the current index remains the main mode — rollback takes minutes and does not require reloading files.

Hybrid processing mode is a compromise for large corpora: critical groups follow the new logic, while the rest is served by the old index. The difficulty here is not in chunking but in the consistency of the two schemes and in understanding which answer came from which branch.

We start with a sample of queries where answers were assembled from broken fragments — it shows which document groups to migrate first.

Common technical concerns about document chunking

Chunking rarely fails at the start — it fails at output: the answer is assembled from fragments, a table arrives without a header, and the source link leads to the middle of a paragraph.

Chunking strategy parameters live in configuration next to the document processor, so during acceptance we check not "whether it is chunked beautifully" but how search behaves on your corpus. Below are the questions engineers ask most often.

What happens to tables?

A table goes as a separate fragment type: the header is duplicated into each chunk, rows are not cut at the window boundary, and the section heading and table caption are preserved nearby. Without this, search finds the row but does not understand which column the value belongs to.

How to check chunking quality?

We take a control set of questions with reference sources and calculate the hit rate into the needed fragment and the share of blocks broken mid-content. Separately, we look at the length distribution: a spike at the boundaries shows that the rule is cutting not by meaning.

Will the base have to be rebuilt when the scheme changes?

Only for the affected document types — the scheme version is stored next to the index; we do not touch the rest. Chunking is reproducible: we transfer the same scheme to a new document type, changing only the splitting rules.

Let's discuss the task and come back with a work plan

For the conversation, it is useful to prepare the document corpus that search will cover: regulations, contracts, technical documentation, and the support knowledge base — in the form in which they currently exist.

Not only the texts are important but also how metadata is linked to them: sections, versions, access rights. If there are typical user questions and examples of successful answers, they immediately set the quality bar and help choose chunking strategies for your content.

Then the work proceeds step by step, and at each we come back with an intermediate result, not a report for the sake of a report:

• we discuss the task: what decisions should be made based on the documents and what counts as an incorrect answer;
• we collect source data: formats, volumes, languages, the share of scans and tables in the corpus;
• we agree on document access — a test environment or an export of an anonymized set;
• we annotate reference questions and answers so that search quality is measured rather than debated by taste;
• we estimate the scope of work for each block: data preparation, fragment splitting, indexing, verification;
• we send an estimate and work plan with stages, timelines, and acceptance points;
• we document what remains with you after launch — configurations, instructions, the knowledge base update procedure.

If some data is not yet available, we start with what exists: the first measurement on a small sample will show where the bottlenecks are and how long it will take to prepare the rest.

Describe the task in free form — we will come back with a scope estimate and a work plan that shows what each party is responsible for.