Why Multi-Index RAG: a single index cannot handle heterogeneous data
A single index is built on the assumption that all data speaks the same language. In practice, it contains PDF regulations, support tickets, exports from the accounting system, emails, and scanned contracts.
They have different structures, chunk lengths, terminology, and fact density, while one representation model is forced to average all of this into a common feature space.
The bottleneck occurs at the candidate retrieval stage, before any answer synthesis. Dense vectors capture semantic similarity well but blur exact identifiers — SKU, contract number, error code, last name: after averaging, they are almost indistinguishable.
Lexical keyword search, by contrast, is reliable on exact matches and helpless where the question is phrased as a paraphrase.
Then degradation accumulates: fragments from an irrelevant source make it into the top results, reranking operates on obviously noisy candidates, and the model builds the answer by meaning rather than by document.
Average relevance looks acceptable — the failure is visible not in the metrics but on specific queries where the answer is confidently wrong.
We break down how this is addressed: sources are split across several indexes by data type, the query is routed across them, and results are merged and reranked at a common level.
What index merging provides: answer quality and resilience
A single index is one point of view on the data: it captures either meaning or exact terms well, and source unavailability turns into an empty answer.
Merging several indexes removes both problems: quality grows not by retraining the model, but because more complete and heterogeneous retrieval reaches the generation input.
Answer accuracy and coverage completeness
A dense vector index finds things close in meaning but loses SKUs, abbreviations, and rare terms; a sparse lexical one does the opposite.
We combine them hybridly — for example, via Elasticsearch with BM25 and kNN fields — merge the results with rank fusion, and run them through a reranker.
In measurements, this increases the share of correct answers from 60% to 85% and coverage completeness — the share of questions for which the knowledge base has any grounding — from 70% to 95%.
Response time remains within budget: merging and reranking add about 200 ms against a target response of up to two seconds.
Resilience to individual source failures
The second effect is fault tolerance.
Each source is queried in parallel, with its own timeout and circuit breaker: if an index is unavailable or degraded, the query does not fail entirely — retrieval is assembled from the remaining ones, and the reranker is disabled first to preserve response time.
We log the share of queries served from an incomplete set of sources: normally it is zero, during a failure it grows, but answers keep coming, and degradation is visible in the metrics before the user notices it.
Multi-Index RAG patterns: routing, merging, federation
The source merging pattern determines how many collections are queried per question and how their answers are consolidated into one context. Below are three approaches that are usually chosen between.
Query routing
A classifier or a compact model assigns the question to one or two collections; the rest are not queried. This is the fastest option in terms of response time, but a classifier error is irreversible: the generator will no longer see a document from an unselected collection. As a safeguard, the rule “below the confidence threshold, query all indexes” applies.
Result merging
Each index returns its own ranked list, and it must be consolidated into a single order — by mutual rank voting or weighted scores. The pattern does not require training, but it is sensitive to weights: otherwise a collection with long documents pulls retrieval toward itself.
Federated search queries all collections in parallel and does not depend on a classifier — at the cost of latency and bloated context. Hybrid search adds lexical comparison to vector comparison, so SKUs and abbreviations stop being lost. Both options are built on Elasticsearch — the search layer takes over merging and ranking.
| Option | When it fits | Limitations |
|---|---|---|
| Query routing | Collections are separated by topic, fast response is important | A classifier error cuts off the needed document |
| Result merging | Sources overlap, a single order is needed | Requires weight calibration between indexes |
| Federated search | Collections are incomparable, completeness is important | Latency and context size grow |
| Hybrid search | Queries contain codes, SKUs, narrow terms | Indexing is more expensive, both index types are needed |
How we implement multi-index search: project stages
Any Multi-Index RAG solution starts not with the model, but with knowledge inventory. Below is the sequence of work as it proceeds in a project: from the data map to handing the solution over to your support team.
Source audit and data map
- Inventory: corporate knowledge base, file storage, accounting systems, tickets, regulations. For each source — format, volume, change frequency, owner, and access rights.
- Data map: we record which questions each source covers, where data is duplicated, and where it contradicts itself. The result is a document used to build the indexing scheme.
- Indexing scheme: a separate index for each data type, its own chunking strategy, and a metadata set: date, owner, access level, link to the primary source.
Search setup and validation on benchmark questions
- Search setup: routing queries to relevant indexes, merging and reranking retrieval, citation rules. We consider an answer without a link to a source fragment invalid.
- Validation on benchmark questions: a set of 80–120 questions with known answers, including those for which there is no answer in the knowledge base. We check whether the needed fragment reaches the top positions and whether the system can honestly say that the data is missing.
- Iterations: we adjust chunking, metadata, and weights — until quality is stable across the entire set, not just on a few successful examples.
- Handover to operations: a procedure for updating indexes when sources change, monitoring the share of unanswered questions, and instructions for data owners.
Implementation artifacts: diagrams, configurations, procedures
After implementing Multi-Index RAG, the client is left not with a presentation but with a set of working artifacts by which the system lives and evolves without us. Each document is tied to a specific node: indexes, the query router, the data access layer. The list below states exactly what is handed over and what is described in each artifact.
Data flow diagram
- Query routing: which index is queried first, by what signals the query goes to the others, and where retrieval merging is triggered.
- Source map: list of repositories, their format, change frequency, and the owner on the client side.
- Index configurations: text chunking parameters, similarity metric, weights for result merging — with an explanation of what changes when a new source is added.
- Degradation points: system behavior when one of the indexes is unavailable and the response mode for incomplete retrieval.
Update and access procedures
- Reindexing: scheduled and event-based triggers, the procedure for full and partial rebuilds.
- Access policies: role-based separation at the index and document level, inheritance of rights from source systems, and retrieval logging.
- Operational documentation: the procedure for accepting a new source, rollback to the previous index version, and areas of responsibility.
Case study: how answer accuracy grew after merging sources
The client is a service company where knowledge is distributed between an internal knowledge base, a ticket archive, and regulations in documents.
The assistant answered only from the knowledge base, so for composite questions such as “repair time under a deferred payment contract” it gave a confident but incorrect answer.
Initial situation
The share of correct answers stayed around 60%: reference questions were covered, but anything that required matching a ticket with a clause of the regulation was not.
Engineers manually double-checked the assistant, answer preparation time reached 40 minutes, and every fourth topic came back again.
The cause was not the model but fragmented context: search ran over a single source, and the relevant fragment from the regulation never reached generation.
What changed after implementation
We assembled several independent indexes — the Confluence knowledge base, the ticket archive, and file-based regulations — and separated them by query type.
Routing sends the question only to the needed indexes, and results are merged by positions and source weights rather than raw similarity scores: a fragment that is first in a weak index no longer overrides an exact hit in a strong one.
| Metric | Before | After |
|---|---|---|
| Share of correct answers | ~60% | ~90% |
| Answer preparation time | 40 min | 6 min |
| Repeat requests on the topic | ~25% | ~8% |
| Answers with a source link | not counted | all |
Accuracy grew not through changing the model, but through context completeness: the answer relies on two or three sources, and the link allows the conclusion to be verified in seconds. The debrief with the support manager is in the testimonial below.
How does result merging differ from federated search?
The difference is in the point where source answers meet and in whether they share a common scale. With federated search, the query goes to each index separately, and the source returns the best fragments according to its own rules.
Result merging consolidates candidates into one pool before ranking — then they acquire a common order.
The key difficulty is the incomparability of scores. Lexical search scores and vector representation similarity lie in different ranges, and bringing them to a common interval distorts the tails: a rare but exact fragment receives a low normalized score. That is why sources return order rather than score, and the lists are merged rank-wise.
Then reranking comes in. The pool is assembled from several candidates per index, duplicates are collapsed, and one pass of a heavy model over the full fragment texts produces a single order.
Accuracy at the top positions grows, but response latency grows along with the number of candidates.
A federated pattern is needed where sources cannot be consolidated into one pool: access separation, different regulations, data does not leave its perimeter. Each node ranks on its own, merging is based on metadata and per-source quotas, and reranking is possible only locally. The downside is that a “talkative” source pulls the context toward itself.
The choice of pattern is determined by source homogeneity and text availability. Homogeneous corpora and the right to read fragments in full — merging with reranking.
Heterogeneous sources with access separation — a federated layer, over which merging reduces to rank-wise consolidation within the permitted subset.
Technical concerns: quality, latency, data updates
Five questions most often come up at the evaluation stage: whether response time will grow, how quickly data becomes outdated, how rights are separated, whether the infrastructure will withstand the load, and who supports the system after launch. We test each concern in a pilot — with metrics on real queries, not on demo examples.
Will response latency grow if there are several indexes?
Search runs in parallel across all indexes: response time is determined by the slowest one, not by their sum. We set the time budget and cutoff rule during design — an index that does not finish in time is skipped, and the answer is built from the remaining ones.
How quickly is data updated, and what about obsolescence?
Each index has its own freshness procedure: critical sources are updated on a change event, reference sources on a schedule. Fragments carry the source date and version, so it is clear which data the answer was built on.
How are access rights to sources separated?
Rights are checked before fragments enter the model context — otherwise data will leak through the answer. The filter is applied to each index separately, and all accesses are logged.
Will the infrastructure withstand query growth?
Indexes scale independently: load on a rarely used source does not slow down a frequently used one. Caching and a reindexing queue smooth out peaks, and we calculate capacity based on the request profile.
Who supports the solution after launch?
We hand over the procedure for connecting a new source, reindexing rules, and a metrics dashboard. Support is a separate mode: scheduled updates, incident analysis, and assistance with expansion.
Let's discuss your task and come back with an estimate and plan
A conversation about Multi-Index RAG starts not with model selection but with understanding the task: which questions the system must cover, where the answers live, and who uses them. This determines how many indexes are actually needed and how to route queries between them.
To come back with an estimate of the scope of work and an implementation plan, rather than generalities, we need inputs on data and scenarios:
- Sources: which systems and files contain the answers, how often they are updated, and whether exports exist in machine-readable form.
- Data requirements: structure, language, availability of metadata and markup, and who is responsible for keeping it current.
- Access rights: how to separate retrieval by role so that one index does not show someone else's data.
- Query scenarios: typical user phrasings and what we consider an accurate and complete answer.
- Volume: how many documents and queries per day, and what growth we expect.
- Constraints: where the perimeter is hosted, what needs to be logged and stored.
- Acceptance criteria: by which quality metrics we consider the solution working.
Describe the task in free form — we will come back with an estimate of the scope of work, an implementation plan, and team composition. If some inputs are not available yet, this does not block the start: we will clarify them during the first review.
