Designing RAG Pipeline Architecture

Why a RAG pipeline answers complex questions inaccurately

The assistant answers confidently and coherently — but not a single fact from your document appears in the answer.

An answer without source grounding looks plausible because the model fills it in based on language patterns, and the retrieved fragment merely serves as a prompt for the topic: the link leads to a page that says otherwise.

In regulations, contracts, and instructions, this is most costly: the employee either acts on a convenient but incorrect answer or double-checks manually — and the assistant does not save time, but adds a verification step.

The second failure — loss of meaning during text chunking. When a document is cut to a fixed length, conditions, caveats, and table rows end up in different fragments, and knowledge base search returns a chunk without the answer itself.

The model receives a snippet and fills in the missing part itself: inaccurate assistant answers here are not a generation defect, but a consequence of how the corpus is chunked.

The third failure — an outdated index. A document was updated, the old revision was deleted, but knowledge base search continues to find its copy: vector recalculation either was not started or did not cover all sources.

The assistant relays a revoked rule with the same confidence as a current one — and the discrepancy is discovered by a human reading the primary source.

All three RAG pipeline errors lie not in the model, but in data preparation, retrieval, and grounding verification — we break down how this is closed.

What a managed retrieval and generation architecture provides

Managed architecture turns answer quality into measurable values. Each answer either relies on a supporting source fragment or is marked as uncertain — the share of answers with a citation is visible in the report and grows through iterations.

Faithfulness-to-source and relevance metrics, as in RAGAS, are measured on a set of control questions before rollout: regression is caught in a test run, not in user complaints.

Assistant response speed depends on query routing, not on a "faster model." Short factual questions go to search over a warmed index with a cache of frequent phrasings; multi-step reasoning is enabled only where the answer would be incomplete without it.

Since retrieval returns a limited set of relevant fragments, time to first token no longer depends on the volume of the document corpus.

Reduced support load comes as a consequence: typical requests are closed with an answer citing an instruction, leaving disputed cases to the operator.

At the same time, an entire class of failures disappears — a confidently fabricated answer, which is more costly than an honest "this is not in the knowledge base," because it undermines trust in the entire system at once.

Resilience to load growth comes from separating layers: indexing, retrieval, and generation scale independently, the bottleneck is visible in metrics and removed by adding nodes.

With a tenfold increase in users, the system degrades predictably — queue wait time grows, not the error rate.

Comparison of options: vector, hybrid, and graph retrieval

The retrieval scheme determines which fragments enter the generation context — this is the step where most RAG pipeline prototypes break.

Vector search over embeddings reliably captures semantic similarity but loses part numbers, codes, and rare terms; lexical search is the opposite. Therefore, in practice, three assemblies are compared.

What changes at the output stage

Reranking results over any of the schemes provides the main gain: the first stage takes a broad sample, the second refines the order.

Fine-tuning the model on "query — correct fragment" pairs from your domain improves accuracy where general models confuse adjacent topics. Graph retrieval does not replace the first two schemes, but adds a layer of relationships to them.

Work stages: from data audit to handover to support

A RAG pipeline starts not with code but with data: until it is clear what exactly is in the sources, any retrieval scheme is built on guesses. Therefore, the stages proceed sequentially, and each ends with an artifact that can be checked before moving to the next.

1. Data audit. We analyze sources — regulations, request base, contracts, knowledge base — and count volume, duplicates, outdated revisions, restricted sections. The output is a map: what we index, what we exclude, and by what attribute we differentiate access between departments.
2. Retrieval scheme design. We determine the size of the semantic fragment, retrieval strategy, set of filters, and rules by which an answer must cite a source rather than paraphrase it in its own words.
3. Index setup. We build the indexing pipeline: format parsing, metadata tagging (department, date, document version), regular scheduled updates instead of manual reloads.
4. Answer quality validation. We prepare a control set of questions, including those whose answers are not in the base. We look at the share of answers with a correct citation, the share of honest "not in the data," and errors on outdated document versions.
5. Pilot and tuning. We launch with one team: real query phrasings differ from test ones; after them we adjust chunking and the cutoff threshold.
6. Handover to support. We provide documentation, the reindexing regulation when sources are updated, a quality metrics dashboard, and the procedure for handling "the answer is incorrect" reports.

Handover is not a final meeting but a transfer of responsibility: the team retains instructions on how to connect a new source or roll back an unsuccessful index update, and a set of metrics by which degradation is visible before user complaints.

Technical documentation, diagrams, and update regulations

The project is closed not with a demo of a working prototype, but with a set of artifacts by which the system can be developed and maintained without us.

We record decisions at the time of development while the context is still fresh — otherwise, six months later, reconstructing the logic costs more than the enhancement itself.

• Architecture diagram — the composition of RAG pipeline components, the data path from source to answer, and failure points: where document parsing, vector construction, retrieval, and model invocation occur.
• Source map — a list of provider systems, data owners, change frequency, formats, volume, and access restrictions for each collection.
• Access rules — rights differentiation at the document and section level so that an employee receives an answer only from fragments they are allowed to access.
• Index update regulation — triggers and frequency of reindexing, incremental update mode, and procedures for when the document structure changes or a source is decommissioned.
• Quality monitoring — a set of control questions, share of answers with a verified source citation, tracking whether the right fragment enters the context, and alerts when metrics decline.
• Solution documentation — deployment procedure, configuration composition, where to view logs, how to roll back an index version, and how to connect a new source.
• Change history — versions of retrieval settings and model instructions with notes on which change led to which quality shift.

The set is suitable for handover to a new team or another contractor: from it, they reproduce the build, explain any architectural decision, and continue development without reverse engineering.

Case: how answers without source grounding were eliminated

An assistant over an internal knowledge base answered confidently, but in a third of cases there was no document citation in the answer: the model filled in the wording from memory.

There was no way to verify such an answer, and employees returned to the usual channel — writing to live support, where a typical question was answered within a business day.

What was changed

We analyzed the pipeline step by step. Documents from Confluence were chunked by semantic blocks while preserving headings and hierarchy, and retrieval was made hybrid — vector along with keyword search, so that exact terms and part numbers were not lost in paraphrases.

The key change was the relevance threshold: if the best fragment does not pass it, the assistant replies that there is no confirmation in the base instead of a plausible guess.

Metrics before and after

Share of answers without source grounding — 34% → 4%, average response time — 9 → 2.5 seconds, live support requests for typical questions — minus 41%.

Every answer comes with a citation to a specific document, and disputed wordings go to knowledge base improvement rather than support correspondence.

We separately set up a weekly review: we export answers without citations and look at what was missing — the document itself, precise wording, or markup. This revealed gaps in the base that no one had seen before.

A month later, the share of answers without a source remained at the achieved level rather than growing back.

How a RAG pipeline differs from model fine-tuning

Both approaches solve the same task — making the model answer based on your data — but operate at different levels. A RAG pipeline does not touch the weights: it finds relevant fragments in the knowledge base and inserts them into the query context. Fine-tuning changes the model itself — it learns the answer format, terminology, and domain logic.

Retrieval is needed where the answer relies on facts and they have to be confirmed.

Regulations, catalog, request history, technical documentation — all of this lives in the index, and the index is rebuilt when the source is updated, so data freshness almost never limits the system. The answer comes with a citation to a fragment, and it can be checked manually.

Fine-tuning is chosen when what matters is not the fact but the manner: a unified format for conclusions, industry-specific wording, stable analysis templates. It is also appropriate if the data corpus is stable and changes once a quarter or less often.

In this case, updating is a separate procedure with markup and quality checks, not index rebuilding.

The choice comes down to two questions: how often the data changes and what exactly the model should reproduce — knowledge or form. Frequent changes and the need for citability point to knowledge base retrieval; a stable corpus and non-standard output format point to fine-tuning. The working option is a combination: retrieval provides facts, fine-tuning sets the style.

A typical mistake is to fine-tune a model for facts that change every month: the rebuild cycle cannot keep up with them, and the answers still diverge from the current state of the base.

We decompose the task into these two layers before work begins and fix which one retrieval covers and which one fine-tuning covers.

Questions customer engineers ask

Implementing a RAG pipeline hinges not on model choice but on corpus quality and data access boundaries. Below are questions customer engineers ask before the start, and our substantive answers: how we handle disparate sources, markup, reliability verification, and updates.

How do you split documents into parts so that answers do not lose meaning?

Chunking follows the document structure, not a fixed length: boundaries of sections, regulation clauses, table rows. We preserve the heading hierarchy in metadata so that the citation shows the path to the clause.

We process tables and diagrams in a separate branch, otherwise numbers lose their connection to the row. We keep overlap between neighboring fragments so that a definition is not split in half.

How do you ensure the system does not invent answers?

The answer is built only on retrieved fragments, each with a citation to the source document. Below the relevance threshold, the system honestly replies that there is no data.

Before commissioning, we assemble a set of control questions with reference answers and run it on every build. We look at the share of answers with a correct citation and coverage completeness; we review disputed cases manually.

Documents change every day — how is the knowledge base updated?

Reindexing is incremental: when a change event occurs, only the affected document is rebuilt; the rest is untouched. We store fragment versions and mark outdated ones inactive so that rollback to a previous revision is possible.

When the representation model changes, we rebuild the entire index on schedule. Once per agreed period, we reconcile the index with sources and resolve discrepancies.

Where is the environment deployed and how are rights differentiated?

We deploy in the customer's infrastructure, without external services or transferring data to third parties. Rights are inherited from the source: retrieval filters fragments by the employee's access level, so restricted documents do not enter answers. All requests are logged with account binding — there is something to show the security service.

Discussing the task: which inputs will speed up the estimate

RAG pipeline estimation hinges not on the scope of work but on the quality of inputs.

If we have a document corpus and a dozen real query phrasings, the indexing scheme, fragment splitting, and retrieval method can be estimated in one pass — without hypotheses about "how it usually is." Below is what is worth collecting for the conversation: this is usually enough to come back with an estimate, work plan, and readiness criteria.

• Knowledge corpus: which documents answer the questions (regulations, instructions, request base), in what formats they are, how often they are updated, and who is responsible for currency.
• Real queries: 20–30 phrasings from employees or clients, including imprecise and multi-topic ones — we will use them to test retrieval quality.
• Sources and access: where the data is stored, whether there is an export or test environment, what restrictions on storage and processing apply.
• Answer requirements: whether a source citation is mandatory, what is considered an unacceptable error, which answer is preferable — "I don't know" or an approximate one.
• Readiness criteria: on which questions and with what share of correct answers the solution is considered accepted and who on the customer side confirms this.
• Infrastructure: where the service will run, whether external services are allowed, whether operation in a closed environment is needed.
• Usage scenario: who asks the questions, through which interface, and how many requests per day are expected.

Such a set covers the main architecture forks immediately — and we propose a staged quality validation plan, not general "development" timelines.