RAG System Development (Retrieval-Augmented Generation)

RAG System Development: Where Answer Retrieval Breaks

An enterprise assistant for internal documentation breaks not at generation, but earlier — at the retrieval stage. The model phrases an answer smoothly and confidently, but if the wrong fragments end up in the context, the smoothness only masks the error.

That is why RAG system development starts with corpus diagnostics and a retrieval strategy, not with model selection.

We examine four failure points that occur in almost every project.

• Fragmented corpus. Policies live in the wiki, contracts in file storage, correspondence in email. As long as there is no unified metadata schema and no owner for each set, knowledge base search cannot filter the needed section and returns a mix from different domains.
• Exact-word search. Lexical matching such as BM25 in Elasticsearch does not see synonyms and paraphrases: “vacation” and “annual paid leave” are different words to it, and half of the relevant documents are left out.
• Answers without a source. If answers with a source link are not built into the output by default, the statement cannot be verified. The employee either goes to read the document themselves or stops using the assistant.
• Outdated copies. Several versions of the same document in the index produce contradictory answers, and an edit in the source does not reach search: the assistant cites a revoked version.

Each of these points is addressed at the level of data preparation and index schema, not the wording of the model query. Below is how we build this layer.

What Changes After Launch: Speed and Failure Class

Before launch, finding an answer in policies, correspondence, and the internal base took an employee ten to twenty minutes, and the found fragment often turned out to be outdated: a new version was nearby, but the person opened the wrong one.

After launch, the answer is assembled from indexed documents in seconds — and arrives together with a link to the specific section of the source.

The key metric here is the share of answers with confirmation: the system returns not abstract text but a fragment indicating the document and version.

It is visible directly in the interface: the employee either accepts the answer or marks it as inaccurate, and the accumulated labeling shows answer quality in fact, not based on the impression from a demo.

Reducing hallucinations is a consequence of the output design, not of persuading the model: if the retrieved fragments do not contain the answer, the system must say “not found” rather than construct it by meaning.

The class of failures due to outdated data is addressed separately — when a document is updated, the previous version is displaced from the index, and the old instruction stops appearing in answers.

The load on support drops because employees resolve some typical questions themselves, without reaching the first line.

Before and after launch, we capture the same set: time to answer, share of answers with a link, share of “not found” refusals, and the number of escalations to the second line — the difference in these numbers is the measured result of the launch.

Architecture Options: Comparing Retrieval Approaches

RAG architecture rests not on the generation model but on how retrieval over the corpus is arranged.

Vector search catches semantic matches but loses part numbers, abbreviations, and negations; lexical search, conversely, is precise on rare terms but helpless before a restatement of an idea in one’s own words. The mode is chosen for the nature of queries and the cost of error in the answer, not “by default.”

Hybrid search with result reranking usually yields the highest share of relevant fragments: the lexical part pulls exact matches, the vector part pulls semantic ones, and reranking brings to the top what actually answers the question.

The trade-off is latency and weight tuning, so we make the choice based on measurements on your query sample, not on general practice.

Implementation Stages: From Corpus Audit to Handover

The order of stages is not accidental: each subsequent one relies on the artifacts of the previous, so a data discrepancy is cheaper to catch during the audit than during acceptance of ready answers.

Before starting, we fix measurable criteria — the share of questions with a correct answer and a mandatory source link — otherwise the argument about quality turns into a matter of taste.

1. Document corpus audit. We review formats, duplicates, versions, expiration, and access rights. The output is a corpus map with the share of materials suitable for indexing and a list of what will have to be cleaned up or supplemented.
2. Environment schema. We separate retrieval and generation: what goes into the index, what goes to the model, where the user rights filter sits. This way an answer will not be assembled from documents unavailable to a specific employee.
3. Data preparation and search index assembly. Chunking into fragments along semantic boundaries, vector representations, hybrid search with reranking. Small fragments provide precision, large ones provide context; we tune the balance based on measurements.
4. Answer configuration. System instruction, citation rules, behavior when data is absent: the system says “not in the base” rather than completing facts. We separately describe how contradictions between versions are handled.
5. Testing answers on real questions. A set of reference questions from your employees, including tricky and borderline ones. We calculate the share of correct answers with a link, the share of honest refusals, and trace each error back to its cause.
6. Handover. Configurations, corpus update regulations, reindexing scripts, metrics, and support instructions. After handover, the customer’s team can add documents without breaking answer quality.

What We Hand Over: Index, Schemas, Regulations

Solution handover is not an archive with source code but a kit by which your team can operate and develop the system independently. What will physically be in your hands:

• Working environment — deployed system, service configurations, storage access, backup and recovery procedures.
• Integration schema — which sources are connected, in what format data is retrieved, how often synchronization occurs, and what happens when a source is unavailable.
• Index design — rules for chunking documents into fragments, metadata composition, procedure for filtering output by user rights.
• Access roles — who uploads documents, who changes output settings, who sees logs; restrictions work at the index level, not the interface level.
• Index update regulations — schedule for full and partial reindexing, deletion of outdated documents, procedure for rollback to the previous state.
• Monitoring and logs — answer quality and speed metrics, degradation alerts, retention periods for query history.
• System documentation — architecture diagrams, support instructions for typical failures, a set of reference queries for verification after changes.

The kit remains living: when sources or the model change, we update schemas and regulations so that the documentation does not diverge from the actual state of the system.

Case Study: Answers from Policies Instead of Manual Search

The internal knowledge base of the manufacturing holding «Severny Kontur» — about 4,000 documents: policies, orders, labor safety instructions, contract templates.

Employees searched by keywords and file names, so an answer to a typical question took 15–25 minutes, and disputed cases still went to a lawyer or HR. At the same time, outdated versions lay in the base and were found on par with current ones.

What We Changed

We assembled a corpus with versioning: each version was linked to its effective date and status, outdated ones were removed from search but kept in the archive.

The index was built by semantic blocks, not whole files — a question about vacation finds a specific clause, not a sixty-page document.

We added a link to the primary source with the clause number and version date in the output, so the answer is verified in seconds rather than taken on faith.

What Changed

The share of questions resolved without escalation grew from 38% to 79%, and the median time to answer fell from 20 minutes to 40 seconds. The accuracy of hitting the right clause on the control sample was 9 out of 10 requests.

The remaining misses occur in situations where policies contradict each other: such cases are marked by the system as a conflict, and it does not choose a side for the person.

Implementation took six weeks — two for corpus analysis and status labeling, two for assembly and output configuration, two for a pilot in two departments with metric measurement.

How Does RAG Retrieval Differ from Model Fine-Tuning?

Fine-tuning changes the model weights: private data ends up inside it, and afterward it can neither be extracted nor updated separately. RAG leaves knowledge outside — in the index of your documents — and for each query the model receives only relevant fragments. This is where all the practical differences between the approaches diverge.

Updatability is resolved at the index level: a new policy or a new version of a contract appears — reindexing takes minutes, and the model itself is not touched.

After fine-tuning, any fresh fact requires another training cycle, and with it the risk of losing what the model could do before. Where documents live their own life, this is a key limitation.

Verifiability is built into RAG: each answer can be broken down into source fragments, and it is visible where the wording came from.

In a fine-tuned model, knowledge is dissolved in the weights — you cannot cite a specific clause of an instruction, and a confident fabrication looks just as smooth as a correct answer.

For internal bases and instructions, this is often more important than the share of correct answers.

Data volume also separates the approaches. Fine-tuning requires thousands of coordinated “question — reference answer” pairs and labeling, otherwise quality on the domain drops.

RAG launches on a few hundred documents and works with them as they are — including tables and scans after recognition.

There is also a downside: fine-tuning better sets style, format, and terminology — the model starts writing the way it is customary in your organization.

Therefore the choice is not “either-or”: first RAG covers working with knowledge, and fine-tuning is added selectively if your own voice is needed. Next we discuss how this looks in implementation.

Technical Concerns: Quality, Updates, Access

Concerns about a RAG system usually relate not to the model but to operation: where data comes from, how the index catches up with corpus changes, who sees what, how to integrate this into current systems, and how to measure answer quality. Below are answers on the merits.

Employees have different access rights — will the system take this into account?

Rights are inherited from the source, not assigned manually. The filter by access lists works before generation: a restricted document will not appear in the answer, citation, or suggestion. Changes to rights in the source system are pulled in during the next synchronization.

Can data be kept from leaving the perimeter?

The environment is deployed in your infrastructure or in a private cloud; only fragments that have passed the access filter go outside. Logs retain identifiers and metrics; content is masked or not written at all. Under strict requirements, we consider an option with the model on your hardware.

How can we tell that answer quality has not dropped?

There is a set of reference questions with known correct sources. We calculate separately: whether the needed fragment appeared in the output and whether the answer relies on it.

The run is repeated after every corpus or settings update — degradation is visible in numbers, not in feelings.

Will current systems have to be rebuilt?

Retrieval and generation are exposed through an interface that integrates into the portal, chat, or work system; login and rights are taken from the existing user directory. Each answer is provided with a link to the primary source — it can be verified in one click.

Let’s Discuss the Task and Come Back with a Plan and Estimate

For the project estimate to rely on facts rather than assumptions, a minimum of source data is enough. Gather what you already have at hand — we will clarify the rest in conversation.

• Which documents and bases go into search: formats, approximate volume, who is responsible for keeping them current.
• 10–20 real user questions and reference answers to them — quality is tuned against them.
• Where the system will run: your environment or external infrastructure, requirements for access and request auditing.
• Usage scenarios: who asks, how often, what we consider a correct answer and what — a correct refusal.
• Which systems need to be connected: portal, CRM, messengers, internal services, directories.
• Who on the customer side accepts the result and by what criteria.

Then the cycle is short: we discuss the task, fix the implementation plan and quality metrics, then a pilot launch follows on a limited set of documents and live employee questions.

The pilot shows how the system answers specifically on your data — before the solution is scaled to the entire environment.

We determine the scope of work after the pilot: until then you are not bound by obligations either on deadlines or on the composition of stages. Describe the task in free form — we will come back with a project estimate and implementation plan.