RAG with FAISS: local vector search without cloud

RAG development hits cloud search limits

In a pilot, RAG development looks flawless: tens of thousands of chunks, instant answers, the index is rebuilt in minutes. Completely different behavior begins on a real corpus, when millions of chunks from regulations, requests, and correspondence enter the knowledge base.

Here the cloud index ceases to be a neutral detail — limits on requests per second, index size within the plan, and reindexing frequency come into play.

Response latency becomes unpredictable. Some requests queue up to the external service, some are retried after a quota rejection, and the upper bound of response time stretches severalfold, although the average on graphs looks decent.

For a conversational scenario, where a person waits for an answer at the screen, it is the upper bound that matters, not the average.

The second line of risk is data outside the perimeter. For search to work, texts of internal documents, excerpts from record systems, and employee correspondence travel outward.

Formally, this is a third-party environment that must be coordinated with the security service, and on some projects it is precisely this coordination that halts implementation.

Local vector search removes all three limitations at once: the index sits next to the application, quotas on requests to it do not apply, and response time is determined by the resources of your own node, not by the pricing tier of an external service.

The FAISS library is suitable for this without reservations — the index lives on disk and in memory, with no network calls per request. Let's break down how this is covered in practice.

What a local vector index provides: speed and control

A local vector index lives in the same environment as the service: document chunks and their vector representations do not leave your infrastructure.

Therefore, search speed stops depending on the external channel and its quotas — response is determined only by the resources of your node.

Data privacy here is not a promise but a consequence of the architecture: neither the source text nor intermediate representations go outward.

FAISS addresses this task with a mature library: the index is built in batches, held in the process memory, and queried without network hops.

Hence predictable response time — fractions of a second per request even with a noticeable corpus volume, and it does not "drift" as the number of concurrent users grows.

Server load shifts to where it is expected. Vector search costs an order of magnitude less than answer generation, so the bottleneck is the model, not retrieval.

If the collection grows, the index is split into segments and queried in parallel: throughput scales horizontally, without calling someone else's service for every operation.

The index is a versioned artifact, and this changes how changes are handled. Updating the corpus is done by rebuilding in a separate branch, then switching with a single action; the previous version remains on disk, and rollback takes seconds.

Reproducibility of results also becomes a property of the system: one index version and one set of search parameters produce the same result — a complaint about answer quality can be investigated rather than discussed.

An entire class of failures is also removed: unavailability of a third-party service, changes to its terms, exhaustion of a limit at peak. Your search will not go down together with someone else's environment — it will go down only together with yours.

RAG development options: cloud, local environment, hybrid

The deployment scheme determines where embeddings and the vector index are stored, who is responsible for updating them, and what happens if a node fails.

The difference between the options is not in answer quality but in privacy requirements, reindexing frequency, and acceptable response latency.

In practice, the choice comes down to three questions: where data may be located according to requirements, who is responsible for updating the index after document edits, and how much time is acceptable between publication of a document and its appearance in search results.

The answers eliminate unnecessary options before development begins — this way the scheme does not have to be changed midway.

How implementation proceeds: from data audit to launch

We start not with the model but with the document corpus: without source labeling, any search answers at random. The first stage is a data audit, and it shows in advance where the scope of work will be large and where quick configuration will suffice.

1. Data audit. We look at what actually lies in the corpus: formats, duplicates, outdated versions, documents with access restrictions. The output is a list of sources with a suitability assessment and the gaps that will have to be closed.

2. Indexing scheme design. We determine which chunks to split documents into, with what overlap, and which metadata to place alongside: section, date, access rights. A mistake at this step is the most expensive — it becomes visible only during validation.

3. Search configuration. We build a local vector index (FAISS) and add metadata filters so that an employee receives only documents they are allowed to access. Reindexing follows a policy, not manual work.

4. Answer assembly. The model receives only the retrieved chunks and must cite the source. If no suitable chunk exists, it answers "not in the knowledge base" rather than making up facts.

5. Answer quality validation. We prepare a set of reference questions with known answers and calculate the share of correct ones, the share with a correct citation, and the share of honest refusals. We set the threshold in advance: until it is passed, we do not move to production operation.

6. Solution handover. We give the customer's team the index scheme, settings, corpus update policy, and error investigation instructions — after that, the system is maintained without us.

Each stage ends with an artifact that can be checked, not a verbal status. Describe the task — we will come back with an estimate and a stage-by-stage plan.

What remains in the customer's environment: configurations and policies

We hand over not a set of files but a working environment: every artifact can be opened, applied, and checked without our involvement.

Below is what exactly remains with your team after delivery, so that you can update the knowledge base, change rights, and investigate failures on your own.

• Working index configurations — parameters for splitting documents into chunks, the vector embedding model, the threshold for filtering out irrelevant results, and reranking settings. They are stored in a version control system, and changes go through review.
• Deployment scheme — the set of services, interaction points, volumes for the index and source documents, and startup order on your servers or in a closed environment with no external network access.
• Data access policies — which document collections enter the index, role-based separation, the request log, and rules for handling personal and sensitive chunks.
• Index update policy — frequency, incremental loading versus full rebuild, the window during which search remains available, and the procedure for rolling back to the previous snapshot.
• Degradation instructions — how to switch search to a mode without reranking or to keyword-based lookup if load increases or the node with the index goes down.
• Solution documentation — architecture diagram, list of failure points, known quality limitations on short and ambiguous queries, and criteria by which an answer is considered degraded.
• Escalation procedure — communication channel, composition of the on-call shift on your side, response times, and conditions under which our engineers get involved.

The list is closed out by acceptance: you launch deployment according to the instructions on your test bench, update the index according to the policy, and reproduce one failure. If something cannot be reproduced without us, we consider the artifact incomplete and refine it.

Case study: search across an internal knowledge base in a closed environment

The operations service of a distributed network of service centers searched for answers in a knowledge base via an external cloud service.

Regulations and incident analyses left the perimeter, and results came back without a link to the primary source — the first-line engineer manually rechecked the answer and still forwarded the request to the second line.

With a base of tens of thousands of documents, the needed chunk landed in the top three in only four out of ten cases, and a compound query took eight to ten seconds to answer.

What was changed

Search was moved inside the customer's infrastructure: a local vector index on FAISS sits next to the database itself, vector representations are computed on your own capacity, and not a single chunk goes outward.

Documents were split into semantic blocks while preserving the heading hierarchy, reranking of retrieved results and a mandatory source link in every answer were added.

Full reindexing runs at night, edits are picked up as a delta within fifteen minutes, and access is granted by role through the existing directory.

Metrics before and after

The numbers were confirmed from both sides: we calculated from measurements on a set of five hundred real questions, and the customer's operations service used the request log for a month.

The discrepancy on key metrics did not exceed two percentage points, so we consider the result reproducible rather than one-off.

How vector search differs from full-text search

Full-text search relies on an inverted index: a document is split into words, normalized, and the query looks for matching lexemes. It is predictable and explainable — you can see which word matched the chunk.

But if a user rephrases the question with a synonym or makes a typo, the results become empty: there is simply nothing for the words to match.

Vector search works differently — the query and chunks are converted to vector representations, and similarity is calculated by the distance between them, not by letters.

Therefore, "how to process a return" and "conditions for refusing a purchase" end up next to each other, although there are almost no common words. This is semantic retrieval: the answer is found by the meaning of the query, not by its wording.

Comparing the approaches comes down not to "which is better" but to the type of query. Where wording accuracy matters — SKU, contract number, name, identifier — exact word matching is more reliable.

Where the user asks in their own words and rarely guesses the document's wording, semantic search wins.

In a working solution, the two schemes do not compete but complement each other.

Full-text search captures exact hits and filters out the unnecessary by metadata, vector search adds semantic matches, and the combined list is re-sorted so that the truly needed chunk appears at the top. This increases both recall and precision at the same time.

We validate result quality on a set of real questions: the share of queries where the needed answer landed in the top three positions is the main metric of such a system.

Technical questions asked before the start

Before the start, the customer's engineering team checks not answer quality but solution manageability: what will happen to already accumulated data, what load headroom will remain, and who is responsible for search after handover. Below are answers to the points that most often slow down approval.

How often should the index be rebuilt?

As long as the model and text splitting rules do not change, the index is updated without stopping search: only new and modified documents are added. A full rebuild is needed when changing the model or revising the splitting — we document the triggers and policy.

What load headroom will remain?

Response time grows nonlinearly after a certain number of concurrent requests, so we do not estimate headroom by eye. We capture a profile on the peak scenario, split the index by collection, and add read replicas — we keep headroom in multiples, not in single-digit percentages.

What happens after handover?

We hand over the deployed configuration, schemas, rebuild policy, and a control set of queries for acceptance. After that comes the warranty period, the agreed incident response mode, and training for your team: it maintains the index on its own, without us.

Let's discuss the task and come back with a work plan

The first conversation takes about an hour and is needed not for a presentation but for gathering inputs, without which an estimate turns into guesswork.

We review the data corpus, load profile, and environment constraints, and in response we send a work plan, timeline, and team composition. The checklist below is what we clarify with you and what we give back.

• Data. Corpus volume, formats (texts, tables, scans, correspondence), update frequency, and access rights. This determines the logic for splitting documents into chunks and the reindexing schedule.
• Load. How many requests per hour, what response latency is considered acceptable, and how many concurrent sessions. This determines whether we keep the index in RAM or split it into shards.
• Environment. Where the data physically resides, whether external cloud access is allowed, and who owns the keys. If external access is prohibited, we design fully local vector search on the customer's servers.
• Acceptance criteria. A set of reference questions with expected answers and sources. We fix the share of correct hits and the share of "I don't know" answers instead of fabrication — acceptance is built on this.
• Roles. Who on the customer side grants access to storage, who confirms labeling quality, and who accepts the stages. Without a dedicated data owner, timelines slip during integration.
• What we send in response. A stage-by-stage work plan with dependencies, a timeline estimate as a range, team composition with areas of responsibility, an environment diagram, and a list of risks with a plan for each.

The next step is a short call for clarifications, after which you receive an estimate and plan in writing. Describe the task: what the data is, what volume, and in which environment it must remain — and we will come back with a concrete proposal by stages.