Graph RAG: why ordinary search breaks on long chains of questions
Ordinary document search splits the corpus into chunks, calculates similarity of each to the question wording, and returns the top few into the model context. The scheme works as long as the answer lies within one paragraph.
As soon as a question requires traversing three-four links between entities, similarity ceases to be a measure of relevance: the question states the outcome, while each link in the chain speaks about its own thing.
Hence two classes of failures in answers. First — the chunk with the needed link does not get into the top of the results: it is not similar to the question lexically or semantically because it describes an intermediate event or property.
Second — the question is aggregative (“what do these cases have in common”, “which areas intersect”), and a sample of several paragraphs does not cover the corpus. The model answers partially or completes the link itself — convincingly and off fact.
Graph RAG changes the unit of search: it indexes not chunks, but entities and relationships between them. During preparation a knowledge graph is built — mentions are extracted, duplicates normalized, relationship types assigned.
Multi-step search goes along the structure: entry points are found, then two-four hops along edges, and a connected subgraph along with source citations enters the context. Connectivity of facts becomes verifiable, rather than guessed by textual similarity.
Then we break down what solutions make up such a system and where its boundaries lie.
What a knowledge graph provides: accuracy and robustness of answers
A knowledge graph stores entities and relationships separately from text chunks. Search goes not by wording match, but along edges: the system moves from fact to fact and assembles an answer several steps ahead. In Graph RAG this route becomes part of the output — it is visible and can be checked.
Share of correct answers on composite questions
Vector search answers confidently while the needed fact lies in one chunk. As soon as the answer is assembled from three-four sources, the share of correct answers falls: some connections are lost between chunks.
Graph traversal keeps these connections, and on multi-step questions accuracy grows by a factor, not by percentages.
Verifiable reasoning chain
The system returns not only the answer, but also a path of four-six transitions. The engineer sees at which edge the reasoning turned, and fixes not the model but the graph itself: adds a connection or refines an entity. The error can be caught before the user sees it.
Context volume per request
Ordinary search puts a dozen and a half chunks into the request “just in case” — the model gets diluted by noise. The graph returns three-five nodes connected specifically to the chain, and the context volume shrinks severalfold with higher answer quality.
At the same time the model call itself becomes cheaper: less input data — more stable response time.
Robustness of retrieval
Rephrasing the question does not change the set of nodes: the route goes along connections, not words. The share of queries where the answer “floats” when wording changes decreases — the result stops depending on how the user asked.
Ways to link facts: graph, vectors, or hybrid
Three modes differ in what exactly is considered a connection between facts. Vector search relies on semantic similarity: two chunks end up in the same results even if they have no common words.
Graph connectivity is explicit edges between entities, so the answer is assembled by traversal, not by selecting similar pieces. Hybrid search takes candidates from both sources and merges them with a single ranking.
How modes behave on real data
| Variant | When suitable | Limitations |
|---|---|---|
| Vector search | Queries are formulated freely, answer lies in one chunk | Does not connect facts from different places, misses on paraphrasing |
| Graph connectivity | Answer is derived from the chain “entity — relationship — entity”: regulations, contracts, reference books | Requires extraction and normalization of entities, errors in edges are expensive to fix |
| Hybrid search | Mixed queries where both wording similarity and hard connections are needed | Two indexes are synchronized on every load, merging stage required |
What determines scheme choice
We look at the nature of the data and the type of question. If the answer is assembled from several documents and the order of connections matters, similarity alone is not enough — a graph is needed. If queries live in user wording, a graph without vector misses on paraphrasing.
In practice scheme choice is rarely a strict “either-or”: hybrid covers both classes of queries, but pays for it with index synchronization on every data update.
How we build a knowledge graph: stages of engineering work
A knowledge graph is not assembled in one pass over documents: an error in the schema at the start costs more than a month of work — all connections have to be relabeled.
Therefore implementation stages go from source inventory to quality measurements, and only after that the graph goes into operation.
- Source audit. We bring regulations, tickets, the knowledge base, and tables into one map: volume, duplicates, access rights, change frequency. Here we also define the fields by which entities are joined together.
- Schema design. We fix node and edge types, mandatory attributes, direction of relationships, and duplicate merge rules. The schema is approved before the first load: otherwise entity extraction yields incompatible types.
- Entity and relationship extraction. Rules cover predictable fields, the language model covers wording in text. After entity resolution, one real object gets one node, not three similar ones.
- Graph assembly. We load nodes and edges into storage, add section hierarchy, relationship weights, and backlinks to the source chunk — every citation is checked against them.
- Control set. We take real employee questions with reference answer chains. On them we measure extraction completeness and multi-step query accuracy, separately on the part invisible during tuning.
- Handover to the team. We hand over the schema, update regulation, reindexing when the source changes, and a quality dashboard: share of correct chains and response time.
The acceptance criterion is not the fact of graph assembly, but the stability of metrics on the control set: if the accuracy of multi-step answers drops when new documents are added, we return the schema or extraction for revision.
Graph schema, access policies, and documentation handover
The project ends not with a successful demo query, but with a state in which your team itself connects new sources, edits connections, and does not break access.
Therefore acceptance goes by a list of artifacts: each sits next to the code, is versioned with it, and is tested on staging before handover.
- Graph schema — node and edge types, mandatory attributes, entity duplicate merge rules, and also what is linked automatically and what a human confirms.
- Enrichment rules — which sources are traversed and at what frequency, how conflict between them is resolved, and by what attribute a record is considered more reliable.
- Extraction and linking settings — size of the parsed chunk, text representation model, confidence threshold for an edge, weights of relationship types. With them the search answer is reproducible, not “comes out differently.”
- Access policies — roles, restrictions on reading individual nodes and entire subgraphs, field masking in the answer, query log with subject and access time.
- Update regulation — what is rebuilt in parts, what requires full graph recalculation, who starts the process, and how the version of connections is checked against the version of sources.
- Evaluation question set — multi-step scenarios with expected paths in the graph; running them shows accuracy drop immediately after edits, not through user complaints.
- Operations manual — restoring connections after load failure, rollback to the previous graph version, typical errors, and escalation procedure.
- Solution documentation — architecture diagram, description of contours, and decision log: why the relationship structure is exactly this and what changed along the way.
Such a kit allows handing over maintenance to your team or returning to development months later without reverse-engineering the code.
Case: how answer accuracy grew on connected questions
The project was conducted for an industrial equipment supplier: about 40,000 documents — regulations, manuals, test protocols.
Pilot user questions are structured so that the answer is assembled from three-four sources: first find the product node, then go to the modification, then to tolerances and maintenance regulation.
At the start search worked by vector similarity of chunks — on single questions this gave an acceptable result, on connected ones it fell apart.
What pre-measurements showed
On a reference set of 300 connected questions, the share of correct answers was 52%.
In 41% of cases at least one connection was lost, and the answer looked plausible but did not contain the needed item — the most expensive error category, because only a subject-matter expert noticed it.
Retrieval assembly took up to 3.1 s, and manual answer verification took the engineer several minutes.
What was changed
Entities and relationships were moved into a knowledge graph on Neo4j, text chunks were left as node attributes. For a query we build a path along connections in two-three hops and only then rank candidates, so a missing link in the chain stops being invisible to search.
We added mandatory references to nodes and documents from which the answer is assembled to each answer — traceability appeared.
What happened on the same set
The share of correct answers grew to 88%, connection losses dropped to 4%, retrieval preparation time — to 0.9 s. Continuous manual verification was replaced by sampling: the engineer reviews one answer out of ten instead of every one.
On the extended question set metrics hold — the graph is replenished together with documentation, rather than degrading as the base grows.
How Graph RAG differs from ordinary vector search
Ordinary vector search ranks chunks by similarity to the query: the system finds text pieces whose vector representations are closest to the question and returns them to the model.
The graph layer answers a different question — not “what is similar,” but “what is connected to what”: entities and relationships from documents are brought into explicit connections, and search goes along them.
The question “who is responsible for this module and what restrictions does it impose on neighboring systems” will decompose into a chain: module → responsible → adjacent systems → restrictions.
Vector search will pull chunks close to the wording, and often these are different documents without connection to each other. The graph traverses the chain along edges, and the answer shows the inference path — it can be checked.
The graph layer is excessive when questions are local and closed by one document, when data is scarce and similarity gives acceptable accuracy, when relationships between entities in the domain are unstable.
Then maintaining connections does not pay off, and vector search works faster and is simpler to maintain.
The approach also has limitations: graph quality is no higher than the quality of entity extraction, the schema requires maintenance when the domain changes, and new data must be passed through relationship enrichment.
Therefore hybrid is more often taken — vector candidate selection and graph traversal on top of them.
We make the choice based on measurements on real queries: a set of typical questions, comparison of answer completeness, share of cases where the chain of facts cannot be assembled from individual chunks.
We connect Neo4j and similar storages only when measurements confirm that connectivity is truly needed.
Technical doubts: data, quality, maintenance
Doubts about Graph RAG boil down to four things: contradictory sources, graph updates, closed environment, and result acceptance. Below is how each of them is solved in the project, rather than postponed to the maintenance stage.
What if sources contradict each other?
The graph does not choose a “correct” document — it stores provenance: every fact retains source, date, and version. Conflicting statements become separate connections, and priority is set by policy: freshness, source status, owning department. The answer shows what confirms it and where the discrepancy lies.
How is the graph updated when documents are edited?
Full rebuild on every edit is not needed: incremental reindexing works — the changed chunk is parsed again, affected nodes and relationships are overwritten, the rest stays in place. Full rebuild follows the regulation, for example after storage migration.
Can the solution be deployed in a closed environment?
Yes. Models, index, and graph are hosted on your infrastructure, external calls are closed by network policy, access is granted by roles. Query logs remain inside the environment: for sensitive data this is a condition, not an option.
By what criteria is quality accepted?
Acceptance runs on a question set from your scenarios: separately simple, separately requiring traversal of several connections. We count the share of correct answers, accuracy of source references, and traversal completeness, and record error classes.
If metrics are not achieved, we improve document parsing and graph schema, rather than fitting answers.
Let's discuss the task and come back with a work plan
To come back with a work plan, rather than general promises, at the start we need a minimum of context from your side. Graph RAG is built around a specific corpus and specific questions — without them the knowledge graph turns out neat but useless in work.
What is needed from you at the first task review:
- 20–30 real user questions, including composite ones where the answer is assembled from several documents.
- Source corpus as is: regulations, contracts, catalogs, technical documentation, correspondence.
- List of entities and relationships important in the domain: client, object, contract, document version.
- Description of how search fails now: where the connection between documents is lost and what is considered an incorrect answer.
- Environment constraints: where we deploy the solution, which systems provide data and under which access rights.
- Desired launch horizon and who on your side accepts the result.
Based on the review, we send a stage plan with approximate timelines: corpus preparation and relationship labeling, graph assembly, connecting graph to search, answer generation setup, acceptance on your question set.
For each stage we indicate what is considered a finished result and which checks it passes — share of correct composite answers, presence of a source reference, system behavior where there is no data in the corpus.
Describe the task in free form, even without a ready question set — we will clarify the rest in dialogue and come back with a timeline estimate and work plan.
