Why a knowledge graph does not assemble itself from texts
Disparate registries, contracts, meeting minutes, and work correspondence usually already lie in a single storage — it seems that this is enough for a knowledge graph.
In practice, unstructured documents do not add up to a coherent structure: search responds with one file but does not show how entities are related to each other, and the analyst continues to compile relationships manually.
The root of the problem is the level of processing: automatic entity extraction works on one document at a time and does not reconcile results with each other. One counterparty in three disparate registries is recorded differently, while different objects bear the same names.
Without normalization and entity resolution, a knowledge graph from texts falls apart into disconnected fragments.
Relationships are even harder. The rules of "who references whom" and "what follows from what" need to be described by a type schema; otherwise, the extracted facts have nothing to be matched against.
The relationship type, its direction, and its grounding in a specific text fragment are defined by an ontology — only after that can queries be built against the graph rather than paging through sources.
A separate topic is data connectivity. We measure it by the share of entities that appear in at least one confirmed relationship and the share of "dangling" nodes with no relationship at all.
When connectivity drops, the graph is formally built but provides no analytical value: you cannot traverse a path from supplier to end object.
Next, we examine how this is addressed: how the entity and relationship schema is built, what confirms each relationship, and which metrics show that the graph is actually assembling rather than simply being filled with nodes.
What automatic entity and relationship extraction provides
The extraction pipeline parses a document into sentences, identifies entities and relationships between them, after which entity normalization reduces spelling variants to a single node: "Romashka LLC", "Romashka", and a typo with a Latin letter become one record with a source history.
Without this step, the graph quickly turns into a set of nearly identical nodes, and a query by organization returns an incomplete answer.
Extraction completeness depends not on the model but on the relationship schema: we fix the relationship types in advance (supplier, owner, branch, conflict) and run the corpus through two independent passes.
A relationship enters the graph only with a reference to the source document and paragraph, so a false positive can always be revoked without rebuilding the entire graph.
Annotation quality control is based on regular stratified sampling: random documents of each type plus all disputed cases go to manual review, and discrepancies between annotators are collected into a separate list.
We track precision and recall separately for each relationship type — this shows exactly what degrades when the format of incoming files changes.
Relationship updates work incrementally: a new document adds edges, and outdated relationships are not deleted but marked as inactive with a date, so the graph remains reproducible at any point in time.
We deliver the result in the standard model — RDF — and as relational tables, so the graph can be connected to existing storage without migrating data.
Practical effect: a corpus of tens of thousands of documents is processed overnight instead of weeks of manual annotation, the share of relationships with a confirmed source reaches 0.9 on the control sample, and the error class "different spellings of the same entity" is eliminated at the normalization stage rather than surfacing in reports.
Construction methods: comparison of approaches and limitations
Automatic relationship construction is always a trade-off between completeness and precision: the broader the rules and dictionary, the more extra edges settle in the graph.
To compare options fairly, we run each on the same corpus and compare against held-out annotations — we calculate precision on extracted relationships, not on individual terms.
Rules and synonym dictionaries are convenient to describe in the portable SKOS format: the schema is edited by a subject-matter expert, not a developer, and it is not tied to a single system. Below are the options that are actually chosen between, along with their limits.
| Option | When it fits | Limitations |
|---|---|---|
| Rules and synonym dictionaries | Narrow terminology, ready-made reference books exist | New spelling or typo bypass the rules; the dictionary is maintained manually, conflicting rules produce duplicate edges |
| Statistical and vector methods | There is a lot of text, no annotation, and thematic similarity of entities is needed | They produce a "similar" relationship without a relationship type; the similarity threshold is selected blindly, and frequent terms cause false positives |
| Trained relationship extraction model | Typed relationships are needed: who mentions whom, what is part of what | Requires an annotated domain; changing the domain leads to retraining, and rare relationship types are not learned |
| Hybrid with manual schema tuning | The domain schema is known and must be followed | More expensive to set up: a subject-matter expert is needed and thresholds must be calibrated on real data |
None of the options yields a ready-made graph out of the box — the limitations of the methods are removed only by combining them.
Therefore, at the start we fix the relationship schema, assemble a reference annotation on 200–300 fragments, and use it to choose where rules are sufficient and where a model is needed.
Implementation stages: from corpus audit to schema handover
The work order is structured so that each next stage relies on the verified result of the previous one: first we understand what is actually in the texts, then we fix which relationships we need, and only then do we configure extraction.
Skipping any step costs more — a schema built without a corpus audit breaks on the very first real documents.
-
Text corpus audit. We count volume and formats, the share of scans and tables, find duplicates, multilingual inserts, and documents with outdated details.
The output is a corpus map: what is extracted confidently, what requires preprocessing, and what will not enter the graph at all.
-
Relationship schema design. We agree on the list of entity types and relationship types, constraints, and inheritance rules, and review disputed cases with your analysts.
We fix the schema in a machine-readable form with versioning — changing it later will not require rebuilding the graph from scratch.
-
Extraction setup. We assemble dictionaries of synonyms, abbreviations, and reference books, configure normalization rules, duplicate merging, and entity linking to stable identifiers.
Here we also decide which relationships are derived by template and which by the meaning of the fragment.
-
Control samples. We manually annotate several hundred fragments and calculate precision and recall separately for each relationship type, rather than as one averaged number. We analyze errors by cause and return to the rules and dictionaries until the metrics stabilize.
-
Validation on real queries. We run typical questions from your specialists and compare the paths in the graph with how a human builds them. This catches relationship gaps that sample metrics do not see.
-
Solution handover. We provide the relationship schema, dictionaries, rules, graph rebuild scripts, and update procedure; we conduct a walkthrough with your team so the graph can be extended without us.
What we hand over: artifacts and support procedures
After delivery, the customer retains not only a working knowledge graph but also the full set of artifacts by which the team can develop the solution without us.
The schema, dictionaries, permissions, and procedures are stored in your repository with version control — they can be read, edited, and handed over to new engineers.
- Graph schema — node and relationship types, mandatory attributes, duplicate merging rules; the schema version is under change control.
- Entity dictionaries — normalized name forms, aliases, and conflict resolution rules when one entity appears in several spellings.
- Access rules — a role matrix: which node and relationship types are visible to whom, including restrictions on source documents.
- Update procedure — frequency of loading new texts, the threshold for rebuilding the affected part, and the procedure for manual review of disputed relationships.
- Solution documentation — the flow from text loading to writing to the graph, configuration description, deployment from scratch.
- Schema maintenance — how to add a new entity or relationship type without rebuilding the entire graph and preserve backward compatibility.
- Reference sample and rollback procedure — annotated texts for measuring precision and recall, graph snapshots for reverting before an unsuccessful update.
We hand over working configurations, not descriptions: during acceptance, your engineers themselves run the next batch of texts and update the graph. After that, the solution lives on your side — with a clear change procedure and verifiable quality.
Case study: knowledge graph for a technical documentation archive
The customer is a manufacturing enterprise with a large archive of technical documentation: equipment passports, maintenance regulations, change bulletins, and failure reports.
The documents lay scattered across folders and systems, search was keyword-based, and the connectivity of sections — which nodes fall into the same regulations, which failures pull the same nodes — was held in the heads of two engineers.
We assembled a pipeline: document structure annotation, entity and relationship extraction, term normalization (one node under five spellings — one record), and selective human review.
The relationships were placed in the Neo4j graph database — traversal of 3–5 hops stopped being manual assembly and is performed by a query.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Relationships in the graph | about 1,200, entered manually | more than 40,000, extracted automatically |
| Manual review of the document stream | the entire array | selective review ~8% |
| Answer to "which regulations does a node affect" | 20–40 minutes through folders | seconds, via traversal by relationships |
| Relationship completeness on the control sample | 55–60% according to expert assessment | over 90%, discrepancies resolved |
Separately about validation: we did not treat automation as the ultimate truth.
Engineers annotated the control sample, discrepancies were analyzed, and extraction rules were corrected — after two iterations the share of disputed relationships fell, and the graph became a working tool rather than a set of relationships that no one had checked.
How does relationship extraction differ from entity annotation?
Entity annotation answers the question "who and what is mentioned": the model goes through the text, identifies fragments, and assigns each a type — organization, person, project, regulatory document, date.
The output is a flat list of mentions with coordinates in the document. Such a list is convenient to count, but it cannot answer any question about relationships: it does not know who appointed whom or what follows from what.
Relationship extraction adds a second layer — relationship types between the found entities: "works at", "owns a share", "supplies", "reports to", "mentioned together with".
Here the context of mention is decisive: the relationship is derived not from word proximity but from syntactic role and predicate, so "the director signed a contract with the supplier" and "the director mentioned the supplier in the report" produce different edges with different confidence.
Between the two layers lies disambiguation. One organization in the corpus appears as a full name, abbreviation, and typo; a person appears by last name, initials, and position.
If mentions are not reduced to a single node, the graph falls apart into isolated fragments, and relationships are lost more often than with an annotation error.
Structural connectivity is what both tasks are done together for. Annotation without relationships gives a list of entities; relationships without annotation give edges with unknown endpoints.
Therefore, we validate the result not by precision on an individual mention but by the share of nodes included in a connected path and the number of relationships per entity: such a graph can already be used to build question answering and chain search.
Technical concerns: quality, volume, and maintenance
Concerns about a knowledge graph repeat: extraction completeness, entity duplicates, compatibility with an already adopted ontology, and the life of relationships after launch. We address each one — through the structure of the process, not promises.
How completely are entities extracted?
We calculate precision and recall on an annotated sample from your documents — separately by node and relationship types, not on a demo set.
Where the model makes mistakes, we add dictionaries and normalization rules; disputed fragments go to manual additional annotation and become training examples. We fix the acceptance threshold before work starts.
How does the graph get rid of duplicates?
One object arrives as an abbreviation, a typo, in a different case, in Latin and Cyrillic. The merging block compares variants by spelling, context, and external reference books and reduces them to a single node with synonyms. Every merge is logged and reversible: an erroneous merge is analyzed rather than remaining forever.
Will we have to abandon our ontology?
No. If you have an industry model, reference books, or an adopted classifier, extraction is carried out within their framework: node types, relationships, and attributes are taken from there, not from our default schema.
We show discrepancies separately — you decide whether to extend the model or normalize the data to fit it.
What happens when the corpus is updated?
The graph is extended in portions: new documents undergo extraction, relationships are added, and outdated ones receive a timestamp instead of deletion. This shows when a relationship appeared and what source it is based on.
A full rebuild is needed only when the schema changes — it is done with validation on the reference sample so that metrics do not drop.
Let's discuss the task: what is needed to start work
The scope estimate for a knowledge graph depends on two things: what the corpus is and which relationships you consider mandatory in the output.
To avoid answering "approximately and someday", the inputs below are enough for us — from them we estimate the number of annotation iterations, the effort required for entity extraction, and the boundaries of the pilot area.
- Corpus composition: how many documents and types, approximate volume, languages, and in what form they currently exist — text files, layout, scans, an export from an accounting system, or correspondence.
- 5–10 examples of real documents of different types: they reveal annotation, term density, and the share of tables and images — this is more accurate than any questionnaire.
- Current dictionaries and reference books: classifiers, registries, lists of synonyms and abbreviations. If they are already maintained, we adopt them as a basis rather than building terminology from scratch.
- Expected result for relationships: which entity and relationship types are needed, whether each relationship requires a weight or confidence score, and how deep to build chains.
- Access rules: where the data physically resides, who the owner is, whether export is allowed, and whether there are fragments that must not be processed.
- Application point: which process should receive the graph — search across the database, reporting, answering employee questions, or handling requests.
This is enough to come back with a scope estimate and implementation plan: where the pilot area runs, what we validate on it, and by which metrics we decide to expand to the entire corpus.
Describe the task and attach document examples — we will return with a timeline calculation, stages, and the set of checks for the pilot.
