Relation Extraction on real texts: where accuracy is lost
In contracts, protocols, and technical descriptions, semantic links between entities always exist: who reports to whom, what references what, which condition restricts delivery.
While relation extraction from text is done manually, these relations live only in the analyst's head and in their notes. Manual document review does not scale: on a corpus of tens of thousands of pages it turns into a separate workstream with its own timelines and backlog.
The first obvious step — templates and regular expressions — breaks on real wording.
“Delivery is made before,” “no later than,” “within the period agreed by the parties” describe the same relation, while the template catches only the first variant and then produces false positives.
The more relation types there are, the harder it is to maintain the rules: every new word requires a new exception, and exceptions begin to conflict with each other.
But the main loss of accuracy happens not in the model, but before it — in annotation quality. A relation must be labeled as a separate object: type, direction, exact boundaries of participants.
Therefore annotation is done in a tool that stores the relation as an independent entity — for example, Label Studio — otherwise the model learns to guess the relation by word proximity and adds its own false positives on top of those already present.
Next we break down how this is addressed: a relation type schema for the subject domain, annotation guidelines, measurement of inter-annotator agreement, and an accuracy report separately for each relation type — instead of one average number that explains nothing on real texts.
Effect of implementation: processing speed, completeness and accuracy of relations
The first measurable effect is document processing speed. The relation extraction module parses a typical contract or protocol in seconds, whereas manual annotation of the same text takes an analyst tens of minutes.
On same-type documents, the flow runs in batches: a rule configured for a template is then applied without human involvement.
Completeness of relation extraction is calculated on a held-out set that the model did not see during training.
We record not one number but a profile: the share of relations found without manual correction, separately by relation types — “supplier — product,” “employee — role,” “document — basis.” Model accuracy is tuned by confidence threshold: higher threshold — fewer false relations, but more misses, and we select this balance for the task, not for a pretty metric.
The share of manual correction is the main working indicator. At the start it can reach half of the found relations; after fine-tuning on the customer's annotation it drops to single-digit percentages on familiar text types.
We read growth in the correction share as a signal: the format of incoming documents has changed or a new supplier with a different layout has appeared.
Robustness on new text types rests on a cycle: uncovered examples go into annotation, the model is fine-tuned, quality is checked against the control set.
We do entity and relation annotation on an open stack — spaCy and formats compatible with it, so data remains portable and not tied to one vendor.
Approaches to Relation Extraction: rules, models, hybrid schemes
The choice of scheme is determined by the structure of the corpus and the cost of error, not by the trendiness of the method. If formulations are same-type and each trigger needs to be explained — rules and dictionaries work.
If texts are real, with paraphrases and typos — neither rules nor a model trained on a small sample hold quality on rare relation types.
| Variant | When suitable | Limitations |
|---|---|---|
| Rules and dictionaries | Narrow domain, stable formulations, explainability of each trigger is required | Maintained manually; break on paraphrases; as the number of rules grows, conflicts appear between them |
| Trained model on own annotation | There are several thousand annotated relations, the domain is stable, relation types are fixed | Consistent annotation is needed; when the domain or document format changes, quality drops and requires retraining |
| Pretrained language model with fine-tuning | Little annotation, diverse formulations, long texts | Weaker on rare classes; harder to operate, requires leakage control and regular updates on new data |
| Hybrid scheme: rules plus model | Mixed corpus where part of the relations is defined by hard conditions | Harder to debug: two sources of errors and a separate conflict resolution layer is needed |
In practice the hybrid scheme more often wins: rules cut obviously false positives and cover relations with rigid context, and the model picks up paraphrases.
Quality is measured separately for each relation type — an average number hides failure on a rare class, which actually determines the applicability of the solution.
How the project is built: from corpus audit to model handover
Relations between entities are set not by text, but by schema: if relation types are described approximately, the model will learn noise and will confidently err on new documents.
Therefore the work goes from corpus measurement to handover, and each stage ends with a verifiable result.
- Corpus audit. Count the real volume, duplicates and encountered entity types, note which relations are missing for training.
- Annotation schema. Fix relation types, their direction and rules for disputed cases — negations, enumerations, mentions in parentheses. Agree on the schema before annotation, otherwise instructions are rewritten together with the completed material.
- Annotation and quality control. Part of the documents is annotated by two people independently, and discrepancies refine the instruction. Agreement is calculated in percent: a low value means both the human and the model understand the task differently.
- Training. A model on a pretrained language representation is fine-tuned on the corpus; we look separately at rare relation types, which are lost first.
- Validation on a held-out set. Metrics are calculated on documents outside training and separately for each relation type — an average score hides failure on rare classes.
- Integration into the process. Connect the model to the current loop: batch processing, confidence estimation for each relation, manual review of doubtful cases. Set the confidence threshold so that the review flow does not grow faster than the team can process it.
- Handover. We hand over weights, deployment configuration, annotation schema and metric breakdown — the team does fine-tuning on new data itself.
What we hand over: annotation schema, model, operating procedure
A relation extraction solution is useless if the annotation logic lives in one engineer's head and the model cannot be retrained without the author.
Therefore, together with the working model, we hand over to the customer's team a set of artifacts — using them a new specialist reproduces the result, and relation extraction continues to work on data that appears after launch.
- Annotation schema. Entity types and allowed relations between them, boundary rules, priority for overlap and analysis of disputed cases where annotators most often disagree.
- Instruction for annotators. Work order, “correct / incorrect” examples on real fragments and a consistency criterion — the share of matching relations between two annotators.
- Model documentation. Which relation types are extracted, what data the model was trained on, where the applicability boundary lies and what error class it allows on short and incomplete texts.
- Set of control examples. Annotated fragments with reference relations: a run through them after each fine-tuning shows whether the model has degraded.
- Fine-tuning procedure. When to collect new data, how to annotate it according to the same schema, how to check quality before rollout and at what point to roll back the previous version.
- Quality report and thresholds. Precision and recall separately for each relation type, plus the confidence threshold below which a relation goes to manual review rather than into the system.
Case: extracting relations between parties in contracts and requests
The task came from a legal service: several thousand contracts and requests per month, with relations between parties extracted using template heuristics. Roles got confused, and a non-standard clause broke document parsing entirely.
The control corpus was assembled from annotated documents and the annotation was cross-checked with the open CUAD dataset — metrics were calculated only on it so that the result did not depend on one export.
Error classes and what was done about them
| Error class | Cause | What was changed | Share before → after |
|---|---|---|---|
| Relation goes to the neighboring party | Annotation did not separate payer and guarantor roles | Separate role labels and a trigger dictionary by clause types | 18% → 4% |
| Relation break between paragraphs | The analysis window broke at the paragraph boundary | Consideration of internal references across the entire document text | 12% → 3% |
| False relations in details | The header was parsed as the contract body | Filter by semantic sections and document structure | 9% → 1% |
Relation accuracy on the control corpus grew from 0.74 to 0.92, and the share of documents sent to manual review — from 41% to 9%.
The decisive factor was not model complication, but separation of roles in annotation: tying a party to an obligation no longer depended on the order of mentions in the text.
The control corpus remained in the project as a regression set: each iteration is run against it in full, and a new relation type is added through annotation, not by code edits.
This gives predictable rollback — when the metric drops, you can see on which documents the annotation diverged.
How relation extraction differs from keyword search
Keyword search answers the question in which documents a word occurs, and returns a list of files by relevance. That is enough to find the primary source, but not to understand how people, companies and events are connected.
The document remains closed to the system: the answer lies inside, and only the file name comes out.
Relation extraction works deeper. The model labels entities in the text — organizations, persons, dates, legal roles — and determines the relation between entities: who reports to whom, who supplies to whom, who signed a contract with whom.
The output is not a list of documents, but triples “entity — relation — entity,” that is, structured data.
From these triples a knowledge graph is assembled: nodes — entities, edges — relation types with indication of the document and fragment from which the relation was extracted.
Traversals go through the graph that search cannot access: two counterparties can be connected through a common executive in a third company, although they are not mentioned together in one text.
The difference is especially noticeable in document analytics. Keyword search gives the share of texts where a mention occurred; the graph answers the question how many counterparties are connected to an organization and through what type of relations.
The former measures the presence of a word, the latter — the structure of the subject domain, and only on it are queries like “who is connected to X not directly” built.
Doubts about data, annotation and model support
Doubts before the start are usually the same: is there enough data, what to do with an already annotated corpus, will the model handle heterogeneous texts, and who is responsible for quality after launch. We address typical cases.
We have little annotated data — will the model be enough?
We count volume not in absolute numbers, but by coverage of relation types. We go in iterations: starting annotation, measurement on a held-out set, adding examples only where the model errs.
Texts without annotation also work — formulations that are not in the sample are still seen by the model.
Annotation partially exists, but was made for another task
We map the existing relation schema to the target ontology — we re-annotate only the missing types, and they are usually the minority. Relations that are reliably derived by rules do not go into manual annotation.
Documents are heterogeneous: contracts, correspondence, scans, tables
We split several document types into separate preparation branches, but bring them to a single relation ontology — otherwise the results are incomparable with each other. We train the model on a sample from each type so that one format does not pull predictions toward itself.
What happens to the model after launch?
Support is a cycle, not a one-time measurement: we monitor the share of confident extractions and new formulations, and errors go to the next fine-tuning iteration. We version the model and annotation, so rollback to the previous working version is a matter of minutes.
Remaining questions — about access, result export formats and acceptance criteria — are resolved before the start so that annotation does not have to be redone during the process.Let's discuss your relation extraction task and corpus volume
A substantive conversation starts with the corpus, not with a discussion of the model.
Relation extraction is checked only on your texts: on same-type documents any scheme looks working, but on real contracts, reports or correspondence you immediately see which relation types are missing and where the rule fails.
From you it is useful to show:
- 20–50 real documents or an export — as is, together with noise, scans and non-standard formatting;
- a list of entities and relations needed on output: types, direction, cardinality;
- one document with manual annotation — a reference against which we will calculate quality;
- where the data comes from and where the result returns: storage, database, internal systems;
- access restrictions: where texts can be processed and what cannot be moved outside your perimeter.
On our side we clarify acceptance criteria: what share of relations must be extracted on your documents, what level of false positives is acceptable, what we consider a miss and what — an extra relation. Both the scope of work and the order of stages depend on this.
Next we prepare a project plan with checkpoints: where you review the result on your texts and what can be adjusted before final delivery. Describe the task and corpus volume — we will return with an estimate and plan.
