Why keyword extraction breaks on real-world texts
Keyword extraction looks like a simple task while texts are uniform and clean: feed a document — get a list of terms.
On a real corpus — contracts, meeting minutes, scans, correspondence — accuracy on real texts drops, and along with meaning come headers, footers, requisites, and fragments of tables.
The first source of losses is terminology itself. In one department they write "metering unit", in another — an abbreviation, in a third — a typo from haste, and the same object splits into a dozen spellings.
Each customer has their own domain dictionary, and without it keyword extraction does not understand what to consider a variant of the norm.
The second is noise in documents. Text comes from layout, from tables, from scans after recognition, where characters get glued together and confused. Then the team resorts to manual term enumeration: stop-word lists, manual rules, synonyms for each new template. The construction holds until the first change of supplier or document form.
For the process, this results in archive search not finding the needed file, and tag-based summaries showing an incomplete picture — decisions are made on data that cannot be trusted. We break down how this is closed and why quality is lost not in the model, but at the input.
Implementation effect: term coverage and volume of manual annotation
At the start of a project for parsing technical regulations, an expert annotated terms manually: about four hundred significant phrases per thousand pages.
Up to a third of them he physically did not have time to mark — they surfaced later, at approval, when the document had already moved further along the process.
A missed term costs more than an extra one: it does not get into the dictionary, and in subsequent documents the system again does not see it.
After setting up keyword extraction, terminology coverage reached 92–96% of expert annotation on the same document type.
The basis is a domain dictionary assembled from already annotated fragments; KeyBERT is fine-tuned on these pairs, so compound names like "cooling circuit pressure sensor" remain intact instead of splitting into separate words.
Manual annotation is reduced by about five times and changes character: the expert does not annotate from scratch but confirms or rejects what is proposed, spending time on controversial cases.
The error class "missed term" does not disappear completely, but ceases to be systemic — it accounts for isolated cases instead of a third of the corpus.
A stable result on new documents is maintained because the dictionary and rules are updated on every pass, rather than configured once for the first file.
Corpus processing speed changes proportionally: a batch run of thousands of pages takes hours, so re-annotation after regulation changes is done without a separate project.
Approaches to extraction: rules, statistics, language models
The choice logic begins not with the model, but with the nature of the data. Where terminology is closed and repetitive — regulations, nomenclature, standard contracts — dictionary rules give predictable hits and require no annotation.
As soon as the corpus becomes heterogeneous, the dictionary starts to lag: each new wording has to be entered manually.
A trainable model wins with a stable domain and several hundred annotated documents — it catches phrases that are not in the reference book. A language model is connected where there is no literal match but meaning is preserved: paraphrases, synonyms, abbreviations.
On most projects we assemble a hybrid scheme: candidates are supplied by rules and statistics, and a model ranks them — rare terms are not lost, and noise is filtered out.
| Approach | When it fits | Limitations |
|---|---|---|
| Dictionary rules | Closed terminology, frequent repetitions, need for an explainable result | Does not find new wordings, dictionary maintained manually |
| Trainable statistical model | There is annotation, domain is stable, speed on large volumes is important | Depends on annotation quality, degrades when topic changes |
| Language model | Paraphrases, synonyms, need to account for context | Requires resources, limited by document length |
| Hybrid scheme | Heterogeneous documents, high requirements for recall and precision | Harder to maintain, needs control at each step |
We fix the choice by measurement, not words: we run the options on your document sample and look at the share of exact hits and recall for each class of phrases. The scheme remains replaceable — components change separately, without rewriting the entire extraction contour.
How implementation goes: from corpus analysis to solution handover
The order of work hardly depends on where keyword extraction is applied — in knowledge base search, handling requests, or review analytics.
First we analyze the corpus, then we fix the annotation, and only then we configure the model: if the steps are swapped, the result is a solution that shows beautiful examples and does not transfer to a real document stream.
- Corpus analysis. We look at what the stream consists of: document types, languages, share of duplicates and service text, whether there is ready annotation. The output shows where normalization is needed — encodings, headers and footers, garbage after scan recognition.
- Annotation specification. We agree on what we consider a key phrase: length, boundaries, inclusion in the heading, handling of synonyms and abbreviations. We annotate a reference set and calculate inter-annotator agreement — without this, quality metrics mean nothing.
- Extraction scheme. We assemble the contour: a base layer on statistics and word co-occurrence, on top — a domain model, plus normalization, stop lists, and customer term dictionaries.
- Domain tuning. We refine parameters on the validation part, add synonyms and product names, fine-tune the model on annotated examples.
- Check on a holdout set. We fix the test set in advance and do not touch it during tuning: we calculate recall and precision by phrases, analyze errors by type. The set remains with the customer as a regression one — any further edits are checked on it.
- Solution handover. We give the code, configurations, model, annotation scheme, and dictionary update regulations. We train the team so that edits are made without us.
What remains with the customer: configurations, annotation, regulations
We hand over not only a working service, but also everything needed for its independent development.
The handover kit is designed for your team: for engineers — settings and code, for analysts — annotation and rules, for the manager — the regulation by which the solution lives on.
- Extraction scheme configuration: text normalization rules, stop words, part-of-speech weights and cutoff thresholds are placed in a separate settings file, not hidden in code.
- Annotated corpus of 1–3 thousand documents in your domain: reference key phrases divided into training and validation parts.
- Annotation instruction: what we consider a key phrase, how to merge compound terms, how to handle controversial cases — so that new data is annotated consistently by different people.
- Text processing scheme from document intake to result output: where checks are, where event logs are written, where to add new steps.
- Dictionary and model update regulations: frequency, who approves changes, how quality degradation on the validation set is recorded.
- Set of checks and run metrics on the reference corpus — one can see whether extraction became more precise or worse after edits.
- Integration documentation: request and response formats, volume limits, behavior on empty result.
Each item is a file, documentation section, or granted access, not a verbal agreement. A month after launch, you yourself change the cutoff threshold or add your own annotation rules and immediately see the effect on the validation corpus.
Case: how document parsing quality changed after implementation
Before implementation, parsing relied on regular expressions and a term dictionary that analysts updated manually after each complaint.
On a stream of similar documents — contracts, technical passports and appendices to them — this was enough for the first hundred pages, then the dictionary diverged from real wordings, and checking turned into continuous rereading.
We annotated a reference sample by field classes and began measuring precision and recall separately for each class, rather than one average score.
Candidate extraction was assembled on spaCy, dictionary anchors and additional counting of compound terms with abbreviations across the whole document, not by paragraph, were added.
The confidence threshold was moved to a separate queue: the analyst sees not the whole stream, but only controversial places.
Metrics before and after
On the same stream, recall grew from 61% to 93%, precision — from 74% to 96%. The share of documents going to manual review fell from 100% to 18%, and parsing a 200-page batch was reduced from three hours to half an hour.
The class of missed terms also shifted. Previously, compound terms and abbreviations whose decoding was in another section were lost; now residual misses are terms inside scans of tables and handwritten edits, where there is no text layer at all.
Each such case goes into the review queue, the dictionary is updated, and on the next document type quality does not drop from scratch.
How task-specific annotation differs from a ready-made term dictionary
A ready-made term dictionary is a static list with form normalization and a match threshold. It confidently answers the question of whether a word known to us occurs, but not whether this phrase names the essence of the document.
Such a list is transparent, explainable to a reviewer, and can be edited with one line — therefore as a starting point it is appropriate almost always.
Task-specific annotation begins not with a list, but with a scheme: what exactly we consider a key phrase in your domain — a product property, defect, party to a transaction, action or condition.
Then, on the corpus of your documents, criteria, controversial cases, and the rule for resolving them are fixed, and annotation is carried out by several people with regular reconciliation.
We separately calculate the share of controversial fragments: if it is high, the instruction is raw, and the model will learn noise instead of a pattern.
The practical difference between the approaches is that the dictionary covers known entities — regulatory terms, codes, stable names — while annotation defines the boundaries of the phrase and its priority among others.
The working strategy is more often hybrid: the dictionary serves as a source of initial labels and a hard layer of rules, on top of which a model trained on annotation parses real wordings.
The choice of strategy is dictated by language behavior and the cost of error.
If the class of terms is closed and changes rarely, the dictionary is enough for a long time; when wordings are updated, synonyms and abbreviations arrive, quality hits a ceiling that is removed only by annotation.
At the start we run both options on one set of documents and show at what share of phrases they diverge.
Common technical doubts: annotation, quality, support
Engineering doubts about keyword extraction rarely concern the task itself — more often the transition: will the solution withstand new document types, how much annotation will have to be collected, by what criteria to accept quality, and what will remain with the team after handover. Below are answers to what comes up most often at the evaluation stage.
New document types arrive — will extraction degrade?
Phrase-level features and domain term dictionaries are configured separately from the model, so a change in document type does not require rebuilding.
It is enough to annotate a sample of new documents and run evaluation on its holdout part: the discrepancy in precision and recall is visible before release, not on a live stream.
How many documents need to be annotated?
At the start — several hundred fragments; then active fine-tuning works: cases with low model confidence and borderline terms go into annotation first.
Each subsequent hundred examples gives more than the previous one, and the annotation volume grows more slowly than extraction quality.
How do we accept the work and measure quality?
We fix precision and recall thresholds before the start — separately for each category of key phrases, for rare terms, and for documents with poor scans. We consolidate controversial cases into one list: your side makes the decision on each, not a metric blindly.
Will we be able to support the solution on our own?
The handover includes extraction configuration, new terminology dictionaries, evaluation runs on a control sample, and a short update regulation.
Adding terms does not require retraining, and scheduled checking is an evaluation run on fresh documents with fine-tuning if degradation occurs.
Let's discuss the task and come back with an estimate and work plan
To keep the conversation straight to the point, attach samples: 20–50 real documents from those that will be processed, including "inconvenient" ones — scans, texts with tables, headers and footers, and broken layout.
The closer the sample is to the work process, the more accurately we will estimate where keyword extraction will hit the quality of the source text, and where rule and dictionary tuning is enough.
It is useful to prepare for discussion:
- list of domain terms and entities that must be included in the result;
- examples of "correct" key phrases annotated manually on at least ten documents;
- expected error class: which is more critical — missing a significant phrase or adding an extra one;
- requirements for the output format: table, file, database record, or output directly in the interface;
- access and storage restrictions: where data is stored and what cannot be taken outside the contour;
- target volume: how many documents per day are processed and at what speed the result is needed.
The estimate is built by stages, and you see its logic. Task specification fixes what we consider a key phrase and how we measure quality. The scope estimate shows which parts are solved by dictionary and rules, and which require a model and annotation.
The implementation plan breaks the work into steps with a verifiable result at each. The pilot stage runs on a limited sample: we compare the output with manual annotation, look at precision and recall, and only then scale to the entire array.
Describe the task — we will come back with an estimate, stage plan, and pilot acceptance criteria.
