Why text summarization breaks on large volumes
On a short email, text summarization looks like a solved problem. On a long document, it falls apart: the content does not fit into a single pass, it is cut into parts, and each part is compressed in isolation from the others.
Meaning rests on connections between sections — a definition in the first chapter, an exception in the fifth — and during compression these connections are the first to be lost.
On a validation set, this yields a noticeable share of lost facts, and no one sees it until the summary is checked against the source.
The second typical failure is that the summary duplicates the introductory part. A generalized announcement and table of contents are readily retold because they are already written in compressed form, while the body with figures, deadlines, and caveats is omitted.
The reviewer has to return to the source text, and all time savings are nullified: instead of reading the summary, they read both documents.
Then a third factor comes into play — the flow of requests grows faster than it can be manually reviewed. Manual review of the flow stops being a control point and becomes a queue: decisions wait, and loss of meaning during compression is discovered only by its consequences.
We explain how this is addressed — where to place quality checks and how to retain meaning in long documents.
What implementation delivers: processing speed and completeness of summaries
The effect of summarization is measured not by "text quality" but by three numbers: what share of documents passes without a human, how long it takes from arrival to a finished summary, and how many key conditions are not lost in the process.
We capture these metrics on a labeled sample of the customer's documents before launch and repeat them on the real flow — then the discussion is about facts, not impressions.
Completeness and coherence of a concise summary
We calculate completeness by preserved entities: parties, amounts, deadlines, obligations, clause numbers.
The extractive approach guarantees that the wording is taken from the source verbatim, but produces a "snippet" of sentences; abstractive writes coherent text, but requires checking for factual distortions.
Therefore we use a hybrid scheme: extraction and quotes for legally significant blocks, paraphrasing for the descriptive part.
As a result, the summary does not need to be rewritten: the share of edits falls from 40–60% at the start to 10–15% after tuning on the customer's corpus.
How much manual work is removed from employees
Typical flows — requests, meeting minutes, contracts, reports — go into automatic processing entirely if the document falls into a class known to the system.
70–90% of such documents pass without a human; the rest are flagged and sent for review along with an indication of where the model is uncertain.
The time to a finished summary is reduced from hours waiting in an employee's queue to minutes, and with it goes a class of errors: "the condition is in the text, but no one read it to the end." We deploy processing on-premises using models of the Llama class — data does not leave the customer's environment, and the issue of approving data exports does not arise.
Extractive and abstractive text summarization: comparison of modes
Extractive mode returns the original sentences: facts are not distorted, but the text remains choppy. Abstractive paraphrases in its own words — it reads smoothly, but details appear that were not in the source.
The hybrid scheme "first select fragments, then paraphrase" removes both extremes if the paraphrase is limited to what was selected.
The choice between an external service, an in-house model, and a hybrid is decided not by quality on the test set, but by where the data lives and who is responsible for an error in the finished document.
| Option | When it fits | Limitations |
|---|---|---|
| Ready-made external service | Heterogeneous flow, no in-house labeling | Data leaves the environment, terms are distorted |
| Fine-tuning an in-house model | Narrow domain, there are "document — reference" pairs | Labeling and regular fine-tuning are needed |
| Hybrid scheme | Long documents, need an exact fact and coherent text | Two control stages, longer preparation |
What the external service takes on
Universal texts: languages, resilience to typos, model updates without your involvement. External services such as OpenAI provide a ready-made abstractive mode, tested on a broad corpus.
There is one weak spot: the service normalizes domain vocabulary in its own way, and this can only be checked on your sample.
Where your own environment is mandatory
Personal data, trade secrets, industry regulations — the text does not leave your environment. Extractive mode works here as a safeguard: the summary contains only original wording, while the hybrid scheme adds a paraphrase on top of what was selected.
We calculate the limitations of the methods on your documents: the share of lost facts and the share of fabricated details.
How we implement: stages of engineering work
We structure the order of work so that summary quality is measured rather than discussed subjectively. Each stage ends with an artifact that can be opened and checked: we approach model tuning with an already fixed definition of a good result.
- Corpus audit. We analyze real documents: formats, languages, share of scans, text lengths, tables, and attachments. The outcome is a data suitability report and a list of what we clean before feeding it to the model.
- Processing scheme. We determine where extractive summarization is appropriate (fast, fragments remain verbatim) and where abstractive is appropriate (paraphrasing with reformulation). We fix the block splitting, overlap, and behavior on short documents.
- Tuning. We assemble the processing pipeline and select the compression level: how many facts and details should remain in the summary.
- Validation set. We run documents that the model did not see during tuning and compare the summaries against the reference.
- Acceptance. We compare the result against the agreed criteria and review disputed cases line by line.
- Solution handover. We provide the configurations, a description of the processing scheme, and an operating manual, and train the customer's team.
Acceptance criteria for summary quality
We look at three things: whether key facts are preserved, whether there are fabricated details, and whether the summary fits within the specified length. We separately check robustness — the same document with a different order of blocks should not produce divergent conclusions.
We agree on the threshold for the share of correct summaries before work begins, so that acceptance is a formality, not a negotiation.
What remains with you: configurations, schemes, regulations
After delivery of summarization, what remains with you is not "a model in the contractor's cloud" but a working environment that can be developed and migrated on your own.
We hand over not a presentation but a set of artifacts: using them, an engineer who did not participate in the project can deploy and configure the system from scratch.
- Processing configurations — parameters for splitting text into fragments, target summary length, thresholds for cutting off insignificant blocks, with comments on what to change for each document type.
- Data flow diagrams — from the source (file, API, database) to summary output, with marked manual review points and rollback branches.
- Access control rules — who sees the source text, who sees only the summary, and how roles and visibility scopes are assigned at different levels.
- Manual review policy — sample size, acceptance criteria, action in case of discrepancy, and the person responsible for approval.
- Instructions for employees — how to submit a document for processing, what to do with an incorrect summary, and where to report a failure.
- Reference text set — examples with the expected summary; they are run after any configuration change.
- Metrics log — accuracy on key entities and the share of summaries sent for revision.
How to maintain the solution on your own
Edits to query wording, dictionaries, and thresholds are made in the configuration and do not require rebuilding.
Once a quarter, run the reference set, compare metrics with the previous snapshot, and if quality drops, roll back to the previous configuration version — it is stored next to the current one.
Case: request flow processing time decreased threefold
The customer is a service company with distributed teams: thousands of policies, meeting minutes, and technical specifications have accumulated in an internal repository based on Confluence.
Analysts manually reviewed a flow of about 400 documents per month to compile short summaries for executives. One summary took 25-30 minutes, the queue grew faster than it was processed, and decisions were made based on outdated reports.
What it was before
The main problem was semantic omissions. Manual review produced 30-34 lost key points per thousand documents: deadlines, dependencies, and responsible persons fell out of the summary if they appeared in appendices or tables.
The share of fully automatic summaries was zero — a machine attempt required human editing, and no time savings occurred.
What changed after implementation
We assembled a two-stage scheme: extractive selection of sentences by semantic proximity to the query, then abstractive reformulation with fact-checking and preservation of links to the source fragments.
The model is trained on the internal corpus and confidently handles the customer's terminology, including tables and nested files.
After implementation, 78% of summaries go into work without edits, and the remaining 22% require a quick review — on average 4 minutes instead of 30. The number of semantic omissions decreased to 3-5 per thousand documents, and the flow processing time decreased threefold.
What to do if short summaries are generated by an external service
An external service handles the flow while texts are homogeneous and non-sensitive.
As soon as contracts, correspondence, or HR documents get into it, there appears a data environment you do not control, and with it — rate limits on requests and model version changes without warning.
Summaries on the same inputs begin to differ, and there is nothing on which to fix a quality metric.
Migration of processing is not done with a single switch. First, a shadow run: your own environment processes the same sample, and summaries are compared by coverage of key entities and the share of lost facts.
Then routing — sensitive and long documents go inside, anonymized ones remain outside, and if the external address is unavailable, the queue switches to your own environment without losing tasks.
Compatibility of schemes saves the most time. We preserve the previous contract: the same request and response fields, the same output format, the same error codes and retry rules — the calling service changes only the address, through configuration, without code changes.
| Option | When it fits | Limitations |
|---|---|---|
| Keep the external service | homogeneous flow, anonymized texts, small volume | request limits, model version changes, dependence on third-party availability |
| Hybrid: part of the flow in your own environment | there are sensitive documents or narrow terminology | need to compare summaries from two schemes and maintain routing rules |
| Full migration | volume is growing, own terminology and predictable latency are needed | own infrastructure, domain tuning, own quality checks |
| Own environment, external as backup | flow stoppage is unacceptable | maintain a compatible contract and monitor summary divergence |
We choose the option by the numbers: the share of discrepancies on a common validation set and the volume after which the external service stops handling the flow.
Technical doubts before starting the project
Doubts before the start rarely concern the model itself — more often the question is whether it will be possible to measure summarization quality and stay within data constraints. Below are four questions that are asked at almost every assessment, with substantive answers.
How to check quality if there are no ready-made references?
We assemble a reference set of 50-80 documents and for each one record the facts that must appear in the summary.
Then we calculate the share of preserved facts and the share of statements that were not in the source — for the abstractive approach, the second metric is more important than the first.
What happens to tables, diagrams, and lists?
A table is parsed row by row: each row is described in text and included in the summary, while the grid itself is preserved as a separate block.
Diagrams are processed together with captions — otherwise the caption loses its connection to the data and an incorrect number appears in the summary.
The documents are confidential — where does the text go?
Processing is deployed in the customer's environment: the model, weights, and logs remain inside, and the service does not access outside. If an external model is needed, anonymized fragments are sent out, not the entire document.
Multilingual texts and narrow terminology — can one service handle it?
The language is determined per paragraph; term glossaries are maintained separately for each language and applied during post-processing. Without a glossary, the model replaces an industry term with a general-language synonym — the summary reads smoothly, but the meaning is lost.
Let's discuss the task: what to prepare for the conversation
The more specific the inputs at the start, the fewer clarification rounds and the more accurate the estimate. Below is a list of what speeds up task analysis: if some point is not yet defined, that's fine — we will fix it together during the conversation.
- 5–10 examples of real documents that need to be shortened: letters, reports, meeting minutes, customer requests. Anonymized copies are sufficient.
- Flow volume: how many documents per day and per month, average and maximum length, peak load.
- Expected result: a summary in one or two sentences, structured bullet points, or a shortened version preserving the original wording.
- Languages and terminology: working languages, industry abbreviations and names that must not be rewritten.
- Confidentiality requirements: where data is processed — in your environment or on our side, whether there are restrictions on transferring texts to external services, and whether access logs are needed.
- Scenario constraints: response speed, share of manual result verification, system behavior when uncertain.
- Responsible people on your side: who provides examples, who accepts quality, who approves changes.
After discussing the task, we will return with an estimate and work plan: what stages there are, what we do at each, how we measure summarization quality, and at what point the solution can be stopped or expanded.
