Why a legal AI assistant errs on real contracts
At a demo, a legal AI assistant answers on a contract confidently and to the point. On a real package — main text, appendices, protocol of disagreements, approval emails — it returns a clause from an earlier version that no longer exists in the signed document.
It is not the model that breaks, but context extraction.
A contract clause makes no sense in isolation from definitions, appendices, and references like "in the manner provided for in clause 7.2": when chunked into fragments, the chain breaks, and retrieval pulls a similarly worded line from another section or from someone else's template.
Then language model errors kick in. A missed condition is filled in with a plausible reference to a legal norm, and such an answer is indistinguishable from a correct one in tone or structure — the lawyer rechecks every paragraph manually, and the time savings disappear.
Errors on risky terms are the most expensive — auto-renewal, unilateral termination, unlimited liability, jurisdiction outside your jurisdiction.
They are either missed or flagged together with harmless clauses, and after two such runs the assistant stops being trusted: contract review goes back to reading by eye.
We break down exactly where processing breaks — in context extraction, in verifying statements against the contract text, or in risk detection rules — and what closes this gap.
What changes after implementation: speed and a single base
Reviewing one package — a contract with appendices, a protocol of disagreements, correspondence on disputed clauses — before implementation took a lawyer several hours: finding the norm, checking the version, running through a checklist manually.
A legal AI assistant changes the order of work: first the machine marks up the document structure, surfaces relevant norms and cases with a similar fact pattern, and only then does a human join the review.
Retrieval here is hybrid — by meaning and by exact wording, with reranking of candidates; otherwise a condition like "no later than ten business days" gets lost among contextually similar norms.
A single knowledge base is assembled once and lives with versioning: the corpus of norms, internal templates, and closed precedents are indexed together, and when a document version changes, old links are marked as outdated rather than silently returned.
Sources are ConsultantPlus and internal regulations, so an answer always comes with a reference to a clause and can be verified rather than taken on faith.
Risk highlighting follows a formalized checklist: unilateral termination, auto-renewal, jurisdiction, penalties, unlimited liability.
The assistant shows the fragment, the risk class, and the norm it contradicts — the share of missed conditions falls because the check does not depend on whether the lawyer is tired by evening or remembered the right clause.
The shift in workload is visible in measurable terms: time to first review drops from hours to minutes, standard documents are closed by template with automated verification, and the specialist joins on deviations.
Every trigger and every rejected suggestion is logged — so you can see how many documents go through without edits, where escalations have decreased, and which checklist items still need refinement.
Connection options: ready-made service or in-house development
Connecting the assistant to company documents is first of all a decision about the data processing boundary, not a choice of interface.
The contract corpus, internal opinions, and client correspondence provide output quality, but these are exactly what cannot be taken outside the perimeter without legal consequences.
Where this boundary lies determines both the scope of support and which questions the assistant can handle correctly.
Implementation options
| Option | When it fits | Solution limitations |
|---|---|---|
| Ready-made subscription service | fast start needed, documents are not sensitive | corpus and queries go to the vendor, output is configured within their framework, behavior changes with their updates |
| External model plus your own retrieval layer | part of the corpus may be transferred outside | document fragments leave the perimeter, quality depends on availability and limits of the external service |
| In-house build in a closed perimeter | attorney-client privilege, personal data, isolated network | requires your own compute capacity and model updates; we maintain the knowledge base together with you |
| Hybrid: sensitive inside, the rest outside | mixed corpus and different case categories | requires query routing and control that data does not cross the boundary |
A closed perimeter is not automatically the best, but it is the most demanding in support: model updates and base reindexing are on us, but not a single document fragment leaves your network.
An external model wins at the start and in breadth of knowledge, but requires a written decision on which case categories may be transferred to it.
Practice shows: the choice is determined not by technology, but by how many queries can actually be allowed outside — often it is half, and then a hybrid closes the task without compromising quality.
Implementation procedure: from document audit to handover into operation
We structure the work so that answer quality is confirmed on your documents, not on demo examples. First inventory of the corpus, then the processing schema, and only then configuration — errors in source markup otherwise turn into incorrect conclusions in the output.
- Document audit. We assemble the corpus: contracts, opinions, templates, correspondence archives. We note formats, duplicates, outdated versions, and documents with restricted access.
- Processing schema. We design chunking, metadata (document type, date, counterparty, area of law), and rules by which the assistant must cite the source.
- Risk detection. We formalize indicators: hidden conditions, missing reservations, discrepancies between versions. We support each rule with examples of triggers and false positives.
- Retrieval setup. We configure fragment extraction and ranking so that the answer relies on relevant clauses, not on formally similar paragraphs.
- Testing on real data. We run a closed set of cases with a known correct answer: we measure the share of correct references and missed risks.
- Solution handover. We deliver the configuration, schemas, corpus update regulations, and error analysis; we show lawyers how to read an answer together with its source.
After that, the corpus and rules live without us: the documentation describes how to connect new document types and change trigger thresholds so that the legal AI assistant does not lose quality as the base grows.
What we hand over: schemas, access policies, regulations
We deliver the solution not as a "black box," but as a package your team can work with further without us.
Everything that describes the behavior of legal document retrieval and the rules for processing answers is documented in writing — when changing vendors or expanding staff, the logic does not need to be reconstructed from code.
- Architecture diagram — component composition, data flows between the application, knowledge base, and model, failure points, and degradation procedure.
- Access policies — roles, visibility separation by cases and clients, logging of document access, procedure for granting and revoking rights.
- Base update regulations — who uploads new versions and how often, how the integrity of the search index is checked, and how rollback is performed.
- Source priority rules — which documents and versions are considered primary, how contradictions between them are resolved.
- Instructions for lawyers — how to formulate a query, how to read references to primary sources, how to mark a questionable answer for review.
- Incident regulations — what to do in case of a leak, erroneous output, or quality drop, and who is responsible for the decision.
- Acceptance checklist — control scenarios with expected results, by which you accept the work.
Such a package allows you to conduct an internal audit, onboard a new developer, and pass a security service review without stopping the product.
Case study: incoming contract review before and after implementation
Legal department of a construction group: six lawyers and about two hundred incoming contracts per month — supply, contracting, lease, services.
Review was done manually against an internal list of conditions: the lawyer read the document in full, compared it with standard wording and the legal database ConsultantPlus, and separately searched for hidden risks — auto-renewal, unilateral termination, penalties without a cap.
What the measurements showed before
The average contract review took 40–60 minutes; for non-standard ones, up to several hours. In a sample audit of already signed documents, a risk was found in roughly every twelfth one: a condition slipped attention due to fatigue or at the end of a large package. This is not negligence — during a day, a lawyer holds dozens of incompatible contexts in mind.
What changed after implementation
The assistant parses the document and returns the structure: parties, subject, terms, payment, liability — and a list of places where conditions deviate from standard ones. Each clause with a reference to a specific fragment and norm. The lawyer checks for 8–12 minutes: time goes to the decision, not to searching.
After three months, review time decreased on average by a factor of four, and the share of missed risks — from 8% to isolated cases that audit no longer confirms. "We stopped being afraid of large packages," says the practice lead.
The decision on each risk remains with the lawyer: the assistant highlights and substantiates, but does not sign.
What to do if a legal AI assistant is already implemented
If the assistant is already working, but lawyers catch it on incorrect norms, the cause is almost always not a "weak model." A regular chatbot answers from general knowledge, while document retrieval is either not connected or returns irrelevant fragments.
The check takes an hour: ten questions whose answers exist only in internal regulations, and verification — does the assistant provide a quote and does it match the source.
| What is configured | When it fits | Limitations |
|---|---|---|
| Regular chatbot without a knowledge base | General legal explanations | Does not know your regulations, confuses versions, does not provide a reference to the norm |
| Document retrieval | Answers on contracts, regulations, practice | Requires regular reindexing and duplicate cleanup; quality depends on chunking |
| Fine-tuning on your data | Terminology, templates, opinion style | Does not add knowledge and does not track changes in norms |
Then work proceeds by logs of real requests, not by impressions: for each question we mark whether the right document was found, whether the fragment is correct, and whether the answer complies with the norm.
Fine-tuning on your own data does not replace retrieval — it changes form, not facts.
Terminology, opinion structure, and style we give to fine-tuning, while current versions and references to ConsultantPlus and internal acts must come from the base: a lawyer trusts an answer that indicates the section, not a smooth formulation.
The metric is simple — the share of correct answers on a benchmark set before and after edits plus the ability to roll back the retrieval configuration in one command. If it is not growing, the issue is in markup and base completeness, not in the model.
Technical concerns: access, accuracy, base updates
Technical concerns around a legal AI assistant concern not the model, but the data perimeter: who sees what, whether an answer can be trusted, how the knowledge base lives, and whether documents leave the perimeter. These are four questions that the customer's engineers ask before the start — we answer each.
How are access and rights separated?
Rights are inherited from groups in the corporate storage — the assistant only narrows them. Retrieval runs only over documents the employee already has access to, so separation between departments works at the retrieval level, not the interface level. Revoking access in the storage is immediately reflected in the output.
How accurate is the answer, and what guarantees it is not invented?
The answer is built only on retrieved fragments and always returns with a reference to a contract clause or article. If there is no support in the base, the assistant reports this directly rather than completing the wording.
Risk detection additionally flags disputed conditions — the lawyer sees a basis for review, not a finished conclusion.
How is the knowledge base updated?
Changes in documents are picked up incrementally, outdated versions are marked as no longer current but are not deleted — history remains for reviewing closed cases. Reindexing runs in the background and does not block the assistant's operation.
Do documents remain confidential?
Deployment is possible in the customer's closed perimeter: data does not leave the perimeter and is not used for fine-tuning third-party models. The request log is stored according to your storage and access rules.
Let's discuss the task and come back with an estimate and work plan
The first conversation is not a presentation, but a task review: how many lawyers are in the department, which document types are reviewed most often, where time is currently lost, and which errors are most expensive.
Based on the results, we come back not with a rough range, but with an estimate of the task and a work plan — what we do in the first stage, what later, and by what criterion you will see that the assistant is working.
- Which documents and in what form are uploaded: contracts, claims, court acts, scans, or marked-up text files.
- How the answer is verified: the assistant must reference a specific clause of the source, otherwise the fragment does not go into the output.
- Where the responsibility boundary lies: the system highlights a risk, the decision remains with the lawyer.
- What risk markup looks like: categories, priorities, and formulations that the department already uses in its work.
- Which roles and accesses are needed: who uploads bases, who sees the request history, how access to confidential materials is restricted.
- A pilot in one department: several weeks on a limited set of documents, with a request log and analysis of erroneous conclusions.
- What we consider the result: the share of answers with a source reference, time for initial document review, the number of missed risky conditions.
Next — a staged work plan with acceptance at each stage: knowledge base assembly, document retrieval setup, risk markup, connection to the lawyer's workstation. Describe the task in general terms — document types, department, access restrictions. We will come back with an estimate, pilot composition, and work plan.