Developing AI for compliance: where manual audit breaks down
The regulatory framework is updated faster than the compliance service can read it. An amendment comes into force, but the audit checklist does not: the auditor follows the previous revision because the new one has not yet been broken down into clauses.
The more jurisdictions and internal regulations, the wider the gap, and people cannot close it — headcount grows linearly, while the volume of requirements grows faster.
The cost of the gap is visible in two places. First — missed violations: the check is formally passed, the non-compliance remains in the document and surfaces during an external audit.
Second — different decisions on identical cases: two specialists read the same rule and interpret it differently because there is no single source of truth, and previous interpretations are not recorded anywhere.
A system based on semantic search removes both distortions. Regulations are split into fragments tied to a clause number, each fragment receives a vector representation and metadata: revision, effective date, scope.
When a document is updated, only the changed section is re-indexed — the answer always references a specific clause of the current revision, and the auditor sees the basis of the decision, not a paraphrase.
A separate loop is regulatory change control: sources are crawled on a schedule, the new revision is compared with the previous one, changes are queued for re-indexing, and outdated fragments are marked invalid and remain in history — transactions from prior periods are still checked against the old revisions. Next we break down how this is closed.
Effects of automation: reaction speed and completeness of checks
The five effects visible already in the first months of the system's operation should be measured on the client's historical data: how much time was spent analyzing one requirement change, what share of documents actually reached an expert, and how often the same question was resolved differently.
- Requirement change reaction window — from weeks to hours: the system monitors external sources such as ConsultantPlus and internal regulations, flags affected documents, and sends them for review.
- Completeness of checks stops depending on time available: the entire document set goes into initial review, not a sample that was glanced through. Missed clauses appear in a separate report.
- Reproducibility of decisions: the same input yields the same output with a reference to a specific requirement clause. The dispute 'why this way' is resolved by a quote from the primary source, not by the reviewer's memory.
- Reduction of manual work: comparison of revisions and analysis of same-type blocks are taken over by automation; experts are left with edge cases and disputed interpretations where qualification is needed.
- Eliminated class of failures: checking against an outdated revision, missed regulatory update, decision without justification — these failures become impossible at the process level, not through employee vigilance.
We measure each item against the same data as before implementation — otherwise it is an impression, not an effect. If a metric does not move, we investigate the cause and change the selection rules.
Operating modes of the verification system: comparison of approaches
The verification mode determines what exactly goes into the verdict and how quickly the system will start making mistakes on a non-standard contract. In our work we use three options — they do not replace each other but cover different classes of defects.
| Option | When it fits | Limitations |
|---|---|---|
| Rules and keywords | The regulation is formalized: deadlines, details, mandatory clauses, prohibited wording | Breaks on paraphrases; each new requirement is a separate rule and a rerun of the entire check set |
| Knowledge base search with source grounding | Semantic verification is needed: scope of liability, representations, payment procedure, termination conditions | Works exactly as well as the collection of internal regulations and precedents is complete and annotated |
| Hybrid approach | The contract is mixed: hard fields and semantic blocks in one document | More expensive to set up: two verification loops and a common layer that reconciles their verdicts without contradictions |
Why the hybrid more often remains in the production loop
Rules cut off formal defects on the first pass, while knowledge base search parses semantic blocks — and each conclusion is supported by a reference to a regulation clause or precedent. Contract review stops depending on whether a lawyer remembers a specific requirement.
The limitation of the approach is common to all three modes: the quality of the knowledge base.
If regulations are not collected, not split into fragments, and not linked to sources, the system produces confident wording without grounding — and such a verdict is more dangerous than its absence.
Implementation stages: from parsing regulations to handover to operations
The work order is structured so that each step relies on a verifiable artifact from the previous one — a rule, an annotated example, or a metric. Therefore at any stage it is visible where quality is growing and where it hits the completeness of the source documents.
-
Requirements analysis. We collect regulations, internal policies, and past orders, and break them down into verifiable statements. Separately, we mark rules that can be formalized as a rule and evaluative ones where the decision remains with a human.
-
Annotation of the document corpus. We form reference pairs 'fragment — auditor verdict', including edge cases and differences in interpretation. The set becomes a measure of decision quality and a basis for future checks.
-
Processing pipeline. We determine how the document is divided into semantic blocks while preserving clause numbering, how the answer is retrieved, and how each conclusion is accompanied by a reference to the rule and document version.
-
Check configuration. We configure rules, confidence thresholds, and the manual review queue for the department's workload: routine hits are closed automatically, disputed ones go to the auditor with a ready summary.
-
Acceptance on historical reviews. We run the system on completed cases and compare against auditors' decisions: completeness of findings, false positive rate, time per review.
-
Handover and training. We provide the corpus update regulation, instructions for auditors, and the error analysis procedure; we teach the client's team to add new rules and regulations independently.
What we hand over: configurations, regulations, documentation
We do not hand over the system as a black box: together with the solution we transfer a documentation set that your team can use to operate and develop it without us. The list is fixed in the handover act — each item has an owner and an entry point in the interface.
- Check configurations — rules for reconciling documents and operations with regulator requirements: rule source, trigger priority, change procedure.
- Data schemas — which sources are connected, which fields come from where, how entities are related, and where the boundary of responsibility between systems lies.
- Data access rules — roles and groups, differentiation by knowledge base sections, procedure for granting and revoking rights, handling of sensitive fragments.
- Knowledge base update regulation — who uploads new revisions, how the actuality date is recorded, what happens to old document versions.
- Decision logging — for each answer, the fragments used, rule version, and time are saved so the reviewer can reconstruct the reasoning.
- Team instructions — scenarios for routine operations: adding a source, analyzing a disputed hit, rolling back a rule, escalating a failure for review.
- System map and maintenance procedure — component diagram, control points, responsible parties, and post-launch work plan.
The set is maintained as a living document: edits to configurations and access rules go through the same approval procedure as system changes, so descriptions do not diverge from the actual state.
Case study: review of contracts and internal regulations
What the work looked like before
The legal department was approving a batch of 400 supply contracts. Three lawyers read the texts in parallel and checked each against internal regulations and standard terms: the batch took about three weeks, and discrepancies between reviewers reached 18% of comments.
One specialist missed a clause, another found it, the parties argued again about interpretation, some comments were duplicated, some were lost in correspondence.
What was changed
We assembled a knowledge base from internal regulations, standard wording, and analyzed precedents.
Contract Register became the source referenced by every output: the system answers not with a free paraphrase but with a regulation fragment with a clause number — the reviewer sees the basis of the comment, not a guess.
On top, we added change monitoring: after a new revision of a regulation, the search index is rebuilt, outdated fragments are marked as no longer in force, and a reference to a revoked clause does not appear in the output.
What happened to the metrics
The same batch of 400 contracts passes review in two days instead of three weeks. The share of discrepancies between reviewers dropped from 18% to 4%: disputed clauses are marked identically, and analysis goes only to genuinely ambiguous conditions.
Coverage is 100% of clauses instead of a manual sample, while the lawyer spends time on 10–15 disputed points rather than reading the entire document.
How does knowledge base search differ from keyword rules?
Keyword rules are predictable: a hit on a known wording is reproduced exactly, it is easy to explain to a reviewer and defend in an audit.
Everything breaks on language variability: 'we are obliged to store and process data' and 'information is processed with the subject's consent' — the same requirement, but no rule will catch it, while negations and homonymy produce false positives.
Knowledge base search is structured differently: the document is split into fragments, each gets a vector representation, and the query is matched by meaning, not by letters. This is how paraphrases and synonyms from GDPR and internal regulations are pulled in.
The price is the risk of a plausible but unconfirmed answer, so in audit a verbatim quote and a reference to the fragment are mandatory.
| Mechanism | What it finds | Where it breaks |
|---|---|---|
| Keyword rules | Known wordings, exact match | Paraphrases, negations, phrase breaks |
| Semantic search over fragments | Semantic matches, synonyms | Requires a confidence threshold and quote |
| Hybrid: rules as filter + search | Paraphrases within the relevant set | Harder to tune thresholds |
In practice the mechanisms are combined: rules narrow the search area, the semantic layer pulls in paraphrases, and every answer must contain a verbatim fragment and its place in the document.
If the quote is not confirmed, the answer is not shown, and the fragment goes to manual review — precision is more important than recall.
Technical doubts we resolve before start
Doubts about such systems are predictable, and in developing AI for compliance we resolve them before start, not during implementation. Below are the things most often checked at the evaluation stage.
Can the system operate in an isolated environment?
Yes, this is a standard mode, not a project-specific customization. The model, search index, and embedding storage remain within the perimeter; only telemetry goes outside — at your choice.
We differentiate access by roles and log requests and outputs, so data security is confirmed by a journal, not a promise.
Where is the guarantee that the system does not invent rules?
Answer quality control is built on extracted fragments: each clause is accompanied by a source reference with the revision date. If similarity is below the threshold, the system returns 'no data' rather than a plausible guess. We measure the share of source hits and correct refusals on your corpus.
Regulations change — what about the index?
Re-indexing is triggered on a schedule or by event: the new revision is added to the base, and the previous one is preserved as a version. The answer is reproduced as of a date — this is exactly what an audit checks.
How much data is needed for evaluation?
For a pilot, one thematic block and 30–50 real questions with reference answers are enough. This is sufficient to measure search accuracy and understand whether additional model tuning is needed.
What about responsibility after handover?
We hand over configurations, the indexing scheme, the update regulation, and the check set with documentation. After that — maintenance or support of your team under an agreed regulation.
Let's discuss the task: what is needed for evaluation and a work plan
The more specific the source data, the fewer assumptions in the project estimate and the more accurate the work plan. Below is the minimum set that removes 80% of uncertainty and allows timelines to be calculated by real scope rather than a range.
If some items are missing — not a problem, we will record them as assumptions and verify them at the first stage.
- Document examples — regulations, policies, instructions, and completed checklists in the form they actually exist: scans, tables, letters, exports from internal systems.
- List of requirements — internal rules and external regulations that must be complied with, marked as to which are hard and which allow interpretation.
- Current review procedure — who checks documents at which stage, what they record, how much time it takes, and where non-compliances are most often lost.
- Perimeter constraints — where data may be placed, whether external services can be accessed, and what requirements apply to access logging and copy storage.
- Reviewer roles — who is responsible for interpreting requirements, who approves the conclusion, who analyzes disputed cases, and how conflict between the reviewer and the process owner is resolved.
- Review history — recurring comments over several periods: they show which areas account for the bulk of manual work.
- Source systems — where documents are stored, how often they are updated, and who owns each repository.
- Readiness criterion — what is considered a successful result, to whom and in what form the report is needed, and how quickly it must update after a requirement change.
Based on this set, we come back with an estimate and plan: stages, scope of work, control points, and what will be needed from your side at each of them.
