Developing an AI assistant for internal regulations and policies

AI assistant for regulations: why employees don't find the answer

An employee asks whether a procurement can be approved without a tender — and goes not to the regulation but to a work chat. The answer comes as a retelling from someone who "has already done it that way," without relying on the current version.

A month later the same question resurfaces, but the answer sounds different: oral tradition accumulates discrepancies faster than the policy is updated.

Search across internal policies should resolve this with a single query line, but usually it doesn't work. Documents are scattered across different places: some in a shared folder, some in an email attachment, some in a knowledge base no one visits.

Three files with similar names are found, and it's unclear which one is in force: the currency of document versions is not recorded anywhere.

Another failure is answers without a source link: they cannot be verified. The wording looks plausible, but it's not visible which clause it was assembled from or whether that clause has been revoked.

A lawyer or HR specialist rereads it manually every time — and it's easier to message them privately than to search yourself.

The cost grows silently: repeated questions consume hours of subject-matter specialists, and interpretations diverge between departments — sales read the discount policy one way, finance another.

We'll break down how this is addressed: an AI assistant for regulations answers based on current versions and shows the clause it relies on.

What changes after implementation: response time and error rate

We measure the effect by metrics taken before implementation and a month after: answer search time, share of incorrect interpretations, number of requests to lawyers, and time to onboard a new hire.

Without recorded values, "it became more convenient" cannot be defended before management, so we take control measurements on the same set of questions.

• Answer search time — manually going through regulations, the knowledge base on Confluence, email, and chats takes 15-20 minutes; the assistant answers in 10-30 seconds with a quote from a document clause, with a link to the original source and its version.
• Share of incorrect interpretations — on a control sample of one hundred typical questions, it decreases from 15-20% to single-digit percentages: each answer is tied to a policy clause, not to a retelling "from memory."
• Lawyer workload — typical questions about vacations, procurement, and approvals go to the assistant, lawyers deal only with disputed cases; the flow of repetitive requests drops severalfold.
• Employee onboarding — a newcomer gets an answer on internal procedures from day one, without waiting for a mentor, and the time to become independent is reduced.
• Document currency — a new version of a policy is picked up by reindexing, the outdated version stops appearing in answers, and no discrepancies between versions remain.

Methods of searching internal documents: comparison

The choice of deployment perimeter determines which search strategies are available at all: a hybrid of vectors and keywords with reranking requires an index that must be hosted somewhere and updated regularly.

Hence three workable schemes — a ready-made external service, an assistant inside the company perimeter, and a hybrid where sensitive sections remain local.

In practice, the choice is dictated not by search convenience but by data classification.

While there is nothing closed in the corpus, an external service delivers results faster; as soon as even one sensitive section appears, it is cheaper to design a local index or hybrid right away than to rebuild the scheme after launch.

Implementation stages: corpus audit, output scheme, acceptance

We start not with the model but with the corpus: until sources are organized by owners and versions, the assistant's answer cannot be verified. Below is the sequence by which implementation proceeds, and the artifact the customer receives at the output of each step.

1. Document corpus audit. We collect regulations, orders, instructions, and internal policies from file storage, email, and the knowledge base, and record for each document its owner, current version, and access classification.

   Duplicates and revoked versions are removed immediately — otherwise two contradictory wordings come up for a single question.

2. Source markup. We convert documents to text form together with tables and appendices, and mark up structural blocks: section, clause, scope, validity period.

   Each fragment is linked to the original source, so the employee sees the answer next to the source clause, not a retelling without support.

3. Answer output scheme. We agree where the assistant quotes a clause verbatim, where it paraphrases in its own words, and where it must return "there is no answer in the regulation" and name the process owner.

   Here we also set priority for document conflicts and a filter by access rights — HR policies must not appear in the output for a line manager.

4. Testing on real questions. We run sets of real employee questions and measure the share of correct answers, completeness of references, and behavior on topics outside the corpus.

   We classify errors by type: a gap in sources, an incorrectly selected fragment, extra interpretation instead of a quote.

5. Acceptance by checklist. We fix thresholds: share of correct answers on the control set, mandatory reference to the clause, correct refusal outside the corpus, role-based access separation, response time.

   Items that fail the check are returned for revision of the output scheme, not written off as a "difficult question."

6. Configuration handover and support. We hand over corpus settings, answer templates, and the update procedure: who enters a new version of a document and how. After that, it is enough to add changes to the sources — the output picks them up without rebuilding the assistant.

Describe which documents and roles you have in your perimeter — we will come back with an estimate and a work plan for these steps.

What we hand over: configurations and documentation

A regulations assistant breaks not on answer quality but on maintenance: a source moved, the department structure changed, a new version of a policy was released — and without a clear configuration the system turns into a black box that only the author can modify.

That is why, together with the working perimeter, we hand over artifacts tied to the actual settings.

• Deployment configurations: indexing parameters, rebuild schedule, limits on file types and sizes, separate value sets for test and production perimeters.
• Source map: connected systems and directories, their owners, frequency of change pickup, and a list of what is excluded from processing.
• Access policies: mapping of directory groups to knowledge domains, rights inheritance and behavior when access is revoked, plus checks for leakage between departments.
• Document preparation rules: chunking into fragments, mandatory metadata, and source priority when versions of the same regulation conflict.
• Support instructions: how to connect a source, change an access policy, roll back the index, and what to look at in logs on failure.
• Update procedure: reindexing frequency, responsible roles, list of checks before publishing, and the procedure for returning to the previous version.

After handover, updating sources and rights proceeds without our involvement; we get involved when the architecture changes or a new knowledge domain is added.

Implementation case study: metrics before and after

The project is an assistant for internal regulations for a distributed team of several hundred people: HR policies, procurement, information security, vacations, and compensation.

There was no single entry point, so everyone searched for the answer themselves and asked colleagues again.

Before: answer search as manual work

The regulations were in Confluence, but search was keyword-based: an employee opened five to seven articles, wrote in a work chat, and waited for someone in the know to answer.

The average path to an answer took about half an hour, and two out of five questions went to subject-matter experts — consultations took several hours a week for them. Another third of requests were repeat ones: the answer was found but not saved next to the document.

What was changed and what happened

The corpus was marked up by sections, owners, and currency dates. The assistant answers only with a link to a specific document clause and honestly refuses if there is no source — this removed the main risk: a confident but incorrect answer.

Frequent questions are cached, and the question log goes weekly to regulation owners so that wording is fixed where people stumble.

At a measurement two months after launch, time to answer dropped from half an hour to a minute, the share of requests to experts fell from 40 to 8 percent, and repeat questions — from a third to single-digit percentages.

Regulation owners received a list of places that are read incorrectly, and document edits stopped being guesses.

How does keyword search differ from semantic answering?

Lexical search uses an inverted index: if word forms or an abbreviation match, the fragment is found. Regulations in Confluence and internal knowledge bases are written in established wording, so such search works reliably for exact terms and clause numbers.

It breaks on a conversational question: an employee asks about "time off for overtime," while the policy says "hour compensation" — there are no matches, and the output is empty.

Semantic answering compares the question and document fragments by vector similarity, so paraphrasing does not hinder it. The flip side is that the model fills in wording, and without a link to a specific fragment the answer cannot be verified.

The relevance threshold and the limit on the number of fragments in the context cut off what is not in the regulations.

In practice, we build a hybrid scheme: lexical selection plus vector search and reranking, and each assistant answer is supported by a link to a fragment with the clause number.

We measure output completeness on a set of real employee questions — the share of cases where the needed clause made it into the top five fragments is the quality metric.

Technical doubts worth resolving before the start

An assistant for internal regulations more often stumbles not on the model but on the state of the knowledge base. Below is what we usually resolve before the start: without answers to these questions, neither the legal department nor the security service will accept the system.

Documents contradict each other — whose answer is more correct?

The assistant does not average the wording: it returns both fragments, marks the discrepancy, and indicates which document is higher-level — a policy versus a department instruction. Such an answer immediately becomes a task for reconciliation, not a quietly incorrect reference.

How to separate access between departments?

Rights are inherited from corporate groups: the assistant returns only those fragments to which the employee has access in the source storage.

HR and commercial documents are unavailable to other departments even with a direct question — filtering happens at the search stage, before the answer is formed.

How accurate is the answer, and what to do if there is none?

Each answer includes a link to the clause and the version date — the employee verifies the source in one click.

If confidence is below the threshold, the system honestly reports that there is no direct answer in the base and suggests the topic owner instead of a generated wording.

Let's discuss the task: what is needed from your side

To come back with a realistic estimate rather than an eyeballed range, we need a minimum of inputs about the document corpus, the perimeter, and access rules. Below is what speeds up the review and allows us to immediately outline the scope of the pilot and the first result.

• List of sources. Regulations, policies, instructions, orders — indicating where they are stored and in what form: text files, tables, scans, internal wiki pages.
• Pilot boundaries. One department or one domain — HR, security, procurement. To start, 10–30 documents are enough; this is sufficient to test answer quality on real questions.
• Accesses and roles. How rights are granted, whether corporate authentication exists, which documents are available to a limited circle of employees.
• Perimeter requirements. Cloud or internal network, what is prohibited from being exported outside the perimeter, whether a request log and department separation are needed.
• Content owner. Who confirms the currency of versions, who is responsible for updating the corpus and marking document versions.
• Acceptance criteria. Share of answers with a correct clause reference, completeness of coverage of frequent questions, assistant behavior when there is no answer in the documents.
• Testing participants. 5–10 employees from different roles plus one person responsible for accepting the result.

The first result — the assistant answers based on your corpus with a link to a specific clause, and you see where documents contradict each other. Describe the task within this scope — we will come back with an estimate, stages, and a work plan.