AI system for identifying knowledge gaps in an organization

Knowledge Gap Analysis: Where Answer Search Breaks Down

The knowledge base is formally filled: policies, instructions, incident analyses, support answers.

But an employee still cannot find the answer in a couple of minutes and goes to a colleague in chat who “definitely knows.” This is the knowledge gap — not the absence of documents, but the gap between what is stored in the base and what is searched for in it.

Answer search works by word matching, while the question is phrased differently from the instruction. At the same time, outdated instructions often weigh more than fresh ones: the document has an old date, but more transitions and internal links — and it comes out first. The employee receives a formally suitable but incorrect answer and stops trusting the base.

Worst of all are questions without an answer. The search silently returns emptiness, the person goes to direct messages, and the very fact of the miss is not recorded anywhere.

The gap does not make it into statistics — meaning it has neither a topic nor a document owner, and no one is there to close it.

Hence the task of analysis: not “rewrite the entire base,” but build a topic coverage map and find which questions remain unanswered. Then separate real gaps from outdated instructions, duplicates, and unsuccessful wording. We explain how this is closed.

Effect: Answer Search Speed and Share of Resolved Requests

We measure the effect not by the number of requests to the system, but by three metrics that we take before launch and several weeks after: time to first correct answer, share of repeat requests on the same topic, and completeness of knowledge base sections.

Measurement on historical requests provides a baseline, so the comparison runs on the same stream, not on feelings.

Time to first answer drops because the answer is assembled from fragments of policies with source attribution, rather than searched manually through folders.

The operator immediately receives a wording with a link to the document and its freshness date — it is faster to verify than to compose anew. On typical questions, the gap between “before” and “after” is measured in tens of times.

The share of repeat requests decreases when the review of each dialogue ends with an edit to the base: if no answer was found or it turned out to be incomplete, an article is created or supplemented in the same section. Such a cycle closes the root cause, not an individual request.

Outdated policies stop surfacing in answers: documents gain an owner and a review deadline, and the system flags fragments that have not been confirmed for longer than a specified period.

Base completeness is measured by the share of typical topics that have coverage — you can see which sections are covered and which rely on verbal agreements.

For teams where documentation lives in Confluence, the index is built on top of existing spaces and pages: there is no need to rewrite the base — it is enough to assign owners and review deadlines.

Analysis Formats: Manual Review, Logs, and Automation

The approach to identifying knowledge deficiency is chosen not by the “sophistication” of the method, but by how the sources are structured.

Expert knowledge settles in the heads of its holders, some in correspondence and internal documents, and some manifests only in the flow of requests: there you can see what people search for and do not find.

If there are two or more sources, no single method alone gives the full picture — it covers only its own slice.

Practice shows: manual review yields a high-quality taxonomy faster, while the flow of requests yields scale and frequency.

Therefore, the scheme is most often assembled as a hybrid, and the choice is determined by the volume of data and how strict the requirements are for the evidence behind conclusions.

The selection criterion is simple: a gap is considered confirmed if it is visible in at least two independent sources — for example, it surfaces in requests and matches the results of manual review.

This filters out random spikes of questions and one-off complaints, and what actually affects work goes into the training plan.

How Implementation Proceeds: Stages of Engineering Work

Implementation proceeds in short iterations: first we establish what counts as knowledge in the organization, then we build a model of subject areas, and only after that do we enable automatic knowledge gap analysis.

Without a knowledge map, the system finds “holes” where there are none, and trust in it falls at the very first demonstration.

1. Source audit. We compile a registry: knowledge base, policies and instructions, support request history, working group correspondence. For each source, we note the owner, update frequency, access via software interface, and text quality.
2. Knowledge map. We divide the subject area into topics and subtopics, and assign a section owner for each. The map sets the coordinate system: a gap is not “no document,” but the absence of an answer in a specific branch.
3. Sample labeling. Experts label several hundred real questions: where an answer exists and is current, where it is outdated, where it is missing. Labeling calibrates thresholds and reveals terminology discrepancies that confuse the model.
4. Holdout sample. We do not show part of the labeled questions to the model during tuning — we use it to measure precision and recall. If false positives are more than a third, thresholds are rebuilt rather than wording being edited.
5. Iterations based on owner feedback. Answers like “this is not a gap” and “here are two different gaps” return to labeling and change the map. Each cycle reduces the share of false positives.
6. Handover to section owners. We provide the update policy, a prioritized gap queue, and training for the responsible people. After that, the system lives without us: a new gap joins the queue of whoever is responsible for the topic.

What We Hand Over: Configurations, Schemes, and Policies

After project delivery, the client’s team retains not only a working system but also a complete set of artifacts by which it can be developed, handed over to new engineers, and shown to reviewers.

Each document is tied to a real system node — this is not a “general outline” description, but something that can be opened and applied in work.

• Entity schema: connections between employees, competencies, knowledge sources, and assessment results — with fields, relationship types, and change rules.
• Access policies: roles, permission matrix for reading and exporting, procedure for personal data processing.
• Update policy: how often the knowledge index is rebuilt, who adds sources, and what signs indicate reconfiguration is needed.
• System documentation: architecture diagram, description of loops, deployment and backup recovery instructions.
• Model and control metrics catalog: what is responsible for extraction, what for matching, which quality indicators are considered the norm.
• Set of validation scenarios: tests for completeness and accuracy of gap detection, so that improvements do not break already working rules.
• Instructions for managers: how to read the gap report, how to dispute a false positive, who receives the signal.

At handover, engineers walk through each item together with the client’s team — where it is located, how it is updated, what to do in case of failure.

Without this set, any improvement starts with restoring the context from scratch; with it, the system can be developed in-house or a targeted task can be given to us.

Case Study: How We Reduced the Share of Unanswered Questions

An illustrative project — the internal knowledge base of a manufacturing company with distributed support. Documentation lived in Confluence, some answers existed only in correspondence and closed requests, and policies diverged by version.

The share of unanswered questions held at about a third of requests: an employee either could not find what was needed or received an outdated instruction.

Average answer search time was 15–20 minutes, and contentious topics were re-reviewed by senior engineers, essentially manually closing knowledge gaps.

What We Did

We assembled a unified index across all sources and tied each answer to its original source fragment: if there is no confirmation, the system says so directly rather than constructing a wording.

Then we enabled a feedback loop — “helped / did not help” marks and topics where no answer was found go into the improvement queue for section owners. The index is rebuilt on a schedule, so outdated document versions are displaced rather than continuing to live in search results.

What Changed

Over three months, the share of unanswered questions fell from 33% to 9%, and average answer search time dropped from 18 minutes to 1–2 minutes.

The dynamics of requests shifted: repeat questions on the same topic almost disappeared, escalations to experts fell by roughly half, and weak spots in policies became visible in practice — those were exactly where answers were most often not found.

How Gap Detection Differs from Knowledge Base Search

Document search solves a task with a known answer: the policy, instruction, or agreement is already described, and the system only has to find the fragment and quote it with source attribution.

Engineering-wise, this is a search layer: documents are split into fragments, converted to vector representations, and ranked by relevance, while access rights filter out what a role is not entitled to. There is one metric here — retrieval precision for a question whose answer exists in the base.

Gap detection is the inverse task. There is no answer in the base, and the very fact of its absence is not recorded anywhere: neither in the table of contents nor in the report on uploaded files.

The system searches not for a document, but for a topic: it collects wordings for which search returned emptiness or low confidence, groups them, and shows which sections are missing.

The second class of gaps is obsolescence and discrepancies. An answer is formally found, but the document refers to a canceled procedure or contradicts a neighboring section.

The signal here is not an empty result, but a conflict: several versions of the same policy, different answers to one question, absence of an owner and a review deadline.

The distinction rests on different input data. Search works with the content of the base, detection works with employee behavior: query history, repeats, refusals, questions that were closed manually. A breakdown by department separates a local gap from a systemic one.

Therefore, an expanded knowledge base grows not by the number of uploaded files, but by reducing the share of unanswered questions. Both tasks live in one loop: search closes the current question, detection turns unanswered ones into a content plan.

Concerns About Labeling Quality and Data Privacy

Labeling and privacy are two issues that surface before the start and largely determine whether the project succeeds.

We address them at the input: we establish the labeling scheme, rules for disputed cases, and the quality metric, and separately — where the data lives and who has access to it. Without this, neither the model nor the knowledge gap report deserves trust.

Experts disagree in their assessments — what then?

We treat disagreement as a metric: if agreement on a class is below the threshold, the class is unsuitable for automation in its current form. We refine the criteria, split merged categories, and put disputed cases into a “requires a human” list. We fix the agreement threshold before labeling starts.

Where is data stored and who has access to it?

Labeling and training take place in the client’s environment: raw materials do not leave the perimeter, and only aggregated gap indicators go outside. Access is role-based, accesses to the dataset are logged, exports are limited and logged. While agreement is below the threshold, the system is not rolled out to production.

Let’s Discuss the Task: What We’ll Need from You

So that discussion of the task does not turn into gathering requirements from scratch, the inputs should be prepared in advance. Below is what actually affects the system design and timeline: without this data, any estimate will be guesswork.

• Knowledge sources: document repositories, internal bases, ticketing system, work chats, meeting records. For each, we need a person responsible for access and the method of provision — a read-only account or a one-time export.
• Section owners — experts who confirm what counts as a correct answer and label the reference set. Usually 2–4 people for key topics; their time is the scarcest resource in the project.
• Examples of real employee questions: 15–30 wordings for which there is currently no answer or the answers contradict each other.
• Definition of a gap: the document is missing, outdated, contradicts another, or exists only in correspondence. Different definitions mean different acceptance checks.
• Data constraints: what cannot be sent for external processing, where extracted fragments and logs are stored, and what retention periods are mandatory.
• Where to return the result: knowledge base, internal portal, or messenger — and who in the company closes the detected gaps.
• Success metric: share of questions with a confirmed answer and time from gap appearance to its closure.

With these inputs, we will return with an estimate and work plan: stages, quality checks, and acceptance points. Describe the task — we will review what you already have and what we will assemble together.