Why the AI support system doesn't find the answer in a minute
The operator receives a question in chat and opens four places at once: an instruction in a shared folder, a regulation in email, correspondence with an engineer, and notes in a personal file.
Fragmented documentation lives in different formats and versions, so knowledge base search returns either an empty result or a paragraph from a canceled regulation.
While the operator verifies dates and clarifies details with colleagues, the customer response time grows from seconds to tens of minutes. The customer doesn't wait and writes again — repeat requests go to the next operator, who goes through the same cycle anew.
The AI support system at this moment answers formally: it returns what was found, not what is needed.
The root is not in reading speed, but in the fact that search goes by words, not by the meaning of the question.
The customer's wording "payment doesn't go through" literally doesn't match the heading "payment processing error", and the index is built from a copy of the base that hasn't been updated since last quarter.
The operator's assistant breaks at three joints: the document is not split into semantic fragments, access rights are not taken into account during retrieval, and source freshness is not checked.
Each joint produces its own class of failure — from an empty answer to a confidently incorrect one, and the operator goes to search manually again.
Next we'll break down how this is closed: how the index is built, how it is updated together with documents, and how the answer comes in seconds, not minutes.
What changes after implementation: search time and answer accuracy
Before implementation, a first-line operator searched for an answer for minutes: several tabs, flipping through regulations, clarifications from colleagues. After, the request goes to knowledge base search, and the assistant returns two or three fragments with source indication. We measure the shift by the same metrics before and after.
Answer search time and the share of operator's independent resolutions
The operator's answer speed increases several times over, but more important is another thing: they stop escalating where the answer is already in the base. The share of independent resolutions grows because the assistant surfaces the needed fragment, not the whole article.
We check accuracy not by eye: once a week we run a control set of several hundred real requests and calculate in what share of cases the fragment actually closes the question.
The source is the working knowledge base: Confluence, an internal portal, or a file storage. We index it entirely, together with attachments, otherwise answers lying in attached documents remain invisible.
Load on the second line and repeat requests
The second line is offloaded not only due to fast answers, but also because we separately clean the base.
The class of errors "answer from an outdated article" goes away: each fragment has a freshness date, and the assistant lowers in retrieval materials not confirmed for longer than a specified period.
Repeat requests fall because the operator gives the customer not a retelling, but a link to the source — the next contact on the same topic no longer requires analysis from scratch.
We calculate the load on the second line in requests per operator per day — the metric is visible before implementation as well.
Options for building a support system: comparison of modes
The choice of scheme comes down to three questions: where knowledge is stored, how the system finds the answer, and who is responsible for the freshness of the text.
In practice, the modes do not compete but complement each other — they start with the cheapest to launch and add the rest as the flow of requests grows.
| Mode | When it fits | Limitations |
|---|---|---|
| Search over a ready knowledge base | The base is marked up: articles, tags, product versions, sections by products | Answer quality equals base quality: an outdated article yields a confidently incorrect answer |
| Hybrid search with reranking | Operator formulations do not match article text, many synonyms and typos | Need to configure the relevance threshold, extra fragments confuse the operator |
| Suggestions by request templates | Flow of same-type tickets: repeat failures, typical setup questions | The template becomes outdated faster than the base and does not cover rare cases |
When each mode fits
Base search is taken when the knowledge base structure is already in order: the cleaner the article markup, the higher the retrieval accuracy on the very first request.
Hybrid search is enabled where the operator and customer call the same thing by different words — the mode pulls in semantic matches and reranks them by weight.
Applicability boundaries and risks at the start
Answer templates give maximum on a narrow set of scenarios but require revision with every product change. The main limitations of implementation at the start: incomplete base, no content owner, and no metric by which the share of accepted suggestions is measured.
Without a base owner and regular revision, any scheme degrades in a matter of weeks — that's why we break down how this is closed with a regulation and markup.
Stages of work: from knowledge base audit to handover of regulations
The order of stages sets one thing: the assistant answers the operator as well as the knowledge base under it is organized. Therefore the first steps are about sources, and only then about setting up search and the assistant itself.
Audit of sources and cleanup of duplicates
- Knowledge base audit: we collect all sources — instructions, regulations, tables, support correspondence history — and record what contradicts each other and where duplicates lie. The output is a source map with owners and the share of outdated documents.
- Source cleanup: we consolidate versions to the reference one, remove duplicates. The base volume usually shrinks by a third without losing content, and hit accuracy grows even before search setup.
- Search scheme: we split documents into semantic fragments, define synonyms and professional slang, so that "doesn't start" and "error on startup" lead to the same regulation.
- Assistant setup: we define the confidence threshold at which it answers, and at which it honestly says "not in the base" and offers escalation. Each answer comes with a link to the document — the operator checks the quote in seconds.
Verification on control questions
- Verification on control questions: we take 50–80 real requests, rephrase half of them, and add trap questions without an answer in the base. We calculate the share of correct hits and separately — the share of cases where the assistant did not invent an answer.
- Operator training: we show how to read the quote, how to formulate a question, when not to trust the answer — otherwise the assistant remains a toy by the second day.
- Handover: we give the base update regulations, search scheme, and a set of control questions, so that after edits accuracy does not slip and is checked by the same procedure.
Configurations, integration schemes, and regulations for the assistant's operation
The assistant lives not in a demo but in the operator's daily shift: the knowledge base changes, the flow of tickets grows, and sooner or later someone must understand why a wrong answer came to a specific question.
Therefore the system is handed over not as a "black box" but together with a configuration description, relationship schemes, and regulations — support and development further do not depend on our team.
Schemes, configurations, and integration descriptions
- Configuration description: which knowledge sources are connected, how search over them is configured, which answers the assistant gives itself, and which it passes to the operator with a low-confidence mark.
- Integration schemes with the ticketing system: which events and in what form go outside (request creation, status change, repeat request), where the "answer — ticket" link is stored.
- Scheme of linkage with the knowledge base: how documents are added, how changed ones are updated, what happens when material becomes outdated or is deleted.
- Access matrix: roles of operator, shift supervisor, base administrator, and analyst — who reads, who edits the configuration, who sees the dialogue log.
- Environment map and order of releasing settings: how to transfer changes from the test environment to production without stopping the assistant.
- Assistant documentation: customer's glossary, list of typical failures and first actions for each of them.
Regulations for updates and review of disputed answers
Regulations fix not "what is correct" but who acts and within what timeframe when answer quality drops.
- Base update: frequency of reconciliation with material owners, person responsible for upload, checking control questions after each update.
- Review of disputed answers: sample of requests with low rating, decision for each — fix the source, search rule, or instruction for the model.
- Change log: what and when changed in the configuration, on whose initiative, and with what result in the share of correct answers.
Case study: how answer search time in the knowledge base was reduced
What it was like before launch
The knowledge base grew to several thousand articles, but searching it had to be done manually: exact-word search found material only if the wording matched.
Average answer search time — four and a half minutes, and a third of requests went to the second level not because of the complexity of the regulation, but because the operator simply did not find the needed instruction.
A separate problem was outdated materials. Articles had no owner: everyone wrote, no one updated, and some of them were quoted out of habit, although the work order had already changed.
Access rights were also shared — the first-line operator saw internal regulations of senior groups, which confused priorities and lengthened search.
What was changed in the knowledge base and access rights
The article corpus was marked up and indexed so that the answer is found by the meaning of the request, not by word match: "card payment doesn't go through" surfaces material about payment authorization error.
Each article was assigned an owner and revision deadline — overdue and draft versions do not appear in retrieval, and the previous revision is saved for rollback.
Rights were divided by competency groups: the first line sees client scenarios, internal regulations open at the escalation stage, and each opening is written to the log.
Metrics before and after: answer search time decreased from 4.5 minutes to 40 seconds, the share of requests closed without escalation grew from 62% to 88%, the share of transitions to outdated articles fell from 19% to 3%.
Article updating stopped being manual work "from memory" — revision goes by schedule and by owners.
What to do if the knowledge base is already deployed but not used?
An unused knowledge base is almost always not about article quality but about the fact that the assistant cannot reach them. The answer is not separated into a distinct fragment, hidden by access rights, or not linked to the request wording. We check three layers — article structure, access rights, and how operators ask questions.
Article structure. We look at whether the material is split into blocks the size of one answer and whether each is titled as it will be asked.
A sprawling article of several screens is poorly suited for document search: it returns it entirely, and the assistant has nothing to quote.
We separately check terminology: if articles say "subscription service" and operators write "SS", without a synonym dictionary there will be no match.
Access rights. Document-level filtering can cut answers unnoticed: the article exists, but it is unavailable to the operator, and the assistant correctly reports that it found nothing.
We check the role map and rights inheritance, then look at retrieval through the eyes of an operator from each department.
Request wording. Operators write telegraphically, with typos and internal slang — "payment doesn't go through", "ticket hangs". Query normalization, synonym expansion, and accounting for frequent formulations from the request log work.
The log is the most valuable artifact: unanswered requests turn into a queue for base additions, and the first edits close the main share of misses.
We check the result on a set of real questions with known answers: we calculate the share of hits in the needed fragment and time to answer. After edits we repeat the measurement — it shows how many requests were closed. Describe your base and operator scenarios — we'll come back with an assessment and plan.
Technical doubts: access, answer quality, updates
The assistant answers not by its own "truth" but by your knowledge base, so engineering questions come down to three: access boundaries, behavior in contradictions, and answer quality after launch.
How is access delineated between products and departments?
Rights are inherited from the employee's account: the assistant sees only those sections to which access already exists in your system. A support engineer will not get documentation of an adjacent product, a contractor — materials of a closed environment.
Boundaries are set on the source side, not by a filter inside the answer, so they cannot be bypassed by rephrasing the question.
What happens if two articles contradict each other?
It does not silently choose the "correct" article: it gives both fragments with source and modification date, and if there is a discrepancy on a key parameter, marks the answer as requiring verification and passes the request to a human. Such cases go into the report — usually these are duplicates after section reorganization.
Answer quality is measured on a labeled set of real requests: the share of answers accepted by the operator without edits and the share that went to escalation. Metrics are taken before launch and repeated after each major base update — so documentation growth does not drag quality down.Support after launch is a regulation, not a one-time handover: we monitor the share of failures, review questions without a found source, update search rules. Your team adds articles in the usual editor, reindexing goes by schedule.
Let's discuss the task: we'll review the knowledge base and come back with a plan
To make the conversation substantive, there is no need to prepare presentations and a technical specification — what already exists inside is enough.
We will review the materials together, record preparation for the start, and come back with an implementation plan: composition of stages, order of work, and assessment.
The most useful thing to bring is a list of sources and an understanding of where operators' questions come from. This is enough to estimate the volume of the knowledge base, integration points, and a realistic timeline.
What is worth preparing for the discussion:
- list of knowledge sources: regulations, instructions, scripts, archive of closed requests, internal reference books;
- integration points — where the operator works and from where the assistant receives request context;
- access rules: which base sections can be used, which remain closed;
- 20–30 real questions with reference answers — the verification set is assembled from them;
- persons responsible for base updating and the update procedure when products change;
- limitations: topics on which the assistant must pass the dialogue to a human;
- acceptance criteria — share of correct answers and search time versus current indicators.
If some points are not ready yet, this does not block the conversation: we will review at the meeting and supplement afterwards. Describe the task — we'll come back with an assessment and plan.
