Why an AI virtual agent does not close requests
An AI virtual agent is usually launched on a stream of typical questions, and in a demo it closes them confidently.
In real operation, the first line of support receives a mixed stream — some order status, some return conditions, some access and billing — and on it the assistant either goes into endless clarifications or answers with general formulations, after which the person still goes to an operator.
A long first response is almost never about response speed, but about the routing design.
When the handoff threshold to a human is set on the principle "if unsure — hand off to an operator", the assistant hands off almost everything, and the request waits in queue along with the rest.
The share of questions closed on the first line drops within a couple of weeks to the level of ordinary correspondence without automation.
Repeat requests grow from the same root: the dialogue does not rise from history, and a person returning the next day starts explaining again — already irritated.
Peak load completes the picture: on Monday morning or after a mass mailing, the queue grows faster than it is handled, requests get lost between shifts and channels — answered in chat, not in email, and in a messenger not noticed at all.
The cause of failure is usually not the quality of individual answers, but that requests have no unified context and predictable routing. We analyze how this is closed.
Implementation effect: response speed and first-line relief
We count first response time not from the moment "taken into work", but from sending the message to a substantive answer. The virtual agent responds immediately, regardless of line load.
During peak hours the gap manifests most strongly: operators' queue grows, while the agent's response time does not change.
The share of autonomous answers is a metric by request types, not one overall figure.
Order statuses, typical questions about conditions, document replacement are closed without a human; disputed cases go to an operator together with the dialogue history so the client does not repeat everything again.
The fate of repeat requests is more important than the number of closed dialogues. If the answer is incomplete, the client returns with the same question — this is visible from the link of repeat sessions of one user. We track the share resolved on the first attempt and analyze each return as a scenario defect.
The logic does not depend on the channel: one scenario serves web chat and messengers, only the presentation of the answer differs — in Telegram some steps are replaced by buttons.
This gives a correct comparison of automatic answers and operator work on the same requests, not on different streams.
The cost of processing a request decreases due to operator time: one contact takes fewer minutes, and the effect grows with the share of autonomous answers. During peak hours the freed-up resource goes to complex cases where a human is truly needed.
Operating modes: autonomous answer, suggestion, escalation
The mode is chosen not by taste, but by the cost of error in a specific stream of requests.
Where the answer is unambiguous and verifiable against the knowledge base, automation speeds up processing; where the decision changes conditions for the client, the final word remains with a human.
Below are three dialogue management modes and the signs by which a stream is assigned to one of them. The boundaries are set by routing rules, not by the "character" of the AI virtual agent.
| Mode | When suitable | Limitations |
|---|---|---|
| Autonomous answer | Typical requests with an unambiguous answer: request status, tariffs, deadlines, procedure | Does not close disputed cases and anything where conditions need to be changed or guarantees given |
| Collaborative mode | An operator conducts the dialogue, while the representative prepares a draft answer and excerpts from regulations | The draft requires verification: speed grows, but the load is not removed from the operator |
| Escalation to operator | Claims, failures, non-standard requests, changed conditions, irritated tone | Without switching rules, the client repeats the history again |
The key link is the transfer of dialogue context: during escalation the operator receives the message history, the recognized topic, the found documents, and what was already offered to the client.
We explicitly prescribe automation limitations: the confidence threshold below which an answer is not sent, the list of topics closed for autonomous mode, and the signs by which the dialogue is transferred to a human.
The main metric is the share of requests closed without an operator, provided that repeat requests do not grow.
Work stages: request audit, scheme, verification, handover
The analysis begins not with choosing a model, but with the history of requests: we export dialogues for a period from channels and the accounting system, label them by topic, and calculate what share is closed by a template answer, where access to data is required, and where a human is mandatory.
Without this figure, it is impossible either to estimate the scope of work or later prove the effect — and the output is a scenario map on which all further engineering is built.
How the sequence is structured
- Request audit. Complete labeling of the export: topics, repeatability, share of typical questions, points of dialogue interruption, and reasons for going to an operator.
- Dialogue scheme. Map of intents, clarification branches, rules for clarifying questions, conditions and format of transfer to an operator together with the correspondence history.
- Knowledge base. Selection of sources — regulations, articles, past answers; splitting into fragments and reference "question — answer" pairs for subsequent verification.
- Test verification. Run on real requests: share of correct answers, correct escalations, and refusals instead of an invented answer to an unknown question.
- Load testing. Parallel dialogues, behavior when external systems and reference books are unavailable, correct refusal when going beyond agreed topics.
- Acceptance. A fixed set of scenarios and threshold metric values; we analyze disputed answers together with your team before signing.
- Transfer to operation. Accesses, monitoring, dialogue log, knowledge base update regulations, and instructions for operators.
After launch into the production loop, metrics continue to be captured — the log shows which topics are growing, where the share of escalations is growing, and what needs additional training. Describe the task — we will return with an estimate and a plan for these stages.
What remains with you: scenarios, knowledge base, regulations
We hand over not a "black box" with support only at our side, but a set of artifacts that your team works with itself: answer edits, new scenarios, knowledge base expansion, channel connection. Everything is stored in your environment and repository — our access for edits is not required.
- Dialogue scenarios — branches with clarifying questions, conditions for transfer to an operator, handling of refusals and negativity; each replica indicates the expected result and next step.
- Knowledge base — articles and "question — reference answer" pairs, labeled by topics and products; separately marked where there is no answer and the representative must clarify rather than invent.
- Rules and limitations — what the representative is prohibited from: promising deadlines, naming conditions outside agreed ones, consulting on topics outside the knowledge base.
- Access rights — roles of knowledge base editor, analyst, and administrator: who edits answers, who sees correspondence, who exports request statistics.
- Support documentation — request routing scheme, description of settings, instructions for adding a new scenario on your own.
- Support regulations — how often we review the base, by which metrics we monitor answer quality, what we do when the share of refusals grows.
Acceptance proceeds by these same artifacts: a scenario is accepted when it passes the verification set of requests, and a knowledge base article — when the answer does not contradict the regulations.
Case study: service department cut escalations in half
The service department of a household appliance distributor handled about 2,400 requests per week with eleven first-line operators. Two thirds of dialogues were typical scenarios: order status, delivery time, address change, return conditions. They took up most of the shift, while complex cases waited in queue.
What the loop showed before implementation
- Average first response time — 4 minutes, during peak hours — up to 22 minutes.
- 38% of dialogues went to the second line, and half of them — on a formal criterion, without real complexity.
- About 40% of operator time was taken by clarifying questions whose answers were already in the accounting system.
Analysis of the request corpus over three months showed: 64% of them are closed by a finite set of rules if the virtual agent sees the order status, warehouse stock, and delivery history.
What was changed and what the result was
The representative was placed at the entrance to Единое окно обращений: it leads typical scenarios to the end, and transfers the dialogue to an operator only for exceptions — together with the history and already collected data.
- First response — 8 seconds, during peak hours — 40 instead of 22 minutes.
- Share of escalations to the second line — 17% versus 38%.
- 62% of requests are closed without human involvement.
- Four freed-up operators were transferred to returns and claims, where a decision requires a human.
Metrics were captured on the same loop and over comparable periods, so the comparison is correct.
Further work was not on the completeness of answers, but on the share of scenarios completed to the end without switching to an operator — this is exactly what determined the final reduction in line load.
AI virtual agent or button bot: what is the difference
The difference is not in the interface, but in who controls the dialogue. A button bot leads the client along a pre-built tree: any step is a choice from proposed options, and a phrase outside the scenario hits a dead end or a transfer to an operator.
The AI virtual agent parses free formulation itself, determines intent, and holds context — the client does not need to pick the right button.
| Option | When suitable | Limitations |
|---|---|---|
| Button scenario | 5-10 typical operations with a finite number of branches: order status, booking, tariff change | Does not understand free formulations, breaks on typos and mixed questions |
| AI virtual agent | Requests are formulated in one's own words, several questions in one message | Requires regulations for transfer to an operator, a set of test dialogues, and answer quality control |
| Hybrid scenario | Free input at the entrance, buttons — at steps with a legally significant choice | Two dialogue management mechanisms in one scenario are harder to maintain |
The button option is justified where the request stream is homogeneous: payment confirmation, branch selection, return processing. The tree is predictable, easy to test, and does not let the client deviate from the regulations.
The key difference manifests on long requests: the context retains the order number, city, previous solution attempt, and they are not asked again at the third step.
When transferring to an operator, they receive a transcript, the determined intent, and the history — in Zendesk and similar systems, the request arrives already parsed, not empty. Next, we analyze how such scenarios are designed and verified on real dialogues.
Technical concerns: data, quality, limitations
Engineering concerns relate to boundaries, not to "ability to speak": where the AI virtual agent makes mistakes, what it does with an unfamiliar question, how it gets access to internal data. Below are direct answers to four of them.
Where does the system make mistakes and what does it do with an unfamiliar question?
Low confidence is a separate scenario, not a reason to guess: the system clarifies or transfers the dialogue to an operator together with the history.
Knowledge base edits are run against a regression set of dialogues, and a drop in the share of correct answers blocks the release.
How are data and access to internal systems protected?
Requests to accounting systems go through service access read-only for the required fields, without the right to change. Personal data is masked in logs and is not stored in training sets, rights are issued by roles and revoked when an employee changes.
What if the internal system does not respond?
Call errors differ — timeout, authorization failure, empty response — and are handled differently: instead of an invented answer, the client receives a correct wording and waits for an operator.
Retries occur with increasing pause, and a spike in failures raises an alert for the on-duty person.
In how many languages does the dialogue run?
The language is determined by the first message and switches during the conversation while preserving context.
Scenarios, entities, and reference books for each language are maintained separately — otherwise answers turn into a literal translation and break the knowledge base search.
Let's discuss your task and come back with an estimate
So that the time estimate is based on facts, not assumptions, we start the conversation with your context. It is enough to describe the task in free form — typical requests, where operator time is currently lost, which systems are already used.
We will clarify what is missing ourselves and show which nodes are closed by the virtual agent and which are more reasonable to leave to people.
Send what you have at hand:
- an export or selection of real dialogues for one to two weeks — they show both customer vocabulary and typical scenarios;
- a list of channels: website chat, messengers, email, phone requests;
- a list of systems the representative should work with — request accounting, order base, delivery statuses;
- a description of escalation rules: when the conversation goes to an operator and what is transferred to them;
- security service requirements for access, correspondence storage, and data anonymization;
- your constraints on phasing — what needs to be launched first, what can wait.
In response, we will come back with an implementation plan: breakdown by stages, scope of work, time estimate for each step, points where you accept the result.
In the plan we separately record which scenarios go into the first wave, how we measure the share of requests closed without an operator, and by which criteria we consider the launch completed.
Describe the task in a convenient form — we will come back with an estimate and a work plan.
