Why an AI chatbot in a bank does not get beyond the pilot
In a demonstration, a bank's AI chatbot responds almost perfectly: scenarios are tailored to examples, wording matches reference, the right document is nearby.
In a live stream, the picture changes in two to three weeks — answer quality drops, and some dialogs end with a call to the contact center.
The bottleneck is almost always the same: the gap between the demonstration set and the real knowledge base. A client writes three different questions in one message, mixes official wording with slang, asks about conditions that were changed last week.
The assistant either confidently answers incorrectly or falls into "didn't understand the question" — and both equally hurt answer quality.
The second problem — the load on the contact center does not decrease but redistributes. Operators get not typical requests but the most complex: those the assistant discarded as unsolvable, and the conversation starts from scratch, without dialog history.
At the same time, assistant pilot metrics look decent — share of answers without an operator and accuracy on a holdout sample — so the solution seems almost ready for launch.
We break down how this is closed: how the assistant connects to the bank's data sources, by what rules it transfers a dialog to a human, and how answer quality is measured not on a demo but on a live stream of requests.
Measurable effect: response speed and line load
The first thing that changes — share of auto-replies, i.e., share of dialogs closed without an operator. Typical scenarios (balance, card blocking, application status, transaction limits) the bot closes itself, and they account for the bulk of requests.
The metric is calculated by actual closure of the dialog inside the bot, not by number of those who reached the button — otherwise the figure is inflated and ceases to mean anything.
Time to first response drops from minutes of waiting in queue to seconds and stops depending on time of day. During peak hours — after a mass mailing, payday, or channel failure — load is distributed across parallel sessions, not across number of available operators.
The queue does not grow linearly: the bot takes the entire wave, and only those who really need a human go to people.
Escalation to an operator is configured by rules, not by a "call a human" button: the transfer includes dialog history, recognized request topic, and everything the client has already confirmed, so the operator does not start the conversation from scratch.
At the same time, the class of failures "client left the chat" goes away — the answer arrives within the waiting window, not after ten minutes of silence.
The cost of processing a request should be measured in operator-minutes: the more dialogs closed in self-service, the less line time goes to repetitive questions, and freed hands handle disputed cases.
We take metrics in Grafana: share of auto-replies, time to first response by percentiles, depth of escalations, and losses at the entrance are visible on one dashboard. Therefore, the effect of improving scenarios is visible within a week, not at quarter-end.
Assistant operating modes: comparison of connection options
Choosing a mode is not about which engine is better, but about the cost of an error in a specific scenario.
Where the answer must match on every request — card blocking, limits, application status — a scripted bot works: it follows an approved regulation, and any of its actions is reproducible during an internal audit.
Where client wording does not repeat, scenarios turn into a tree with hundreds of branches that no one can maintain.
The question "why is the payment not going through" has dozens of causes — here an assistant on a language model over the knowledge base answers more accurately and does not force the client to go through a menu. Comparison of options by applicability and limitations:
| Mode | When it fits | Limitations |
|---|---|---|
| Scripted bot | Regulated operations: blocking, limits, application statuses — the answer must be identical and verifiable | Does not understand paraphrases outside the built-in branches; support grows with the number of scenarios |
| Assistant on a language model over the knowledge base | Consultations, analysis of frequent questions, long client wording; the base is updated more often than scenarios | No verbatim matching of answers — source verification and topic restrictions are needed |
| Hybrid mode | Main option for a bank: the model conducts the dialog and clarifies, critical actions are passed to a scenario with confirmation | Explicit routing between layers and a log of which path the request took are needed |
| Voice assistant | Inbound line, when it is inconvenient for the client to type; confirmation of operations by voice | Recognition errors on noise and names; escalation to an operator is mandatory |
What remains outside automation
All modes share one common limitation: the assistant does not make decisions that the bank is not prepared to explain.
Disputed operations, complaints, and everything that goes beyond the boundaries of approved scenarios and the knowledge base goes to an operator with the full dialog context.
Implementation stages: from dialog audit to handover to operation
Implementation in a bank does not tolerate "turn it on and see": the cost of an error is an incorrect answer about a product, exposed personal data, or a lost request. Work proceeds sequentially, and each stage removes its own class of risks rather than adding features. Below is the route from request history to handover to operation.
-
Dialog audit. We export request history from chats, telephony, and email, label it by topics, and calculate the share of repetitive questions. The output is an intent map, a list of scenarios the bot closes itself, and an explicit list of what cannot be automated.
-
Scenario design. We assemble a dialog map with branches, conditions, and escalation points, and fix wording and prohibited topics. Texts are approved by support and legal: the bot does not improvise where an answer has legal consequences.
-
Knowledge base setup. We bring regulations, tariffs, and standard answers to a unified form, remove contradictions between documents, and version the source. The answer comes with a link to a specific clause, so it can be verified.
-
Setup of transfers to an operator. We define routing rules by topic, language, and line load; the dialog goes to a specialist together with history, and the client does not have to repeat the question.
-
Acceptance checks. First a run of a regression set of dialogs, then a shadow stream and a small share of live traffic.
We monitor the share resolved without an operator, the number of incorrect answers, and the correctness of escalations — until thresholds are reached, the traffic share does not grow.
-
Handover to operation. We hand over documentation, the knowledge base update scheme, monitoring with alerts, and a regulation for rolling back to the previous scenario build, and train the customer's team.
After launch, the cycle does not close: new request topics return to the audit, and scenarios are updated along the same route.
What remains with you: configurations, regulations, documentation
The assistant remains with you not as a "black box" on our side, but as a set of artifacts that your own team works with.
Everything configured during the project is transferred to your repositories and knowledge base — with an explanation of why the parameter is set this way and what will change if it is touched.
- Configured scenarios and instructions for the model — including failure branches, topic restrictions, and wording tested on real dialogs, not in a test set.
- Integration scheme: which systems the assistant reads and where it writes, in what order, what happens when a source is unavailable — with failure points and default behavior.
- Role matrix and access policies: which employees see dialog logs, who edits scenarios, who can export data and on what basis.
- Escalation regulation: by what signs a conversation goes to an operator, at what moment, with what conversation history, and what is considered an abnormal termination.
- Assistant documentation: knowledge map, procedure for updating the answer base, rules for introducing new products, and typical errors during edits.
- Dialog logs and metrics: labeling by topics, share of requests closed without an operator, reasons for failures — in a form suitable for regular review.
Such a set allows you to develop the assistant on your own, and to involve us selectively — for new scenarios or analysis of complex areas. Next we will break down what the handover looks like step by step.
Case: first-line assistant in a contact center
The pilot was launched in a bank contact center, where the first line handled loans, cards, blocking, and disputed operations.
Typical questions were held by operators: during peak hours the queue grew, about a fifth of requests — calls and chat messages — remained unanswered, and time to first response at peak reached several minutes.
What was changed
The assistant was placed on the first line before the operator and connected to the same systems as the workstations: product directory, application statuses, operation checks.
The answer is assembled from the knowledge base and client data, not free generation — if confidence is insufficient or a disputed operation is involved, the dialog with saved history goes to an operator.
The integration was done into the existing Genesys environment: routing and operator workstations were not rebuilt.
Metrics before and after
- Share of autonomous answers: 0% → 61% of requests are closed without an operator.
- Time to first response: from 3-4 minutes at peak to 10-15 seconds, including at night.
- Missed requests: from 18-20% to 2% — the queue stopped hitting the shift limit.
- Escalations: 39% of dialogs go to an operator with ready context, repeat requests on the same question were halved.
The load on the shift shifted toward complex cases: regulatory questions stopped occupying it.
What to do if a scripted chatbot is already running
An existing scripted bot is not dead weight but a labeled corpus: intents, slots, confirmation branches, error handling. It is not rewritten from scratch — it is transferred as is and a language layer is built around it.
We do not touch what is already measured: share of resolutions on first contact, context retention, correctness of transfer to an operator. This is the baseline against which we compare the new branch.
Reuse of scenarios begins with a catalog of intents and "question — expected action" pairs. Trees remain a fast path for operations with rigid logic: card blocking, application status, limits.
The language model is connected where templates are insufficient — free wording, incomplete queries, several intents in one phrase.
We build the gradual transition on routing: a classifier at the entrance decides whether it is a known intent or not. A known one goes to the scenario, an unknown one or one recognized with low confidence goes to the language model with search over the knowledge base. Both branches work in parallel, and the share of hits in each is visible in the logs.
We preserve metrics with a shadow mode: the new branch answers first, but answers are compared with reference ones, and disputed dialogs go for review. Rollout proceeds by segments, not all at once.
We calculate replacement risks in advance — loss of edge scenarios, divergence from regulatory wording, increase in response time.
We do not delete scenarios until the new branch outperforms them on metrics: rollback during this period is done by switching the route.
Common technical doubts before launching an assistant
Before the first launch, a bank's technical services ask a similar set of questions. They concern not the quality of the model's answers, but how the solution fits into the security perimeter, regulations, and current request systems. Below are short answers on each point.
Where are dialogs processed and how is banking secrecy ensured?
Deployment takes place in the bank's infrastructure: client correspondence does not go to external services.
Personal data is masked before being passed to the model, logs are stored depersonalized, and access to the full text is granted only to roles entitled to it under the regulation.
What does the assistant do if the model is not confident?
A confidence threshold is set: below it, the bot does not guess but transfers the dialog to an operator together with the history and a draft answer. The share of such transfers and the percentage of operator edits are the main pilot metrics.
How is the knowledge base updated?
The source is approved bank regulations and products. Each update passes review, is indexed by version, and is recorded with an author; in case of regression, we roll back to the previous knowledge set without restarting the assistant.
How does this integrate into current request systems?
The assistant connects to the bank's channels and ticketing system through standard integration interfaces: the client is identified by session, the answer arrives in the same window, and the dialog history is written to the shared request card.
How is employee access delineated?
Roles are separated: knowledge configuration, dialog viewing, and analytics are different permissions. An employee sees only their segment of requests, and every action is recorded in the audit log.
If the needed question is not in the list — describe your system architecture, and we will analyze it separately.Let's discuss the task: where implementation begins
Starting work does not require a ready bot, a labeled dataset, or a chosen model. Organizational and technical context is needed — from it we assemble a stage plan and estimate. Below is what we request before the first iteration.
- List of scenarios and autonomy boundaries: which requests the bot closes itself, which it transfers to an operator, how it handles a disputed operation.
- Knowledge sources: regulations, tariffs, product terms, a base of typical questions — in working form, with a content owner who confirms relevance.
- Access to systems: a test environment of the core banking system or CRM, a payment gateway sandbox, test accounts. For a prototype, read-only mode on a limited set of fields is sufficient.
- Security requirements: what is prohibited from being transferred to external services, where and how long dialog history is stored, who at the bank approves these rules.
- Target channels: mobile app, internet banking, messengers, voice line — the dialog structure and scope of integrations depend on this.
- Acceptance criteria: share of requests without an operator, answer accuracy on a reference set, response time, share of correct escalations.
- Contacts on both sides: who is responsible for answer content and who opens access.
With this set, we return with an estimate and stage plan: a prototype on a limited scenario, testing on historical requests, a pilot in one channel, then expansion.
If some items are not ready yet — let's discuss the task: usually the gap is closed by regulations and access, not technology.
