AI chatbot development: where the dialogue with the client breaks
A dialogue in a messenger breaks not at the wording of the answer but earlier — at context retention.
The client writes "what if I return it," meaning the order from three messages above, while the bot sees only the last reply and steers the conversation into a generic scenario.
In AI chatbot development, we start exactly with these points: they determine whether automation will be useful to support or add work for it.
The second failure point is a question phrased in the client’s own words. The knowledge base has no phrase "I want the same, but cheaper," and instead of a clarifying question, an off-topic answer arrives.
The third is a dead end in the scenario: the handler does not find a branch and offers an operator, but at two in the morning there is no operator, and the client closes the conversation.
In the morning, the same question arrives in Facebook and Instagram — already cold and with irritation.
For support this turns into a load that is poorly visible in reports. The share resolved on first contact falls, repeat requests grow, and operators deal with what the bot should have closed itself.
Inside the conversation this reads as "the bot doesn't understand," in metrics — as a growing queue and time to first response.
Auto-replies in messengers win not by the number of replies but by how many dialogue bottlenecks are closed before handover to a human. Here is how it is done: dialogue memory, semantic search in the knowledge base, and a dedicated night-time flow.
What changes after implementation: measurable effects
The effect of AI chatbot implementation is measured not in the number of dialogues but in support metrics: auto-reply share, first response speed, behavior during peak hours. Below is what exactly changes for each item and how it is recorded.
Share of requests without an operator
The bot closes typical scenarios — order status, delivery terms, booking, returns — and they account for the bulk of incoming requests.
On Meta channels — Facebook and Instagram — the auto-reply share reaches 60–75%, and the rest goes to an operator with the dialogue context already gathered.
First response speed
The bot replies in the very first message, regardless of time of day. The "until morning" queue, because of which some requests ended in purchase abandonment, disappears: the median first response stays within seconds instead of hours.
Work during peak hours
The peak after a newsletter or publication no longer hits the size of the support shift. The bot holds the surge — the number of simultaneous dialogues grows, not the waiting time, and after the surge the load returns to normal without manual reconfiguration.
Reduced load on support
Operators switch to complex cases, not "where is my order." A whole class of failures disappears — a request left unanswered outside the shift; repeat questions about the same issue are reduced thanks to a single knowledge base.
Implementation options: knowledge base search and fine-tuning
Before writing code, we separate two different tasks: where the bot gets facts and how it phrases the answer. Mixing them in one layer is the most common reason the bot confidently gives outdated delivery terms or invents timelines. Then the choice comes down to four working schemes, and they are not mutually exclusive.
| Option | When it fits | Limitations |
|---|---|---|
| Knowledge base search (RAG) | Catalog, terms, and policies change often, and a source link is needed | Quality depends on splitting sources into chunks and on the retrieval layer; it does not change communication style |
| Fine-tuning on chat logs | Many similar dialogues, a recognizable tone and intent labeling matter | Knowledge is fixed at the time of training: editing a policy requires retraining and re-validation |
| Hybrid scheme | High volume of requests plus strict requirements for wording accuracy | More expensive to support: two quality loops and constant dialogue labeling |
| Ready-made scenario templates | A narrow set of scenarios where predictable answers matter | Does not cover free-form wording, and the branch tree quickly grows |
We assemble the hybrid like this: fine-tuning is responsible for form, search for facts, and escalation rules decide when to bring in an operator. Content is updated at the source, and the bot answers according to the new version without retraining the model.
We measure quality by the share of answers with a confirmed source and the share of dialogues resolved without an operator.
Work stages: from analyzing chat logs to handover to support
The work proceeds in iterations: each stage ends with an artifact that can be checked before moving to the next. Below is the path from access to chat logs to the bot running on live traffic, when support understands how to manage it.
- Chat log audit. We export the request history from Messenger and Direct and label it by topic and tone. The output is a frequency map: which questions make up the bulk of the volume and what the bot closes itself versus what it passes to an operator.
- Scenario scheme. We build a dialogue tree: greeting, clarifications, order and return branches, escalation conditions. We align disputed points with support so that the bot logic does not diverge from company policy.
- Knowledge base preparation. We clean the sources — catalog, delivery terms, support answers, instructions — and split them into semantic chunks so that the bot answers on a specific point and does not invent what is not in the base.
- Channel setup. We connect the page and account, configure message intake, follow-up templates, and handover of the dialogue to an operator while preserving context.
- Testing on dialogues. We run the bot through real requests from the audit — with typos, mixed languages, off-topic questions. We calculate the share of dialogues closed without an operator and separately analyze cases of confident incorrect answers: we adjust rules and the base for them, then repeat the run on the same set.
- Launch and handover. We roll the bot out to traffic in parts: first to a portion of requests, then to all. We give support a panel with dialogue history, instructions for editing scenarios, and the procedure for returning to operators.
Deliverables: settings, schemes, and policies
We assemble the deliverables so that after handover to the team there is no "only we know where it lives." Everything related to dialogues, knowledge sources, and rights lives in your environment under version control — the Meta platform here is only a channel, and the scenario logic remains with you.
- Scenario configurations: dialogue graph, intent branching, texts, and answer variants — in editable files, not in the contractor's closed panel.
- Knowledge source map: which bases, documents, and policies are connected, how often they are reindexed, which fields enter the answer context.
- Access policies: administrator, operator, and analyst roles, the boundaries of what is available to the bot, an action log, and the procedure for granting access when an employee leaves.
- Request handling policy: rules for escalation to a live operator, maximum waiting time, behavior on an uncertain answer and on failure of an external service.
- Team instructions: quick start, how to update content without a release, how to roll back a scenario to the previous version.
- Set of checks: reference dialogues that are run after every edit and catch breakages before publication.
- Handover to the team: architecture review, collaborative editing of a live scenario, and a support period until your engineers become self-sufficient.
The package removes the main risk — dependence on a single contractor: your team can change terms, add a product line, or adjust the tone of answers on its own without waiting for us.
Case study: auto-replies instead of a queue — metrics before and after
Before implementation, support in two social networks operated in catch-up mode: 4-5 thousand requests per month, the queue grew at night and on weekends, the first response came on average after 43 minutes, and every fifth message received no answer at all. Shifts were spent on repetitive items — order status, availability, delivery times, return terms.
What was changed
The bot answers from the knowledge base with reliance on sources — catalog, delivery terms, return rules — and shows the section from which the answer was taken.
The model was fine-tuned on labels from historical dialogues: wording became closer to that of operators, so clients less often ask to call a human.
Routing was rebuilt: with low confidence or a direct request, the dialogue goes to an operator together with the history and collected data.
| Metric | Before | After |
|---|---|---|
| Share of requests closed without an operator | 12% | 68% |
| Average first response time | 43 min | 4 min |
| Average resolution time | 6 h 20 min | 1 h 10 min |
| Share of requests without an answer | 21% | 3% |
| Requests per operator per shift | 34 | 19 |
What the numbers are based on
The snapshot was compiled from Meta Business Suite and dialogue logs: six weeks before launch and six weeks after, for the same request types, so that seasonality and advertising spikes did not distort the picture. The night queue stopped being a bottleneck.
Some scenarios were deliberately left to humans: payment disputes, repeat returns, complaints — the bot collects data and passes them to an operator with a priority mark.
How does knowledge base search differ from model fine-tuning?
These are two different tools, and confusing them is costly in support. Knowledge base search inserts fragments of your documents into the context, and the model answers from current sources. Fine-tuning changes the model's behavior: answer format, tone, adherence to policy. The first is responsible for facts, the second for manner.
The choice begins with how often knowledge changes. If terms, rates, and policies are edited regularly, search gives the correct answer immediately after the document is reindexed. With fine-tuning, every edit is a new set of examples and a repeat run that takes days.
This is where the difference in maintaining the solution lies. Updating knowledge with search means uploading new files and configuring chunking; rollback means returning to the previous index version.
Fine-tuning requires maintaining a set of examples and checking that the model has not lost previous skills. The more often the subject area changes, the more expensive the second path is.
The effect on answer quality varies. Search provides grounding in a source and a link to it, but it fails if documents are poorly structured or the question is phrased unusually.
Fine-tuning evens out style and format, but does not add facts: where there is a gap in the examples, the model will start constructing the answer itself.
How to choose an approach: we collect real client questions and look at what is missing — knowledge or behavior. For facts, we set up knowledge base search; for tone and format, fine-tuning, most often in one environment. We measure the share of correct answers before and after, and only then expand the solution.
Technical concerns: data, quality, error control
Chat logs are the most sensitive area, so we agree on the storage policy before launch. Transmission and storage are encrypted, access to dialogues is granted by role, and anonymized messages go to analytics — without name, phone number, or order number.
If policy prohibits sending chat logs to external services, we deploy request processing in the customer's environment.
An incorrect answer is not only a factual error. The labels include: an off-topic answer, an invented detail (deadline, condition, warranty), a refusal when an answer exists in the knowledge base, and loss of context in a long dialogue.
Labels accumulate, and once a week we review a sample: what to add to the knowledge base, which wording to rewrite, and where the model must hand the conversation to an operator.
Are chat logs used to fine-tune general models?
No. Customer data is not used by default to fine-tune third-party models. Fine-tuning is done on an anonymized labeled set inside your environment — this is documented.
Who is responsible for quality after launch?
We are. Metrics, error analysis, and knowledge base updates are included in support. The customer only needs to confirm business rules: what counts as a correct answer.
Is it visible where the bot got the answer from?
Yes. Each reply stores its source — a knowledge base fragment or a rule. This makes error analysis substantive rather than a debate about wording.
Let’s discuss the task: what will be needed on your side
To ensure the chatbot project estimate is based on facts rather than guesses, at the start we need access to real context: live dialogues, the knowledge base, and accounts.
The list below covers most questions — based on it, we come back with an implementation plan, stage boundaries, and the order of starting work.
- Administrative access to the Facebook page and Instagram business account with rights to connect the app.
- Knowledge base in its original form: policies, support articles, catalog, price list, operator scripts. It does not need to be cleaned up in advance — analyzing and structuring it for knowledge search is our part of the work.
- 200–300 real dialogues from recent months, including unsuccessful ones and escalations: they show the actual language of clients and the places where the bot makes mistakes without fine-tuning on your examples.
- List of scenarios with mandatory handover to an operator: complaint, return, legal question, non-standard request.
- A person responsible for scenarios on your side — someone who approves the tone, wording, and escalation rules.
- Load data: daily number of requests and peak hours — they determine resilience requirements and the bot's behavior during a surge.
- Rules for handling personal data: what is permissible to send to the model and what remains in your environment.
Describe the task in two or three paragraphs — channel, goal, what is already working.
We will come back with a project estimate, implementation plan, and proposal for starting work; we will assemble the first working prototype on your dialogues so that answer quality is checked before the main development, not after release.
