Why dialogue management falls apart on live scenarios
Most often a scenario passes the demo and falls apart at the third or fourth message from a live person.
The reason is not the model, but the fact that the conversation state is not stored separately anywhere: the next reply is derived from the last phrase, not from the whole history.
As soon as a person answers off-script — "yes, but Wednesday is better" — the branch stops matching reality.
Hence the loss of dialogue context: the bot confuses answers, re-asks for fields that are already filled, and rolls back two steps. The user reads this as "they don't hear me" and leaves, while a formally successful dialogue with no result remains in the logs.
Unhandled requests are the second failure point. An intent that isn't in the branching tree cuts the conversation short: the bot goes silent or repeats the menu.
The list of such phrases grows faster than the scenario, and without a separate recognition layer every new request again hits a dead end.
The third is handover to an operator. The dialogue is cut off mid-sentence: the agent receives neither the history of replies nor the filled fields, and the person explains the task again from scratch. The outcome is a lost customer or manual transfer of data from the chat into the accounting system.
These are not mistakes in individual replies, but the absence of a control layer that holds state, recognizes intent, and passes the context over in full. Let's break down how this is closed.
The effect in metrics: self-service answer rate, handover to an operator, drop-offs
We measure the effect along the end-to-end request funnel, not from the bot logs: the self-service answer rate is the share of requests where the last message belongs to the bot and there was no handover to an operator within the wait window.
The metric is fluid: shift the confidence threshold and there are fewer self-service answers, but also fewer false triggers. That is why we fix the thresholds together with the client before launch and change them only based on accumulated statistics.
The bot's response time at peaks does not grow, because routing by intent is deterministic: typical scenarios are closed by rules and slots, and the language model is connected only on branches where there is no template answer.
Add to that a cache of frequent formulations and explicit degradation: if an external service does not respond, the bot does not go silent but offers a retry, an alternative step, or a handover to an operator.
We count failure classes separately. Intent not recognized — treated with labeling and clarifying questions in the branch. Intent understood but no data in the system — fixed by integration and a clear message about the reason.
A handover policy triggered by risk or emotion — this is not a failure but a planned trigger, and we keep its share manageable. A communication failure — retries and degradation without loss of dialogue context.
The result shows up in reduced support load: it is not just the number of requests to an operator that falls, but their composition that changes — the complex, disputed and emotional ones remain.
We describe branching policies declaratively, as rules and scenarios in Rasa, so the handover boundary can be moved without rewriting the dialogue.
Three modes of assembling response logic and their limitations
The modes differ in where the decision about the next reply is made: in described rules, in retrieval over a knowledge base, or in a generative model. Below is where each hits its ceiling and in which case it is swapped for another.
A scripted bot is predictable: the answer is the same at any moment, it is easy to verify and explain, but as the number of topics grows the combination of branches grows faster than the list of questions, and an undescribed path turns into a dead end.
Retrieval over a knowledge base covers the long tail of formulations by answering by meaning, but does not run a multi-step dialogue.
Generative answers sound natural, yet without grounding in a retrieved fragment they fill in the gaps and give a different result for the same question.
| Mode | When it fits | Where it hits the ceiling |
|---|---|---|
| Scripted bot | Repeatable operations: request, status, booking — where identical wording matters | More branch combinations than questions; an undefined path is a dead end |
| Retrieval over a knowledge base | Reference on documents, instructions, terms — the long tail of formulations | Does not run a dialogue, easily loses the context of previous replies |
| Generative answers | Free speech, paraphrases, clarifications | Without a source it fills in the gaps, the result is not reproducible from run to run |
| Hybrid scheme | Different request types: operations, reference, and free dialogue together | Requires routing and a request log to debug the boundaries between modes |
The working configuration in most projects is hybrid: a request goes through routing and is sent to its own mode, while critical operations stay on the scenario.
This way reference does not break the accounting logic, and generation answers only where there is a verified source beneath it.
Implementation stages: from scenario audit to handover into operation
Dialogue management is poorly suited to "tuning up" on a live bot: transition rules drift apart, and the bottleneck only becomes visible on traffic. That is why we work in stages, and each one ends with an artifact that can be verified before the next step.
Order of work
- Audit of scenarios and current requests — we export the dialogue history, calculate the share of drop-offs at each step and the topics most often handed to an operator. The output is a map of problem transitions.
- Intent labeling — we assemble a corpus of real phrases, mark up intents, slots, failures, out-of-scenario questions and typos. The agreed labeling becomes the reference for validation.
- Dialogue scheme design — a state graph, mandatory clarifications, re-ask rules, escalation to an operator and fallback branches. We fix the scheme before configuration.
- Configuration and assembly — transition rules, reply wording, confidence thresholds, dialogue length limits. Everything that changes without rewriting code we move into configuration.
- Bot testing — running the reference labeling, "blind" dialogues by engineers, checking failures and dead-end branches. We catch errors before the production launch.
- Validation on production traffic — launching on a subset of requests with an operator working in parallel. We look at the share of dialogues closed without a human and the number of re-asks.
- Commissioning and handover — the dialogue scheme, the regulation for editing scenarios, monitoring dashboards and training for your team. From then on you make changes yourselves.
Deliverables: configurations, dialogue schemes and regulations
A working model is not the only thing the client is left with. Dialogue management outlives the first version of the bot: request topics change, services are added, escalation rules are refined.
That is why we assemble the deliverables so that dialogue branches can be edited without digging through someone else's code and without coming to us for every formulation.
- Dialogue scheme as a state graph: branches, transitions, return points, behavior on an unrecognized reply — it shows where to embed a new scenario.
- A set of intents and entities with example replies — it also serves as a regression set: after edits we run it and see whether neighboring branches have degraded.
- Handover policies to an operator: escalation conditions — how many misunderstandings in a row, keywords, topics outside competence — the composition of the transferred context and the rules for returning to the bot.
- Logging settings: the intent that triggered, recognition confidence, the fact of escalation, the session identifier; the export form for analysis.
- Bot documentation: the integration map, behavior when external systems are unavailable, failure points and actions for each of them.
- Editing regulation: who changes branches, which checks are mandatory before deployment, how a version is rolled back and how the change history is kept.
- Instructions for operators and an acceptance checklist — so that operation does not depend on a single person.
Case: fewer handovers to an operator and stable answers at peaks
What it was
The bot had been built for a single scenario and parsed a reply by keywords and regexes — to the paraphrase "what if I want to return it" it answered with the wrong branch.
Context lived within a single step: slots named at the start of the conversation were lost by the third reply, and the bot asked for the order number again.
The result: 41% of dialogues ended in a handover to a human, operator wait time at peak reached up to seven minutes, and the flow of requests to support grew faster than the number of dialogues.
What we changed
We assembled the core of the dialogue management on Rasa and moved to an explicit model: intents with a confidence threshold, slots with required status and a fill source, scenario states stored separately from the reply text.
Branching is described declaratively, so a new topic is added by configuration rather than by a code patch.
The escalation rule became part of the scenario: three unrecognized replies in a row, a direct request for a human, or a critical intent — and the dialogue goes over with its history and filled slots, so the operator does not re-ask from scratch.
What it became
The share of self-service answers grew to 79%, peak wait time dropped to 40 seconds, and the number of handovers to an operator halved while traffic grew.
Every step is written to the log together with the intent and confidence, and the set of real requests became a regression test: before rollout you can see which scenarios have slipped and where the confidence threshold triggers too late.
How dialogue management differs from a bot scenario
A scenario is a transition tree: the bot's state equals the step number, and the user's reply merely selects the next branch. As long as the person follows the expected path, everything is predictable.
But if they change the topic, ask a counter-question or go two steps back, the tree breaks — the bot repeats the menu or leads into a dead end.
Instead of a step number, dialogue management holds a model of the conversation state. In it, recognized intents, filled fields, a stack of unresolved topics and a flag that the user was not understood live separately.
A reply does not switch the bot to a branch but updates this model — which is why changing the topic and returning to a deferred question within one session does not require new branches in the scheme.
Context management is a set of rules, not the model's memory. We define how many replies to store, which values count as confirmed, when to ask a clarification and when to silently accept an answer.
A separate policy is responsible for escalation: an explicit request to call a human, two misunderstandings in a row, negativity or going beyond the topic — the dialogue goes to an operator together with its history and already collected data.
The scenario is not thrown away in the process. Branches where an error is unacceptable — payment, identification, request confirmation — remain strictly deterministic, and the dialogue layer works around them.
This hybrid gives freedom in the conversation without losing control at critical steps, in any channel — from a website widget to Telegram.
Technical concerns: context, access, rolling back changes
Context depth is a configuration parameter, not a property of the model.
We keep the last replies in full and roll earlier ones into a summary that is updated after each block of turns: the bot remembers the essence of the request and the agreements, but does not drag the entire correspondence into every request — response time stays predictable.
We issue access in a targeted way: a scenario gets only those operations of internal systems that it actually needs, with restrictions by request status and operator role, and everything else is requested by a human. We deploy updates as revisions, so yesterday's working version stays at hand.
How deeply does the bot remember the dialogue?
The context window is configurable: how many of the last replies are passed verbatim and after which threshold the history is rolled into a summary.
The summary is stored separately from the conversation log, so a long conversation does not bloat the request and does not hurt response speed. If a client returns a week later, we pull up their card by identifier rather than the entire past correspondence.
How does the bot get access to internal systems?
Through a separate service layer with a limited set of operations: read a request status, add a comment, create a request. Credentials are stored in a secrets vault and are unavailable to the bot, and every call is recorded in the log with the dialogue identifier. Access is revoked at any moment, without stopping the bot.
What to do if a new version of the scenario performs worse?
Scenarios are versioned like code: each revision is a separate artifact, and model instructions change independently of the logic. We release first to a small share of traffic and compare metrics with the previous version.
If the share of dialogues that needed an operator grows — we switch to the previous revision, which takes minutes and requires no rebuild.
How is personal data handled?
In the logs we mask numbers, addresses and payment details; raw records live for a limited period and are then deleted. For refining scenarios we use anonymized samples.
If the client's requirements say data must not leave their environment — we deploy processing inside, and issue access to the logs by role.
Let's discuss the task and come back with an estimate and a work plan
So that the scoping takes a single pass rather than a series of clarifying emails, it helps to send the inputs right away — in whatever form they already exist. Perfectly formatted documents are not needed; working materials are enough.
- Scenarios: what the bot closes on its own and where it must hand the dialogue over to an operator.
- Real requests: 30–50 dialogues from the current support channel in their original form — they show live formulations rather than invented ones.
- Source systems: CRM, knowledge base, billing system, booking — what is available programmatically and what will have to be provided as an export.
- Escalation rules: at what moment and to whom the dialogue goes, and what the operator sees in the card.
- Channels and constraints: messengers, website chat, voice; requirements for response speed and data accuracy.
- Success criteria: the share of dialogues closed without a human, the number of clarifying questions per dialogue, the share of errors in answers.
From this we assemble a project estimate and a work plan: stages, the scope of work for each, dependencies on your side and the points where we show an interim result.
We design dialogue management before writing code — the transition scheme and branching rules are agreed in advance, so expensive rework at later stages does not arise.
Describe the task — we will come back with an estimate and a work plan.
