Why intent detection breaks on live traffic
In a pilot, intent recognition delivers accuracy near 95% on a labeled set of a few hundred utterances. On live traffic, the same model makes mistakes on every third phrase, and the share of manual transfers to an operator grows from the first days of operation.
The first source of errors is the long tail of requests. People write in ways the model has not seen: “can it be with delivery, but not this week”, “tell me, I’m confused about the tariffs”.
Each such wording occurs once, does not enter the training set, and in total these utterances account for half of requests.
The second is ambiguous formulations and mixed topics in one utterance. “Refund my money and cancel the subscription” — these are two different intents in one message, but the classifier is trained to choose one class and silently loses the second. The user does not get an answer to half the question and repeats it, already irritated.
The third is the lack of a unified dictionary. Until annotators and analysts agree where the boundary is between “cancel order” and “change order”, similar phrases go to different classes.
The taxonomy blurs, the metric on the holdout set looks decent, but intent recognition errors surface in the dialogue.
On top of that comes input noise — typos, transliteration, recognized speech. That is exactly where everything that worked well in the demo breaks. Next we discuss how this is addressed.
What a managed intent taxonomy gives
While intents are collected as a list of phrases “as they came out”, close classes overlap, and the bot responds off-topic.
A managed taxonomy is a hierarchy domain → intent → sub-intent, where each class has a response rule, and for everything uncovered a separate out-of-scope class is created. Accuracy is calculated per domain separately, and a weak domain does not blur the overall picture.
| Option | When suitable | Limitations |
|---|---|---|
| Flat list, 30–50 intents | One support scenario, narrow product base | New topics fall into the nearest intent — false positives grow |
| Hierarchy domain → intent → sub-intent | Several areas, different teams | Requires labeling and arbitration of disputed examples |
| Taxonomy with explicit out-of-scope class | Open base, stream of unfamiliar formulations | Requires regular review: some new topics are moved into real intents |
The effect is visible on three metrics. The auto-response share grows from 50–60% to 85%: the model stops guessing between similar intents.
The manual transfer share falls 2–3 times — the operator connects only when confidence is below the threshold or with an explicit out-of-scope request.
Stability on new topics is provided by the out-of-scope class: unfamiliar wording is not pulled toward the nearest intent but enters the review queue, and the taxonomy is revised based on these examples.
A whole class of failures also goes away — “the bot confidently answered wrong”, which is more expensive than an honest transfer to an operator. Confidence threshold per domain and explicit hierarchy — a proven practice in open platforms like Rasa.
Approaches to intent detection: rules, learning, hybrid
| Approach | When suitable | Limitations |
|---|---|---|
| Rules and dictionaries | 10–30 intents, stable formulations, no labeling yet | breaks on paraphrases and typos; dictionary grows faster than accuracy |
| Learning on labeled dialogues | live speech, thousands of examples, intents overlap | requires labeling and re-annotation when the taxonomy changes; rare intents underperform |
| Hybrid approach | rules as a fast layer on critical intents, model on the rest | harder debugging: the error lives either in the rule or in the model |
The choice here is determined not by fashion, but by dialogue volume and stability of the subject area.
Rules and dictionaries give predictability on a narrow set of intents, learning on labels — generalization to paraphrases, hybrid — a compromise between controllability and flexibility.
A hybrid is usually assembled as a cascade: the dictionary layer handles unambiguous requests, the model distributes the remainder, and the confidence threshold decides whether to answer automatically or clarify.
The threshold is selected by the confusion matrix — too high drives the operator transfer share up, too low increases the number of incorrect answers. We fix it on validation and revisit it with every taxonomy change, otherwise quality quietly drifts.
What to choose depends on what is more expensive specifically for you: a false answer or an extra operator transfer. Describe the task and show dialogue logs — we'll return with an assessment of which of the three options will cover your intents.
Implementation: from dialogue audit to handover of regulations
Intent detection implementation is not a one-time model training but a cycle: audit of real dialogues sets the taxonomy, labeling refines it, and the holdout set shows where the classifier errs on new formulations.
We proceed in stages; after each, a measurable result is visible — share of correct detections, number of extra clarifications, depth of the operator queue.
- Dialogue audit. We export request history for a period, calculate the share of unrecognized and scenarios with repeated clarifications. This shows which intents should be separated and which a regular menu covers.
- Taxonomy schema. We assemble intents into a hierarchy: top is the client task, below clarifications and entities. We check class intersections so similar requests do not compete for one label.
- Intent labeling. We write rules for disputed cases and keep a control sample for consistency checking. Annotator disagreement above the threshold returns us to schema edits, not training.
- Confidence setup. We set the threshold below which the bot asks a clarifying question instead of guessing, and the rule for operator transfer. False confidence on a rare class costs more than an extra clarification.
- Holdout set. A set of dialogues not included in training is run before release: we look at the confusion matrix by class, not one overall accuracy — otherwise rare intents hide behind the average.
- Validation and handover of regulations. We release in shadow mode: the model classifies, the old scenario still answers. Then we hand over the update regulation — retraining frequency, degradation metrics, owner responsible for taxonomy edits.
Delivery package: taxonomy, schema, regulations
Artifacts that describe what we consider an intent and how we handle each response remain with the customer along with the code. Without them, the model in a month turns into a black box, and any scenario change requires returning to the contractor. Below is the package we fix in acceptance.
- Intent dictionary (taxonomy). Intent tree with description, example phrases for each class, boundaries between close intents and labeling rules for disputed cases. A separate block — what we intentionally do not attribute to any intent.
- Routing schema. Table “intent → scenario or operator”, confidence thresholds, behavior at low confidence and priorities if several classes trigger simultaneously.
- Processing policies. What goes into logs, what is masked or not stored at all, retention period, access roles, and procedure for exporting data for error analysis.
- Taxonomy documentation. Procedure for adding and renaming intents, dictionary versioning, owner on the customer side and business approval points.
- Retraining regulation. Update triggers (share of unrecognized, drift in user formulations), frequency, who labels new examples and how we validate quality before release.
- Control set and quality report. Fixed dialogue sample, metrics per intent, comparison with the previous version and rollback procedure on degradation.
- Support instructions. Where to route non-typical requests, how to escalate and who updates the dictionary, so new cases are not lost between teams.
Case: how false operator transfers were eliminated
What was
The client came with a support bot that transferred more than a third of dialogues to an operator — including those where the answer was already in the knowledge base.
Intent taxonomy was assembled manually and contained intersecting classes: “order status” and “where is the package” went into different categories, and the model assigned the request to both with equal confidence. The confidence threshold was fixed at launch and not revisited.
On the holdout set, the share of correct recognitions was 72%, the share of false operator transfers — 34%. Every third dialogue took operator time, although automation would close it faster.
What changed
We rebuilt the taxonomy: removed intersections, separated close intents and added a separate class “not recognized” instead of forcing the request into the nearest category. We labeled a stratified holdout set in Label Studio — now metrics are calculated on the same data.
The confidence threshold was selected by the confusion matrix: a false transfer is more expensive than a clarifying question, so the bot first asks again instead of calling a human.
What became
Share of correct recognitions on the same set — 94%, false operator transfers — 7%. Share of dialogues closed without operator involvement grew from 58% to 88%.
Monitoring was set on the share of low-confidence recognitions, intent distribution by day and divergence with manual labeling: after editing formulations in the interface, growth of such responses was visible the same day, and retraining took hours instead of a week of complaints.
How intent differs from dialogue scenario
An intent is what a person wants to get in one utterance: return an order, find out delivery status, change address. A scenario is a sequence of steps and dialogue states that serves this desire: clarifying details, slots, calls to internal systems, completion point.
The layers live by different laws: intent is determined by the model from text and updated by retraining on labels, the scenario is described by a transition graph and updated by editing a branch.
If they are mixed in one schema, every new customer formulation turns into a separate graph branch. The taxonomy grows, the number of transitions grows combinatorially, and any wording edit requires regression of the entire dialogue.
Worse: utterances with mixed topics — “I want to return an order and also change the address” — have nowhere to go, because a graph branch implies one goal at input.
A routing layer is needed between these layers. It receives the utterance, computes a confidence score for each intent from the taxonomy, applies the threshold and conflict resolution rule, and only then passes control to the scenario along with the current dialogue state.
The scenario knows nothing about formulations, and the model — about slots and steps. Edits on both sides remain local, and regression does not spread across the entire schema.
A clarifying question is part of routing, not a separate branch. When two intents are close in confidence or an utterance mixes topics, the router asks a clarifying question and holds the dialogue until it selects one goal.
This way the scenario graph remains linear and predictable, and disputed cases are resolved in one place — where they should be resolved.
Technical doubts before start
Before start, the customer checks not the model itself but its operation: how current logic is transferred, how much data is needed and what will happen to quality in six months. Below are five questions that we most often discuss in a technical discussion.
What to do with old rules?
We leave them as a separate layer: regexes handle hard cases like order numbers, the model — free formulations. Priority on conflict is set explicitly and visible in logs.
How many phrases need to be labeled manually?
We calculate the volume from the number of intents and variability of formulations, not “by eye”. Start — several hundred phrases per class, then the sample grows based on uncertain predictions.
Where to put unrecognized phrases?
Into the review queue, not into a stub. We introduce a confidence threshold and a “not understood” class; the share of such requests is a metric showing where the taxonomy is lacking.
Does the model degrade over time?
Yes, if the clients' language changes. We monitor request distribution and accuracy on the control sample; on degradation we fine-tune on fresh data — in the stack on Rasa this is a regular cycle.
How is the taxonomy updated?
We create new classes based on statistics of unrecognized phrases, preserving name compatibility. We do not delete old ones abruptly — we move them to archive so reports remain comparable.
Let's discuss the task and return with an assessment
For an assessment we need not general wishes but your data: in the first conversation we analyze how the bot processes requests now and where it loses users.
If there is a chat export and a list of integrations, we estimate the volume of the intent taxonomy and scope of work already from the conversation.
What we'll discuss and what inputs are needed from your side:
- Audit of the current bot — which layer is responsible for recognition: rules, regular expressions or a trained model, and on which requests it errs.
- Dialogue inputs — log export for a period, share of unrecognized utterances, several hundred labeled examples for key scenarios.
- Taxonomy boundaries — list of scenarios and desired granularity: large intents or subclasses with a separate reaction to unclear input.
- Data and access — channels (messengers, web widget, phone), integrations with CRM or reference system, test environment for running the model.
- Language and style — supported languages and dialects, share of typos and transliteration, whether voice message processing is needed.
- Acceptance criteria — target share of correct detections, acceptable false level, on which sample we calculate quality.
- Start order — what we do first: measuring current quality, labeling, draft taxonomy.
- Work plan — stages, checkpoints, team composition and procedure for handing the model over to your engineers.
After the conversation we'll return with an assessment, work plan and list of risks. Describe the task and send the log export — that's enough to move forward.
