Intent Detection for Chatbots: Taxonomy, Model, Monitoring
"I want to order pizza", "status of my order", "how to get a refund" — three different intents. If the chatbot cannot distinguish them, the user leaves. Without quality intent detection, a bot cannot correctly route requests, handle errors, or sustain a dialogue. Misclassification leads to customer loss and increased operator load. We solve this by designing the taxonomy, selecting the model (BERT, LLM, or hybrid approach), and implementing monitoring.
The result: >90% accuracy on target intents, p99 latency <200 ms. Our intent detection solutions have been tested under real loads: up to 10,000 requests per day, with quality degradation of no more than 3% after six months. Experience: 7+ years in NLP, 20+ projects, including bots for e-commerce, fintech, and technical support. Savings on request handling for clients: up to 60% reduction in manual support costs. Fast payback — within 2–3 months. Typical project cost ranges from $5,000 to $20,000 depending on complexity.
Designing Intent Taxonomy
Rule: each intent is a single concrete intention with a clear handler. Common mistakes:
- Too broad intent:
help— unclear what to do. - Duplication:
order_statusandcheck_my_order— same intent. - Fuzzy boundaries:
complaintvsnegative_feedback— often ambiguous.
For a moderately complex chatbot: 20–50 intents. More than 100 is a sign of poor architecture; revise the hierarchy.
Why Hybrid Approach Is Better Than Classic
The classic approach (Rasa NLU, Dialogflow) trains on labeled examples: deterministic, fast (<10 ms), but requires 50–200 examples per intent and does not adapt without retraining. LLM-based (GPT-4o-mini with few-shot) requires no training, is flexible, but slower (200–500 ms) and more expensive in cost per token. Hybrid — BERT as the first pass (fast, cheap), LLM as fallback for low-confidence and edge cases. In our projects, we use exactly this scheme: it provides a balance of speed and coverage.
| Approach | Speed (p50) | Training | Flexibility | Inference Cost |
|---|---|---|---|---|
| BERT fine-tuned | <10 ms | 50–200 examples/intent | Low | <$0.0001/request |
| LLM (GPT-4o-mini) | 200–500 ms | zero-shot | High | $0.002/request |
| Hybrid (BERT→LLM) | <15 ms (BERT) / fallback | Combined | Medium | $0.0003/request |
Comparison of key metrics: hybrid is 10x faster than pure LLM with comparable coverage. More about the BERT model.
How We Implement the Hybrid Classifier
-
Audit and taxonomy design. Analyze dialogue history, identify frequent topics, build an intent map. Conduct a workshop with your team to define intent boundaries. Result: a document with hierarchy, slots, and fallback rules.
-
Dataset labeling. Collect 100–300 examples per intent (real user phrases). Add confusion matrix negative examples — phrases from similar intents to "stretch" class boundaries. Example: "I want to change my order" vs "I want to cancel my order" — different intents, but both must be in the dataset.
-
Training and calibration. Fine-tune BERT (e.g., DeepPavlov/rubert-base-cased) with cross-entropy. Calibrate confidence threshold (usually 0.6): when confidence is lower, pass the request to LLM.
from transformers import pipeline intent_classifier = pipeline( "text-classification", model="./intent_classifier", tokenizer="DeepPavlov/rubert-base-cased", top_k=3 ) def detect_intent(text: str) -> IntentResult: results = intent_classifier(text) top = results[0] if top["score"] < 0.6: # Fallback to LLM return llm_classify_intent(text) return IntentResult(intent=top["label"], confidence=top["score"]) - Integration and deployment. Deploy the model via Triton Inference Server or ONNX Runtime — latency p99 <50 ms on GPU. Set up monitoring: confusion matrix weekly, OOS logs, feedback pipeline. We ensure the model does not degrade on new data: if accuracy drops below 85%, an automatic trigger initiates retraining.
Monitoring Confusion Matrix and Preventing Drift
The confusion matrix is the main tool for identifying problematic intent pairs. If, for example, order_status and change_order are often confused, add more confusing negatives to the dataset. Regular analysis of OOS logs helps detect new user intentions that should be extracted as separate intents. As a result, classification accuracy does not decline over time.
| Metric | Without Monitoring | With Monitoring |
|---|---|---|
| Accuracy after 6 months | 82% | 88% |
| OOS requests share | 15% | 8% |
Project Deliverables
- Documentation of intent taxonomy (Google Docs / Confluence)
- Labeled dataset (JSONL, CSV format)
- Trained model + Docker image for deployment
- Integration with your bot (REST API, gRPC)
- Load testing (result: p99 latency <200 ms)
- Monitoring and alerting (Grafana dashboards)
- Training of two of your engineers on the pipeline
Timeline and How to Order
Development timeline: from 2 to 6 weeks depending on taxonomy complexity and dataset size. Cost is calculated individually after an audit. We will evaluate your project for free — contact us for a consultation. Order intent detection development and reduce interpretation errors by 30–70%. Get a consultation — we will select the optimal stack for your budget. Our team is certified in MLOps and guarantees performance improvements.