You open your email on Monday morning — 400 inquiries, each requiring manual sorting. Operators spend an average of 5 minutes per analysis, and routing errors delay responses by hours. Manual classification doesn't scale: as business grows, the number of inquiries doubles, but the support team doesn't. We implemented an ML classifier that determines the topic and routes the inquiry to the right specialist in seconds. A typical scenario: a company with 50,000 inquiries per month spends 4,000 hours per year on manual sorting. Automation saves 3,200 hours and $50,000 annually — that's $15.63 per hour saved. Our approach includes taxonomy development, model training, and drift monitoring — a turnkey solution within 3–10 business days.
Automated Inquiry Classification: Taxonomy and Model
The first and most common mistake is an improper class hierarchy. Too few categories (e.g., 3) — all non-standard requests fall into "Other." Too many (500+) — the model cannot learn, accuracy drops below 70%. Fuzzy class boundaries confuse both the model and operators.
A two-level hierarchy has proven effective: the first level has 5–15 broad categories (technical issues, financial matters, contracts), and the second level has subcategories for precise routing. For example:
Technical Issues ├── Connection Problem ├── Slow Speed └── Account Errors Financial Matters ├── Payments and Tariffs ├── Refunds └── Debt According to Yandex research, a two-level taxonomy reduces classification error by 30% compared to a flat structure. [Yandex Research, 2023] We always start a project by auditing current inquiries and agreeing on the taxonomy with the business customer.
Why BERT Fine-Tuning Delivers 95% Accuracy While TF-IDF Only 85%
| Method | Accuracy | Data Requirements | Inference Speed | Implementation Cost |
|---|---|---|---|---|
| TF-IDF + Logistic Regression | 82–88% | 200 examples/class | <1 ms | Low |
| BERT fine-tuning (rubert) | 90–95% | 500+ examples/class | 5–10 ms | Medium |
| LLM zero-shot (GPT-4o-mini) | 85–92% | 0 examples | 200–500 ms | High |
TF-IDF suits quick prototypes: trains in minutes, interpretable. BERT fine-tuning is our primary method: it adds 8–12% accuracy given quality labeling. BERT outperforms TF-IDF by 10–15 percentage points but requires more data. We use LLM for new categories where historical data is absent — no fine-tuning, just a prompt with class descriptions.
How to Handle Inquiries with Multiple Topics?
"My connection is not working, and I want to change my tariff" — two classes simultaneously. We apply three strategies:
- Multilabel classification: sigmoid + threshold 0.5 — the model outputs all applicable labels.
- Sentence splitting: each sentence is classified separately, results aggregated.
- Primary + Secondary: we select the main topic (e.g., "connection problem") and a secondary one ("tariff change").
In one of our projects, 30% of inquiries contained multiple topics. Using multilabel with a threshold of 0.4, we improved routing accuracy by 22%.
What to Do About Data Drift?
Topic distribution changes: promotions increase the share of financial inquiries, seasonal incidents shift technical ones. For example, after a large promotion launch, the share of financial inquiries grew from 20% to 45% in a week — our monitoring detected the drift and automatically triggered model retraining.
We configure monitoring with a chi-square test: compare the rolling distribution over a week with historical data. On significant deviation (p < 0.05), an alert is sent, and we reassess the model — adding new classes or fine-tuning the existing one.
| Metric | Normal Range | Alert Threshold |
|---|---|---|
| Share of "Technical Issues" | 30–35% | >40% or <25% |
| Share of "Other" | <5% | >10% |
| Model Accuracy | >90% | <85% |
What's Included
Our turnkey solution includes:
- Taxonomy Development – audit 500+ inquiries, build class hierarchy.
- Data Collection & Quality Control – label 200–500 examples per class, manually verify 20%.
- Model Development – baseline (TF-IDF), fine-tuning (BERT), optionally LLM for rare topics.
- Testing – evaluate accuracy, precision, recall, p99 latency on a held-out set.
- Deployment – REST API on FastAPI, Docker containerization, Prometheus monitoring.
- Documentation & Training – describe taxonomy, routing, operator instructions.
- Warranty – model support for 6 months, updates on drift.
Work Process
- Taxonomy Analysis: audit 500+ inquiries, build class hierarchy.
- Data Collection & Quality Control: label 200–500 examples per class, manually verify 20%.
- Model Development: baseline (TF-IDF), fine-tuning (BERT), optionally LLM for rare topics.
- Testing: evaluate accuracy, precision, recall, p99 latency on a held-out set.
- Deployment: REST API on FastAPI, Docker containerization, Prometheus monitoring.
- Documentation: describe taxonomy, routing, operator instructions.
- Warranty: model support for 6 months, updates on drift.
Timelines and Pricing
Timelines: from 3 to 10 business days depending on taxonomy complexity and data volume. Pricing starts from $5,000 for a basic solution. Contact us for a project assessment. Order classifier implementation and get a free engineer consultation.
Our team has over 5 years of experience in NLP and over 50 successful projects in automating inquiry processing. We guarantee quality at every stage — from labeling to monitoring.
This classifier cuts processing time by 80% — saving thousands of operator hours per year. Contact us to discuss your project.