Text classification: why manual triage does not scale
The flow of requests grows faster than the team can process it.
Manual triage of requests runs into human throughput: one operator processes a limited number of emails per shift, and as soon as the flow exceeds the norm, the request processing queue grows faster than it is cleared. Delay accumulates—unprocessed items wait, reminders and follow-up emails pile onto them.
The cost of delay is not only the response time to a request.
While the queue is shorter than a shift, the pause is almost unnoticeable; when it exceeds a day, some requests turn into “the customer left before waiting.” At the same time, labels drift: discrepancies in labels between operators consistently arise on ambiguous categories, and the higher the pace, the greater the share of such discrepancies—people have neither the time nor a single criterion to decide identically.
Next, analytics suffers: accumulated label discrepancies break summary reports, and decisions are made on noisy data—a topic seems unpopular, although demand has simply spread across neighboring categories.
Text classification closes the most expensive stage—routing: the model assigns a request to a category, returns a confidence score, and passes only ambiguous cases to a human.
Below we break down how this is handled in practice: from statistical models like TF-IDF to language models, how they differ, how quality is measured, and how the new stage fits into the current request processing queue without stopping support.
Where automated request triage delivers a measurable effect
The share of automatically triaged requests is the baseline metric, and it must be calculated by requests closed without human involvement, not by the number of model responses.
On a typical flow—tickets, order questions, complaints, document requests—text classification confidently closes from half to three quarters, and the rest deliberately goes to an operator.
The speed of sorting the flow changes more noticeably than it seems: previously a request waited for an operator in a common queue, during peak hours—until the next shift.
Now request routing happens at the moment of arrival: topic, urgency, language, and addressee are determined immediately, and waiting is measured in minutes, not shifts.
Reduction of manual processing is a consequence of the operator no longer being a sorter and working only with complex cases.
If the model confidence is below the threshold, the request goes to a human with a note and the reason for the decision, rather than to a random department or the common queue.
The most valuable thing here is the eliminated class of errors, not speed. These are requests that sat unprocessed for days, duplicates from one client, requests to the wrong addressee, and underestimated priority for urgent ones. Such a class is closed by a processing rule, not by the vigilance of a specific employee.
Verifiability of the effect is built in before the start of automated triage: we take historical flow, label a sample, and see what share is handled without a human and where the model makes mistakes.
If the flow is small or homogeneous, there will be no noticeable share of automatic decisions—this is also a fair result of the analysis, and it is better to get it before work begins.
Comparison of approaches: rules, learning from examples, ready-made models
The choice of approach is determined by three parameters that are usually not stated at the start: how many classes need to be distinguished, how many labeled examples are available, and how “live” the wording in the input data is.
On a formal vocabulary, rules win; on stable subject matter, learning from labeled examples wins; on a long tail of classes and a flow of new wording, a ready-made language model wins. Below is a comparison with limitations that appear in operation, not in demos.
| Variant | When it fits | Limitations |
|---|---|---|
| Rules and dictionaries | Classes are described by formal features: details, markers, template requests | They rely on manual maintenance, break on new wording, every dictionary change is a regression risk |
| Learning from labeled examples (TF-IDF and linear models) | There are 1–2 thousand examples per class, subject matter and style are stable | Accuracy on rare classes drops, labeling has to be updated when the subject matter changes |
| Ready-made language model | Many classes, little labeling, live wording and user queries | More expensive at volume, quality control and an explicit rollback point to the previous version are needed |
External measurements on open text processing task sets aggregated by Papers with Code show that with 1–2 thousand labeled examples per class, the gap between classical models and ready-made language models almost disappears. It persists in two places—rare classes and wording that was not in the sample.
Therefore, in operation, a hybrid is more often assembled: a feature-based model closes frequent classes as a fast filter, and a ready-made model is connected where classifier confidence is below the threshold.
Implementing text classification: stages from audit to handover
Labeling and training are half the work; the other half is a reproducible order in which the result is transferred to new data. Below are the stages of implementing text classification as we go through them: from data audit to handing the model over to operation.
- Data audit. We export the corpus, calculate the share of duplicates, empty fields, and texts without useful payload, and look at the distribution by topic. We record what is labeled and what will have to be labeled, and how many examples need to be added to reach a working volume.
- Class schema. We assemble a taxonomy without intersections: one text—one class or several, what to do with mixed cases. We run the ambiguous sample through two annotators and resolve discrepancies with rules, not verbal agreements.
- Working model and baseline. We go from simple frequency models to vector representations and language models, increasing complexity only where it gives a gain on the holdout set. We select the confidence threshold according to the cost of an error.
- Setup and integration. We describe input preprocessing, garbage filtering, model and labeling versioning, and logging of predictions together with confidence—without this, error analysis a month later turns into guesswork.
- Quality check. We calculate precision and recall for each class, analyze the confusion matrix and borderline examples. Separately, we look at behavior on data from another period or source: this shows where the model relies on time features rather than the meaning of the text.
- Handover to operation. We provide the retraining procedure, rules for rolling back to the previous version, monitoring of the share of uncertain answers, and instructions for the customer’s team.
Delivery package: models, schemas, and labeling regulations
A model without an environment is an artifact that in a month no one will be able to retrain or explain. Therefore, we assemble the delivery package so that the text classification task lives without us: from weights to the rules by which annotators handle ambiguous cases.
- Trained model and invocation code — weights, an inference wrapper, pinned dependencies; it comes up on your infrastructure with one command, without building “from memory.”
- Class schema — a taxonomy with boundaries between similar categories, examples for each, and a list of pairs that are confused most often.
- Labeling regulations — instructions for annotators: conflict resolution, escalation procedure for ambiguous cases, requirements for agreement between annotators.
- Data access policy — where texts are stored, who reads them and on what basis, what is masked before logging, and how the request trail is structured.
- Model documentation — quality metrics for each class, not one averaged number; known limitations and signs of beginning degradation.
- Set of validation examples — a reference sample with labels on which every new version is run: a drop is visible before rollout, not after complaints.
- Deployment configurations — environment descriptions, service schemas, limits, and queues; the deployment is reproduced without reading someone else’s code and without questions to us.
We hand all this over together with a walkthrough for your team: where to change the class list, how to add examples, and when it is time to retrain.
Case study: incoming request triage before and after implementation
Initial picture
Requests fell into a common queue, and triage relied on keyword rules: they confidently handled 42% of the flow and fell apart on paraphrases.
The rest was manually sorted by operators—the median time to reach the right specialist was about three hours, and the share of incorrect labels reached 28%. On peak days, the queue grew faster than it could be processed.
What was changed
From the archive, we assembled a corpus of requests and fixed the class schema, and set aside a control sample—one thousand requests—and did not use it in training.
On it, we measured quality at each step: first a baseline model on frequency features, then a model on a pretrained language representation, and on top—rules for rare and legally significant categories. Labeling was cross-checked with operators so that the labels matched their picture of the world.
After implementation, 88% of requests are triaged automatically, the share of incorrect labels is 6%, and sorting time dropped to 12 minutes. The control sample is relabeled monthly: if discrepancies on it exceed 5%, retraining on fresh data is launched.
Such a loop shows a shift in customer language or the emergence of a new category before it is noticed through complaints—the logic of quality assessment is covered in ACL Anthology publications.
How does a custom model differ from a ready-made language model?
This is not a choice of “better or worse,” but of what data and what cost of error. A custom classifier is trained on your labeling; a ready-made language model solves the task from an instruction, without training. The difference shows up on your classes, not on demo examples.
A custom model wins where classes are narrow and close: “complaint about delivery times” and “claim about quality” are almost the same for a general model, but for the business they are different processing routes.
Trained on your examples, the classifier draws the boundary the way you drew it, and on its own classes gives quality measurable on a holdout set.
A ready-made language model is needed when there are no labeled examples and a solution is required quickly.
It handles wording that was not encountered in training and does not require corpus preparation—but on rare categories it answers inconsistently, and quality has to be checked selectively: the answer is generated anew on each request.
Adaptation for new classes is also structured differently. For a custom model, labeled examples are added and it is retrained—behavior is reproducible, metrics are comparable from version to version.
For a ready-made model, the instruction is edited and successful examples are selected, and after editing it cannot be guaranteed that previously correct answers have not shifted.
In practice, a combination more often works: a ready-made language model labels a large body of text, a custom classifier is trained on that labeling, and then it works quickly and predictably. The selection criterion is one—quality on your classes and how often those classes will change.
Common technical questions about the project
Labeling and model support determine whether classification pays off. Below are five questions most often heard in technical sessions, with substantive answers.
How much data needs to be labeled to start?
For a baseline model using sparse features, a few thousand examples per class are enough—this is sufficient to close 80–90% of typical requests. A neural network requires a larger volume, but we move to it only where a simple model consistently makes mistakes.
What should be done if there is no historical data?
We start with rules and keyword phrase search: they close obvious cases and at the same time label the corpus. Confident answers become the training set—history accumulates in the process, and a separate archive is not needed.
Annotators interpret categories differently—is that a problem?
This is the main quality ceiling. Before labeling, we fix an instruction with borderline examples, then calculate agreement between annotators, and discuss ambiguous cases in a session. If a category can be read in two ways, we fix the instruction, not the data.
How do you catch quality degradation?
We monitor the share of confident predictions, the class distribution, and the share of requests sent to manual triage.
A change in input texts is data drift and is treated by retraining; a drop in metrics with the same input is degradation, and rolling back to the previous version restores quality.
How often should the model be retrained?
By event, not by calendar: metrics have gone beyond the threshold or a batch of labeled corrections has accumulated. Each cycle produces a version with a metrics log and the ability to roll back, so retraining does not become irreversible.
Let’s discuss the task: what we need from you and when we’ll return with a plan
For the assessment to be substantive and not “based on feelings,” we need a minimum of context: where classification fits into the process, what volume of texts passes through it, and what counts as an error.
It takes one conversation with the customer, but it removes half the clarifying questions at the assessment stage.
- Problem statement: what process we serve—request routing, review labeling, document sorting—and what exactly the model decides.
- Source data from the customer: export of texts for a period, preferably with labels already assigned for at least part of the examples.
- List of categories and rules for their intersection: what to do if a text fits two at once.
- Quality requirements: what share of errors is acceptable and which error is more expensive—a miss or a false positive.
- Integration point: where we return the result—to the ticketing system, document storage, or internal service, and in what form.
- Constraints: personal data, closed environment, storage requirements—they determine which models are permissible.
The assessment procedure is as follows: we analyze the data, check whether there are enough examples for training, and return with a work plan.
As the first stage, we set up a pilot on one process—it shows real quality on your texts, and only after that do we scale to the remaining flows.
Describe the task and attach what you have on the data—we’ll return with an estimate and a work plan.
