Why citizen appeal classification breaks under high volume
Manual sorting of appeals holds only up to a certain volume.
An operator reads the full text, assigns it to a topic based on keywords, and passes it on — at hundreds of appeals per day this still works; at thousands, the queue in the contact center starts to grow, and the time to first response increases faster than headcount.
Then topic routing breaks. A single topic often involves several agencies, and each sees only its own queue — an appeal sent to the wrong inbox comes back or hangs until someone follows up manually.
Duplicates make the picture worse: the same appeal arrives through several channels, it is registered again, and a noticeable share of resources goes to processing repeats instead of new ones.
An appeal is rarely lost as an obvious failure — usually it is a quiet break at the handoff. There is no end-to-end identifier, no log of transfers between agencies, and at any moment no one can say who currently has the appeal.
A single intake window without such a link turns into just another entry point rather than a single point of responsibility.
The cost of an error does not end with delay: an incorrectly identified topic produces a response sent to the wrong place, the citizen writes again, and the flow feeds on itself.
We break down how this is closed — where manual sorting is replaced by a model, how topic routing accounts for agency rules, and how a single intake window with end-to-end status is built.
What changes after implementation: the effect in process metrics
A measurable effect appears only where the classifier's decision and the operator's action are recorded in a single processing log.
For each appeal, we save the proposed category, the model's confidence, and the final human decision — the discrepancy becomes a training set, and quality grows with the volume of the flow rather than on its own.
Additionally, we label a control sample manually: it shows whether the share of automatic sorting is growing at the cost of incorrect routes.
The confidence threshold sets the boundary for automatic processing: typical topics — complaints about the work of management companies, payment questions, appointment scheduling — go to the relevant agency without an operator, while disputed wording is escalated to a human.
The threshold is configured separately for each channel, so the flow through Gosuslugi, paper correspondence, and email do not interfere with each other. As confirmed examples accumulate, the threshold rises, and the share of automatic processing grows without loss of accuracy.
Response time for the applicant is reduced at the stage where a queue used to build up: registration and routing to the destination happen at the moment of receipt rather than after manual sorting.
The share of repeat appeals works as feedback — if a person writes again about the same issue, the route was chosen incorrectly or the response did not resolve the matter; we analyze such chains separately and add them to training.
The load on operators stops depending linearly on peaks: sorting goes to the machine, while specialists focus on substantive responses and review of disputed cases.
The morning wave of appeals no longer grows along with the queue, and freed-up hands are used to reduce processing times for complex categories.
Processing modes: comparison of implementation options
The choice of processing scheme is dictated not by the model but by the load profile: how many appeals arrive per day, whether there are peaks, whether a response is needed within minutes or a batch export every few hours is sufficient. Below are four setups we have deployed, with their applicability and limitations.
| Option | When it fits | Limitations |
|---|---|---|
| Batch flow processing | Appeals accumulate; completeness of processing matters more than response speed | The response lags behind receipt by the export window; peak days are processed with delay |
| Streaming processing | Minute-level response is needed; appeals go to assignees immediately | Sensitive to failures: without retries and a queue of unprocessed items, appeals are lost |
| Hybrid processing scheme | Peaks and baseline differ by orders of magnitude; some appeals cannot wait | Harder to maintain: two branches, two sets of metrics, discrepancy monitoring required |
| Processing with operator involvement | Topics are new, training examples are scarce, and the cost of error is high | Staff load grows with the flow; quality depends on people rather than the system |
The key parameter of any scheme is the confidence threshold: below it, an appeal does not go into automatic routing but enters manual processing. It is this threshold, not the choice of setup, that determines the share of appeals processed without a human.
Almost always, it is worth starting with the batch setup: it provides labeled data and quality metrics on which the streaming or hybrid branch is then built.
Implementation stages: from flow audit to operation
Before training the model, we measure the flow itself: how many appeals arrive, through which channels, and how they are currently distributed among assignees.
At this stage, it is clear where manual sorting consumes time and where simple rules already work and just need to be extended.
- Appeal flow audit: we export history, calculate the shares of typical and atypical requests, and record the current processing time and the points where an appeal hangs without an assignee.
- Design of the class and route scheme: we agree on the list of categories, priorities, and which unit receives each topic. The scheme must match the actual organizational structure; otherwise the route will hit the absence of an assignee.
- Example labeling: we assemble a training sample, label disputed appeals manually, and define rules for borderline cases. Rare topics need enough examples; otherwise the model dumps them into popular classes.
- Confidence threshold tuning: the model outputs probabilities for each class, and the threshold decides whether an appeal goes automatically or to manual processing. Too low a threshold produces errors; too high a threshold returns all work to the operator.
- Model quality validation: on a holdout sample, we calculate accuracy by class, separately review false positives on rare topics, and assess robustness to typos, short appeals, and mixed-language appeals.
- Pilot on live flow: the system runs in recommendation mode — the model suggests a route, and the operator confirms or corrects it. We analyze discrepancies and retrain until the share of corrections stabilizes.
- Handover to operation: we provide the configuration, the procedure for labeling new examples, and drift monitoring — when the topic distribution changes, the model is retrained on operator corrections.
Delivery package: models, routing rules, procedures
The delivery package is not an archive of files but a working setup: it can be handed over to another team without reconstructing the logic from correspondence. Each artifact is versioned and has an owner — it is clear who changes what when new types of appeals appear.
- Classification model and its card. Where it was trained, what share of appeals it labels automatically, what the accuracy is on a holdout sample, and on which wordings confidence drops.
- Topic reference and rubricator. A hierarchy of categories with example wordings, update rules, and the procedure for approving new topics without rebuilding the entire system.
- Routing rules. A table “topic → unit → response time,” including branches for mixed appeals where the model sees two topics with similar confidence.
- Operator procedure. How to confirm or correct an automatic label, what to do when confidence is low, and where an appeal goes if the topic is not determined.
- Access policies. Roles and visibility boundaries: the operator sees the appeal, the analyst sees anonymized statistics; access log and rules for handling personal data.
- System documentation. Flow diagram, integration points with the record system, and the procedure for updating the model and rolling back to the previous version.
- Validation appeal set. Several hundred labeled examples that we run every update against to make sure that old routing branches are not broken.
Everything is delivered within the customer’s environment, with instructions for retraining and an analysis of typical failures — the team does not need to come back to us for every change to the rubricator.
Case study: unified contact center and topic routing
Initial situation
The client is a regional appeal intake operator: eight incoming channels, dozens of topics in the classifier, and manual sorting.
The operator read the text and chose the topic manually, so during peak weeks up to a third of appeals went to the wrong unit and came back marked “outside competence.” The topic processing time reached up to four business days — even before the appeal reached the assignee.
We assembled a single intake point — Unified Appeals Window: email, website forms, phone conversations, and chat correspondence are consolidated into one flow with common numbering and history.
The classification model was trained on the archive of appeals and labeling by the specialists themselves: it determines the topic, subtopic, and urgency flag.
Disputed cases with low confidence go to the operator — the rule was deliberately kept strict because the cost of error in social topics is higher than speed.
Result
After two months, the share of appeals processed automatically without operator correction reached 78%, and routing time decreased from two to four days to minutes.
Repeat redirections between units fell by a factor of three, and the share of responses that violated the regulatory deadline fell from 22% to 4%.
The model is retrained on operator corrections: every topic correction is a signal for the next iteration. We review the topic list based on accumulated statistics once a month — citizen wordings change faster than any fixed reference.
What to do if appeals are already sorted manually?
Rule-based processing breaks on the very first appeal written differently from what the template author expected: a synonym, a typo, two topics in one message, a quote from someone else’s text.
Each new case adds a rule, rules begin to conflict, and fixing one reduces accuracy on neighboring ones — maintenance grows faster than the number of appeals.
There is no need to turn off the rules overnight. The working scheme is a hybrid approach: a deterministic layer catches formal and rare categories where unambiguous wording matters, while a trainable model handles the variable part — topics, nested questions, implicit requests.
Features from the rules are fed into the model as input, so it inherits accumulated expertise rather than starting from scratch.
| Option | When it fits | Limitations |
|---|---|---|
| Rules only | 3–5 stable categories, formal features | Breaks on paraphrases; the cost of edits grows |
| Trainable model | Many categories, free-form wordings | Requires labeling; rare classes fail |
| Hybrid approach | Transition without quality loss at the start | Requires coordination of layers and priorities |
| Model plus operator review | High cost of error | Lower share of automatic decisions |
Migration to the model is staged: first it works alongside the rules, its decisions are compared with operators’ conclusions, discrepancies are analyzed and added to the labeled data.
Automatic routing is enabled for classes where accuracy stays above a given threshold; the rest goes to manual processing.
A pretrained language model, ruBERT, helps here — it already understands Russian syntax, and fine-tuning requires many times fewer labeled examples than training from scratch.
Technical concerns before starting the project
Concerns before starting are usually the same: will citizen appeal classification work on topics that were not in the sample, what to do about typos and mixed languages, who sees personal data, and how the result gets into record systems. We address each point specifically.
Rule changes deserve special mention: the rubricator and routing logic are stored in configuration rather than inside the model. Therefore, a new branch or a change of priorities is a procedural edit with a run on a set of past appeals, not retraining from scratch.
What if a new topic appears that was not in the sample?
An appeal with low confidence across all categories is not “glued” to a random class; it goes to the “undefined” branch and to an operator for processing.
Then the labeling pipeline takes over: disputed examples enter the queue, the rubricator and model are updated once per iteration — new topics become material for retraining rather than a routing failure.
What about typos, abbreviations, and text in different languages?
The text undergoes normalization: abbreviation expansion, correction of common typos, transliteration, and tagging of mixed fragments.
The language is detected at the input, and each has its own processing branch, so an appeal with insertions in another language or agency abbreviations does not lose its category.
How is personal data protection ensured?
Anonymization happens before classification: names, addresses, phone numbers, and document numbers are masked, and only text without identifiers goes to the model.
Access is role-based, every access to data is written to a log, and processing can run entirely within the customer’s environment.
How is the result transferred to record systems?
Transfer goes through an API or bus: the card receives the category, priority, and assignee. Retries and duplicate protection ensure that if the system is unavailable, the appeal is not lost and will arrive after recovery; discrepancies are visible in the exchange log.
Let's discuss the task and come back with an estimate and plan
The first step is a short pre-start audit: we look at where appeals come from, in what form, how many there are, and how they are labeled now.
Without this, any timeline estimate will be guesswork, and citizen appeal classification will run into the quality of the source data rather than the model.
To make the conversation concrete, it is useful to prepare:
- a list of intake channels and formats — letters, electronic forms, scans, call transcripts;
- an export of historical appeals and the decisions and routes already made on them;
- the agency’s current rubricator, if one exists, or its working draft;
- examples of disputed cases where an appeal fits several topics at once;
- flow volumes and peak periods when manual sorting stops coping;
- requirements for handling personal data and rules for access to it;
- handoff points: where the route is returned, and who confirms it and how;
- acceptance criteria — the target share of correctly routed appeals and the acceptable share of manual processing.
After the audit, we come back with an implementation plan: scope of work by stage, timeline estimate, areas of responsibility, and what is required from your side. Describe the task in free form — that is enough to start, and we will clarify the rest in conversation.
