AI request processing system: where the flow of requests breaks
An AI request processing system rarely breaks on the model itself — it breaks at the junctions.
At peak load, the queue of requests grows faster than operators can handle it, and some requests settle in the channel buffer: without an owner, without a deadline, without a trace in the accounting system. For the client, such a request is as good as lost — they wait for a response that will never come.
The second bottleneck in processing is duplicates.
The same request comes from email, messenger, and a website form; without an end-to-end identifier and idempotent processing, the system creates two or three tasks, and the person receives either two different responses or none at all. Redelivery of a message when an external service fails adds another round to this.
Next is routing. A classifier with low confidence sends the request to the wrong queue, the request is tossed between teams, and the first response time grows by an order of magnitude. The response arrives at the right place, but not to the point.
And manual handling of similar messages: operators spend a shift on "where is my order" and "how to extend", while complex requests wait in queue.
We'll look at how this is addressed — where to put queue depth control, duplicate merging, confidence threshold, and handover to a human.
What changes after implementation: speed, auto-reply share, failures
Implementation is justified when the effect is measured from a working system, not from a demo.
We take measurements twice: on historical requests before launch and on the live flow a few weeks after launch — the same metrics and observation horizon, so the numbers are comparable.
We collect data in Grafana: for each response, the source is visible — model, knowledge base, or operator.
Speed and quality of response
First response time drops from hours to seconds: a typical question is handled immediately, a composite one — while data is collected from accounting systems.
We look not at the average but at the tail of the distribution: it is what determines whether the client will wait for a response or leave.
- First response time — median and 95th percentile, separately during working hours and night windows.
- Share of automatic responses — requests closed without an operator and confirmed by the client, not just "response sent".
- Quality — share of cases without repeated clarification and reopening of the request.
Cost and resilience
The cost of processing a request decreases not through cuts but through redistribution: routine types go to automation, operators take on complex cases. We test the same loop at peaks — we raise the load up to a threefold spike.
- Resilience to peaks — first response time remains in the same corridor, the queue does not grow like an avalanche, failure degrades into "I'll respond later" rather than silence.
- Eliminated class of failures — lost requests, responses from an outdated knowledge base, duplicates on repeat contact.
We include these metrics in acceptance: if the response time tail and the auto-reply share do not move, the task is not solved.
Operating modes: full automation, operator assistant, hybrid
A request processing system can operate in three modes — from fully autonomous handling to suggestions for a live operator.
The choice is determined by the type of request, the cost of an error, and whether it is necessary to show the client the basis on which the decision was made.
IT service management practices ITIL are convenient here as a general framework: routing is based on the type of request, not on the channel from which it came.
| Mode | When suitable | Limitations |
|---|---|---|
| Full automation | Repeatable requests with an unambiguous answer: ticket status, service conditions, standard documents | A non-standard case still goes to a human; confidence control of the response is needed |
| Operator assistant | Complex and emotional requests where a human makes the decision | Speed is limited by the operator; wording is less uniform |
| Hybrid mode | Mixed flow: the system conducts the dialogue, and when confidence is low, connects a human | Requires clear routing and context handover without loss |
How to choose a mode for the type of requests
We start with analyzing the flow: which topics repeat, where the answer is unambiguous, and where access to internal data is needed. If the share of unambiguous scenarios is high, autonomous handling closes them without an operator and relieves the first line.
Where modes hit limitations
Full automation breaks on conflicting and non-standard requests — such a dialogue the system must be able to hand over to a human along with the history and already collected data.
An operator assistant does not speed up work if the suggestion takes longer to verify than to find the answer manually.
Implementation stages: from request audit to handover to operation
Implementation of an AI request processing system begins not with model configuration, but with an audit of what is already happening in support.
We export the history for several months and calculate the shares of topics, channels, and repeat contacts: from this data it is clear which requests are closed with a standard answer, and which necessarily go to a human. Then the work proceeds in stages, and each ends with an artifact that can be verified.
Data and logic preparation
- Request audit. We label topics in the history and calculate their shares — we get the boundary of what can be automated without loss of quality.
- Routing scheme. We fix which types of requests go to automatic response, which go straight to a human, and by what criteria the system hands over the dialogue.
- Example labeling. For each scenario, we collect a set of real dialogues with correct answers and topic labels — this is the benchmark for configuration and checks.
- Scenario and rule configuration. We describe dialogue branches, response templates, clarifying questions, and the procedure for requesting data from accounting systems.
Acceptance checks and handover
- Testing on the live flow. We enable the system in shadow mode: it generates responses, but they are sent only after verification by an operator.
- Acceptance checks. On a holdout set of dialogues, we calculate the share of correct answers, correct handovers to a human, and requests closed without operator involvement.
- Handover to operation. We provide documentation, access, a regulation for editing scenarios, and training for the support line — further edits are made without us.
What we hand over: configurations, schemes, and regulations
After launch, the system remains with you not as a black box, but as a set of artifacts that your team can read, modify, and pass on.
We deliberately do not hide the request processing logic in the contractor's internal settings — otherwise any scenario edit turns into a separate project with waiting.
- Scenario configuration: rules for parsing the request text, response templates, confidence thresholds — in a repository under version control.
- Routing scheme: how a request is distributed by topics, queues, and operators, where automatic response is enabled, and where handover to a human is mandatory.
- Access policies: roles, visibility scopes of client data, request logging — who sees what.
- Escalation regulation: conditions and priorities for handing over the dialogue to an operator, response waiting time, shift responsibilities.
- Instructions for operators: how to take over a dialogue, correct a response, mark a handling as erroneous — with worked examples.
- Documentation: description of integrations, entity dictionary, list of checks during launch and rollback.
Post-launch support
The first weeks of support are tuning on the live flow: we look at the share of requests that the system wrongly routed to a human and adjust thresholds.
Then management transfers to your team — we train administrators, fix the procedure for releasing changes in scenarios, and remain in touch for integration questions.
Case study: relieved the first line and reduced response time
The client is a chain of service centers, up to 1,200 requests per day from four channels: website chat, messengers, email, and phone.
Everything flowed into «Единое окно» (Single Window), and distribution among operators was manual: priority was effectively determined by whoever managed to take a request from the queue first.
What it was before
During peak hours, the first response stretched to 40–50 minutes, about 8% of requests remained unanswered — the client left without waiting.
An operator handled no more than 45 dialogues per shift, while two-thirds of requests were standard: order status, warranty conditions, appointment rescheduling.
Hiring and training new operators consumed more time than they added in throughput, and the evening queue of unassigned requests became the norm.
What decisions were made
The flow was restructured: at the input, a classifier determines the topic and intent; standard requests go to scenarios with data verification in internal systems — order status, warranty period, available appointment slots.
The response is generated automatically and returned to the same channel; text is closed in chat and messengers, and speech synthesis is added in telephony.
If the classifier's confidence is below the threshold or there is negativity in the dialogue, the request goes to an operator with the chat history and a response suggestion, rather than being lost in the queue.
We control quality by labeling disputed dialogues and regular retraining on real wording — the model gradually covers phrasings that were not in the training set.
After several months of operation at full load, 60% of requests are closed without human involvement, the first response comes within a minute, and the load on first-line operators has fallen by about three times.
The share of unanswered requests decreased from 8% to 0.4%, the evening queue disappeared, and freed-up operators switched to complex cases and returns, where a human is really needed.
How request classification differs from routing
Classification answers the question "what is the request about", routing — "who handles it". The first relies on labeling and topic taxonomy, the second — on regulations: areas of responsibility, response times, priorities.
As long as both layers live in one rule, departmental changes drag along model retraining, and a new topic breaks the distribution queue.
| Criterion | Classification | Routing |
|---|---|---|
| What it determines | topic and type of request | executor, queue, priority |
| What it is based on | labeling and topic taxonomy | regulations and areas of responsibility |
| How often it changes | rarely — when new topics appear | often — when teams change |
| Quality metric | accuracy on holdout set, share of ambiguous | share of requests closed without redirection |
The confidence threshold is the boundary not between "right" and "wrong", but between an automatic decision and handover to a human. Raw model scores are not calibrated, so the same threshold on different topics means different things.
We calibrate them on a holdout set and maintain two boundaries: above the upper one, the request goes along the route immediately; between the boundaries — a clarifying question; below — an operator.
Ambiguous requests are a normal class, not a failure: a letter may concern both payment and return; sometimes the topic is clear, but there is no suitable queue. For them, we keep two or three hypotheses with scores and a clarifying question.
Separate layers give the main thing: the taxonomy changes once every few months, while distribution rules change with every reorganization, and route edits do not require retraining the classifier.
Technical doubts: data, languages, escalation, and control
Doubts at this stage are predictable: mixed languages within one dialogue, truncated history from a previous system, responsibility for an incorrect auto-reply. We explain how each is addressed by processing logic, not promises.
Client writes in two languages in one dialogue — will the response be unclear?
The language is determined per utterance, not for the whole dialogue, so the response goes in the language of the current message.
If the utterance is mixed or the language is not confidently recognized, the system does not guess: it asks a clarifying question or hands the dialogue to an operator.
Request history was exported incompletely, and the system is already running
Missing data is not guessed: a request without context is processed as new, and if background is needed for a decision, the dialogue goes to an operator with a note.
We pull history via export from the previous system and verify by client identifier and channel, while the old system continues to respond until the new one passes testing on the live flow.
Who is responsible for an incorrect auto-reply?
An auto-reply is sent only when confidence is above a set threshold; below — escalation to an operator. For each response, the data source, model version, and text are saved, so incident analysis takes minutes, not days.
How is quality and retraining controlled after launch?
Operator edits become labeled examples; we check labeling quality selectively, and disputed cases go to arbitration. Retraining is regular, but only after running on a holdout set — so that improvements do not break what already worked.
Let's discuss your task and come back with an estimate and plan
So that the estimate is based on your real environment, not on general assumptions, it is enough to prepare for the conversation the inputs on how request processing is currently set up.
A ready-made technical specification is not required — during the task discussion, we analyze the current flow and what should change in it.
- Request channels and their shares: email, website chat, messengers, calls with transcription — which of these goes into a common queue and which is processed separately.
- Export of history for several months: in what form requests are stored, whether there is category labeling and saved operator decisions.
- List of escalations: which topics necessarily go to a human, by what criterion the rule triggers, and what is considered critical.
- Access and data: test environment, accounts for integrations, rights to read history, presence of personal data and requirements for their storage.
- Integration points: where the system should return the result — to a ticket system, CRM, or knowledge base, and in what format.
- Target metrics: share of requests closed without an operator, first response time, routing accuracy — we will use them to measure the result.
- Staged constraints: what must work in the first loop, and what can be postponed without losing meaning.
From this set, an estimate and work plan are assembled: stages, composition of integrations, environment requirements, and the procedure for quality checks on your request history.
We come back with a document that records the scope, control points, and risks — we discuss it and move on to preparation for launch.