Auto-generation of support replies: LLM and Agent Assist

Why automatic response generation breaks down on real requests

Automatic response generation confidently passes a demo on a dozen clean questions and falls apart on a live stream.

The reason is not the model, but what real requests look like: typos, screenshots instead of text, a quote from a previous conversation, and the actual question hidden in the middle of a paragraph.

The classifier sees “delivery,” although the question is about a partial refund, and inserts the wrong article.

The second bottleneck is the knowledge base. Connecting to Zendesk or a similar system does not help if it contains three versions of one instruction, the old one is marked current, and some conditions exist only in operator correspondence.

The model faithfully reproduces this disorder: the text is smooth but contradicts the current procedure.

Incorrect wording costs more than silence: the operator spends time checking and editing the draft, and the client receives an answer that has to be retracted.

Escalation breaks separately — if it is not defined at what confidence level and on which topics the system must stop and call a person, support automation takes it upon itself to answer complaints and payment questions in general terms.

The result is increased load on first-line operators. They rewrite the suggestions, stop trusting them, and disable them; and the less they are used, the less data for fine-tuning: the circle closes. We examine how this is addressed at the level of data, rules, and quality metrics.

What automatic response generation provides: speed, accuracy, workload

The first changes are visible at launch: the generator inserts a ready draft response for the operator, and first response time drops from hours to minutes — while the person still sees the text before sending.

At this stage the gain comes not from replacing the employee, but from removing the pause for searching for a template and restoring dialogue context.

Next, the system is tuned on request history: topics are labeled, typical scenarios are identified, and a confidence threshold is set below which the response goes to the operator with a note, rather than to the client.

From this point, the share of requests closed without human involvement grows — usually for stable topics (order status, access, plan, typical malfunctions).

After a few more weeks of operation, topic hit accuracy becomes visible. At first it is limited by the mixing of similar topics, so the taxonomy is cleaned up, and problematic pairs are separated with examples.

The share of repeat requests and clarifications drops: the client gets a substantive answer the first time, rather than a series of clarifying questions.

For the support line, the effect is expressed in redistribution of workload. Simple requests are closed automatically, operators handle non-typical cases rather than processing repetitive correspondence.

The peak of incoming flow no longer requires expanding the shift — the queue is smoothed by the instant first response.

Next we examine how this is addressed at the level of data and integrations.

Operating modes: operator suggestion, draft, full automation

The modes differ not by the “intelligence” of the model, but by how many decisions it makes on its own and when a person gets involved.

The control is the same: the confidence threshold together with routing rules determines whether the response goes to the operator for revision or straight to the client.

Operator suggestion — the assistant proposes wording, the operator reads and sends it themselves: the client sees nothing that has not passed through a person.

Draft for the operator — the ready response is already inserted into the window, and editing and confirmation remain; typing time is eliminated, control remains.

In full first-line automation, the model answers on its own, and anything below the confidence threshold or caught by an escalation rule is returned to a person. It is this return, not the answer itself, that determines whether the mode will survive peak-season load.

Modes are not chosen once and for all: they are enabled by request groups and switched without rebuilding the assistant.

The reference points are the share of typical questions, the acceptable error class, and the fallback requirement; then the decision is refined based on review of real dialogues.

Implementation stages: request audit, routing scheme, validation

Implementing automatic response generation is not connecting a model, but setting up a loop where generation does not spoil the customer experience. The order of work is structured so that each step relies on data from the previous one.

1. Request audit. We export history for several months and label it by topics, channels, and recurrence. The output is the share of requests that are closed from the knowledge base and topics where we do not yet apply generation.
2. Request routing scheme. We determine where the suggestion appears: a draft for the operator, an answer without a person, or escalation. We define the confidence threshold at which the system stays silent and the rules for returning the dialogue to a specialist.
3. Source preparation. We collect regulations and knowledge base articles, remove outdated versions and contradictions. Without this step, generation confidently reproduces the errors already in the base.
4. Pilot on one line. We enable automatic generation on one channel or a narrow group of topics. The operator confirms or edits each response — this is how we collect labels and see real errors.
5. Control sample. We manually review part of the dialogues: share of edits, share of escalations, cases of confident incorrect answers. These metrics show whether quality is growing or just volume.
6. Handover to operation. We fix the knowledge base update procedure, the person responsible for edits, alerts for a drop in the share of accepted responses, and the shutdown procedure. Then we expand the loop to the remaining lines.

What we hand over: configurations, procedures, documentation

We hand over the solution not as a “black box” that only the author can maintain: the client’s team retains a full set of artifacts with which automatic response generation can be developed on their own. Below is what exactly is fixed at handover and in what form it continues to work.

• Scenario configurations — which requests are generated fully, where the operator is offered a draft, and where the response is assembled from a template; confidence thresholds and topic filters are moved into editable parameters that can be changed without modifying code.
• Escalation rules — the list of topics that the model does not close on its own (payments, personal data, complaints, legal wording), conditions for transferring the dialogue to an operator or shift supervisor, and the procedure for external service unavailability.
• Operator procedures — how to accept, edit, and reject a generated response, in which cases manual review is mandatory, how to mark errors: these marks become training material for the next version.
• Knowledge base instructions — requirements for the structure of articles and fragments, rules for formatting headings and synonyms, and the update procedure under which a new answer is found by search without retraining the model.
• Solution documentation — the scheme of request passage from channel to sending, integration points with the ticketing system and reference guide, and a description of all settings and dependencies between them.
• Evaluation dialogue set — fixed requests with reference answers and “accepted / rejected” labels, used to check quality after every change to the knowledge base or rules.
• Operation and rollback instructions — how to switch to a previous configuration version or to fully manual mode if response quality drops.

After handover, you can change scenarios and rules yourselves, and involve us selectively — for new topics or a noticeable increase in request volume.

Case study: reduced request handling time in the support line

Before implementation, the support line handled requests manually: the operator searched for the article in the knowledge base, assembled the response, and decided when to involve the second line. Typical requests — access, ticket statuses, return conditions — took half a shift. Median first response time stayed around 40 minutes and doubled during peak hours.

The pilot was launched on one line and one shift so the comparison would be correct: the same workload, the same categories, the same set of operators.

The assistant was connected to the ticketing system Zendesk and the knowledge base: it selects articles, prepares a draft response, and the operator confirms or edits it.

For a closed set of categories, autonomous responses were enabled with an escalation rule: if the answer is not confirmed by a source or the question is sensitive, the request immediately goes to a person.

Metrics before and after

What sustained the result

Once a week we review operator edits: if a draft is regularly rewritten, it means the knowledge base lacks an article or wording. We add materials and rebuild escalation rules by category — two months after launch, response time had not increased despite a larger request flow.

How does operator suggestion differ from full automation?

Operator suggestion and full automation are different modes in terms of the cost of error. In the first, the model forms a draft response and surfaces relevant knowledge base fragments, but a person sends it: incorrect wording costs the operator a few seconds to edit.

In the second, the response goes to the client without review, so any inaccuracy becomes a public fact — and it must be handled as an incident, not as a draft.

This leads to different data requirements. For a suggestion, search over documentation with fragment ranking is enough: the operator filters out outdated items and fills in context themselves.

For automation, a reference is needed — labeled “request → correct answer” pairs, intent coverage with frequency, a separate set for checking for fabricated facts, and regression runs before every knowledge base update.

Quality control in automatic mode is built on routing, not on trusting the model. A request goes to a person if confidence is below the set threshold, the text has no source support, the client writes in a negative tone, or the topic is not on the allowed list.

Launch is canary-style: first a few percent of traffic, then expansion as the share of edits and repeat requests remains within normal limits.

Choosing a mode is not an “either-or,” but a sequence. They usually start with suggestions: they deliver an effect immediately and at the same time collect a corpus of edits that later becomes a training set.

Then they automate narrow classes of requests — high-frequency ones with a short unambiguous answer and a known information lifespan. The rest remains on suggestion, and switching between modes is done by configuration, not by rewriting logic.

Technical concerns: hallucinations, data leaks, labeling

Before launch, the same objections almost always come up: the model will invent something not in the instructions, correspondence will leak outside, and labeling will turn out to be subjective.

They determine whether automatic response generation will relieve the operator or create new incidents. We examine how we address each of them.

How data will not leak outside

Requests are processed in an agreed environment, personal data is masked before reaching the model, and logs store only metadata: request type, source, relevance score. Fine-tuning of external models on customer data is disabled.

Labeling is subjective, quality cannot be measured

Labeling is performed by several specialists, disputed cases are reviewed by an arbiter, and agreement between them is a separate metric: it shows how unambiguous the instruction itself is.

From labeled examples, we assemble a golden set and run it on every change to the prompt or knowledge base.

The knowledge base has contradictory articles

First, an audit: duplicates, outdated instructions, articles without an owner. If the sources still conflict, the system does not choose silently — it marks the conflict and passes the request to an operator with both articles indicated.

Let's discuss the task and come back with an estimate and work plan

So that the scope estimate is based on data rather than assumptions, a specific minimum of source data is needed. Below is what we request at the start of discussing the task and what we send in response.

• An export of anonymized requests: 300–500 dialogues across 3–5 frequent topics. They show customer wording, operator response length, and the share of typical questions — that is, a realistic ceiling for automatic generation.
• The knowledge base in its original form: articles, regulations, ready templates. We assess completeness, duplicates, and contradictions — this determines the amount of preparation before training.
• A person responsible for the knowledge base on the client side, 2–3 hours per week: they approve facts, topic priority, and the rules for what may be shown to the client.
• The list of systems where the response draft is returned: operator workstation, request channel, record-keeping system, and the method of exchange between them.
• Constraints: personal data, excluded topics, requirements for storing correspondence. We fix them before development, not after the first checks.
• Acceptance metrics: share of responses sent without edits, request handling time, share of rejected suggestions.

In response, we send a scope estimate broken down by blocks, an implementation plan by stages with acceptance points, and a draft composition of source data with notes on what is missing. We suggest testing on your own requests, not on demo examples.

Describe the task in free form — we will come back with an estimate and work plan.