Agent Assist in production: where the operator prompt breaks
A real-time operator prompt breaks not on the model but on the state of the dialogue.
Speech recognition lags behind the speaker, past utterances are already stale: the customer has moved on from the original question, but the engine still suggests an answer to the previous topic.
The operator sees the card several seconds after it would have been useful and either ignores it or responds off-topic — the quality of the operator's responses drops even though Agent Assist is formally running.
The second class of failures is irrelevant prompts that duplicate what has been said. If search runs on the last utterance without accounting for the entire request history, the output repeats what the operator has already said or generic wording from the knowledge base.
You need a filter for what has already been voiced, freshness-based prioritization, and an explicit topic state — otherwise the panel stops being read.
The third and most expensive failure is loss of request context during transfer to the second line.
A prompt session is usually tied to the channel and operator rather than to the request itself: the new line receives an empty history, and the customer retells the problem from scratch.
This is fixed by storing context at the request level and passing it along with the transfer — with a brief summary, status, and already collected data.
We check such systems by metrics: the share of prompts the operator actually inserted into the response, time to first relevant prompt, and the number of times the customer repeats the same information after transfer. Next, we break down how this is addressed.
Effect on the line: response speed, accuracy, workload
The effect of prompts is measured not by the feeling that it has become more convenient, but by numbers taken from the line before enablement and 4-6 weeks after.
That is why we record baseline indicators at the start and compare them against the same reports: Agent Assist runs on top of the current telephony and customer database, and the prompt reaches the operator in the same window where they conduct the conversation. The checklist below is what we use to calculate the effect.
- Request handling time — the wording of the answer and a link to an article appear in the third to fifth second of the conversation, not after the call at the end of the shift. In typical scenarios, handling time decreases by 15-25%.
- Share of independently resolved answers — some questions stop going to the second line because the prompt closes a knowledge gap during the conversation. The share of requests closed on the first line grows from 55-60 to 70-75%.
- Mentor workload — mentors stop being a live reference desk and handle only complex cases. The knowledge base with prompts handles a newcomer's typical questions, freeing up several hours a day for the mentor.
- Newcomer onboarding — time to target productivity accelerates by 1.5-2x: prompts work on real requests from day one, not after two weeks sitting next to a senior operator.
- Operator turnover — the routine of searching and the fear of making a mistake go away; the effect is visible in the share of employees who leave in the first three months and in returns to the line after a break.
- Support line productivity — growth in the share of independently resolved answers yields more handled requests with the same headcount. We separately monitor the share of prompts accepted by the operator — if it falls, the accuracy of the output is corrected rather than blamed on people.
Measurements remain in the metrics dashboard after launch: the team sees where prompts work and where the knowledge base needs improvement.
Agent Assist modes: comparing implementation options
Operator prompts differ not so much by model as by the rigidity of the logic: in some cases it is a pre-described scenario, in others an answer selected based on the meaning of the request.
The mode determines the amount of data preparation, when the prompt appears in the dialogue, and who is responsible for an error — the author of the rules or the model.
| Mode | When it is suitable | Limitations |
|---|---|---|
| Rule-based prompts | A stable flow of typical requests, regulated steps, regulatory requirements | Breaks on non-standard wording; the rule set is maintained manually, otherwise it diverges from real dialogues |
| Answer selection based on similar requests | A large dialogue history, a wide range of topics | Quality depends on the cleanliness of the history: outdated and erroneous answers appear in the output alongside working ones |
| Prompt based on the meaning of the request | Different wordings, multiple channels, a long tail of topics | Requires a source link under each answer and a confidence threshold, otherwise the operator receives plausible but incorrect text |
| Automatic actions in the dialogue | Repeatable operations: filling fields, sending a link, transferring to the right queue | Require permissions, a log, and rollback — an error here changes data in the system of record rather than merely advising |
The history for the second mode is taken from a service system, whether Salesforce Service Cloud or a similar platform, so the modes are almost never enabled separately.
Rules enforce mandatory steps and guide a newcomer, semantic selection covers the long tail, and automatic actions are connected last — after the history has been labeled and rollback has been configured.
The degree of line automation grows gradually: first the prompt is visible only to the operator, then some actions move to automatic execution with mandatory confirmation. This phased enablement shows where a mode makes mistakes before the error reaches the customer.
Implementation step by step: from request audit to shift handover
Development starts not with the model but with an analysis of how the shift actually answers customers.
A request audit shows what share of questions repeat, where the operator spends the longest searching for an answer, and which topics are covered by outdated or contradictory articles.
- Request and knowledge base audit. We analyze dialogue history, identify recurring scenarios, and compare them with regulations: where an answer exists but takes long to find, and where it does not exist at all.
- Prompt flow diagram. We determine at what point in the dialogue the prompt appears, what the operator sees at low confidence, and how they reject it without losing the context of the conversation.
- Knowledge source setup. We split articles and regulations into semantic fragments, set validity and owners, and remove duplicates — otherwise the system returns similar but different answers.
- Pilot launch. We enable prompts on one shift and a narrow set of topics, limit the load, and collect statistics without rebuilding the entire workflow.
- Prompt quality acceptance. On labeled real dialogues, we calculate the share of accepted prompts, the correctness of article links, and cases where the operator corrected the system.
- Shift training. We show how to read a prompt and how to mark an incorrect one: operator marks become the main channel for improvement.
- Handover to operation. A procedure for updating the knowledge base, an owner responsible for prompt quality, and a clear rollback to working without prompts.
What remains with you: configurations, tagging, procedures
We do not leave the system as a black box: everything that determines prompt behavior is handed over in your environment and continues to work after the project ends. You receive not a set of scripts but a reproducible configuration that your team can edit and extend.
- Working prompt configurations — trigger rules, confidence thresholds, wording templates, and response tone. Edits take minutes and do not require a rebuild: you switch a rule, test it on a group of operators, and enable it for everyone.
- Request flow diagram — which events come from telephony, chat, and the system of record, when the prompt is generated, and what is returned to the operator. The diagram shows failure points and places where the answer can be enriched.
- Knowledge base tagging — article structure, tags, splitting into semantic fragments, and mapping to request types. This is what determines prompt quality: unlabeled material causes false positives.
- Operator access rights — roles, visibility of knowledge base sections, restrictions on sensitive data, and a log of prompt accesses. The audit shows who saw what.
- Knowledge update procedure — who makes changes, how often the search index is rebuilt, and how it is verified that an outdated article is no longer shown to the operator.
- Support documentation — where configurations are stored, how to roll back to the previous rule set, which metrics to monitor, and which procedure to follow when there is a deviation.
- Quality metrics — the share of accepted prompts, answer preparation time, and triggers on irrelevant queries.
After handover, you do not need to come back to us for every edit to wording or a threshold — the rules and access rights remain in your hands.
Case study: how response time and transfers to the second line were reduced
The pilot was launched on one contact center queue — 48 operators, three high-volume request topics, and an overloaded second line.
Before launch, the average request handling time was around 7:40, 62% of requests were closed on the first line, and repeat contacts on the same topic reached 14%.
Baseline
The operator searched for an answer in the knowledge base, internal chat, and with colleagues: time to the first substantive answer was 95 seconds, and every third complex case went to the second line.
The second line acted as a buffer — engineers handled typical questions that the operator could have closed themselves, and the ticket queue in Zendesk grew faster than the headcount.
What we changed
We built a vector index over the knowledge base and labeled closed dialogues — the prompt reaches the operator together with a source fragment.
The confidence threshold was set so that a questionable option was not shown at all, and escalation rules were rewritten: only disputes over contract terms, payment gateway failures, and repeat contacts after an unsuccessful resolution went to the second line.
Metric dynamics
Over six weeks of the pilot, handling time decreased to 5:10, and time to first response decreased to 40 seconds.
The share of requests closed on the first line grew from 62% to 81%, transfers to the second line decreased by 43%, and repeat contacts on the same topic went from 14% to 6%.
Prompt completeness, that is, the share of cases where the suggested answer matched the final resolution, rose from 71% to 88%; discrepancies were reviewed with training specialists and the index was additionally trained on them.
How does an operator prompt differ from a customer chatbot?
The difference is who makes the decision and who bears the risk of error. An operator prompt works on top of a live dialogue: the system pulls in knowledge base articles, request history, and the probable next step, but a human sends the answer.
A customer chatbot is a standalone interface: the machine makes the decision, and the operator gets involved only on escalation.
Hence the different automation boundary. A prompt only needs to be useful: the operator will ignore an irrelevant fragment, and the dialogue will not fall apart.
A bot's error is immediately visible to the customer, so confidence thresholds, an explicit scenario for returning to a live line, and test coverage of all conversation branches are needed.
The requirement for the customer also differs. Customer self-service works on typical requests where a person is ready to interact with a machine: order status, changing data, a standard calculation.
Where the solution is non-standard or the cost of error is high, the machine loses to a live conversation, while a prompt, conversely, strengthens the operator.
That is why the metrics are different too. A prompt reduces handling time and the time for a newcomer to reach the norm, and evens out quality between shifts, but does not reduce the number of operators.
A bot removes a share of typical requests from the first line — first-line automation is visible exactly here.
In practice, the tools complement each other: the bot closes simple cases and passes the rest to the operator, while the prompt guides them through a complex conversation.
We separate them by request type and customer readiness — this determines where automation is appropriate and where a human is needed.
Limitations and FAQs: where prompts make mistakes
Operator prompts are a probabilistic tool: they rely on speech recognition and knowledge base search, not hard rules. Therefore, the important question is not whether it works or not, but how the system behaves under uncertainty.
Below the confidence threshold, we show nothing: the operator sees an empty field, not questionable advice. Let us address other technical doubts that come up before launch.
How accurate are the prompts?
Accuracy is not a single number but a threshold setting. We output 1–3 options instead of a long list and raise the threshold where false advice is more costly than a miss.
In the pilot, we calculate the share of accepted and false prompts, and shift the threshold based on this data.
What to do about speech recognition errors?
On a noisy line, names, SKUs, and numbers are distorted. We extract key entities in a separate layer using a dictionary and ask the operator to confirm them, rather than relying only on the general recognition stream.
How to handle languages, accents, and dialects?
A separate recognition model is selected for the language and accent. We test it on recordings from your line — terms, service names, insertions in another language — and supplement the industry dictionary before launch.
How does the knowledge base not become outdated?
Each prompt references an article and its date, not the model's "memory." Articles have an owner and a validity period; expired ones are removed from the output — the operator does not quote a discontinued promotion to the customer.
Why do operators stop trusting prompts?
Trust breaks when advice is intrusive or wrong twice in a row. The prompt does not block the operator's work, there is a "not helpful" mark, and the first weeks run in observation mode and do not affect their metrics.
Let's discuss the task: we will review your request flow and return with a plan
Agent Assist is evaluated on a live request flow, not on demo dialogues — that is why the first review uses your data.
For an assessment, we need an export of dialogues from several weeks, a list of systems the operator moves between, and the rules by which senior shift members currently provide prompts. This is enough to understand where a prompt saves time and where it adds noise to the work window.
- Initial process review. We look at the request flow as a whole and find places where the operator loses time: searching the knowledge base, repeated clarifications, and transfers between lines.
- Task assessment. We return with a list of scenarios, priorities, and a list of data that needs to be prepared on your side.
- Work plan. A document with stages, checkpoints, and criteria by which each stage shows that the prompt is working.
- Alignment. We fix the scope of the first stage, the order of access to systems, and the role of your specialists.
- Launch. We connect the request source, assemble prompts on dialogue history, and present them to the operator in a test shift.
- Post-launch support. We monitor prompt quality on the live flow, update them on new requests, and investigate failures.
Describe your task — we will return with an assessment, a work plan, and a first stage that can be launched without stopping the current line.