Development of an AI system for contact center quality assurance (QA)

AI service quality assessment: where manual control breaks down

With selective call listening, a team lead can physically review only a few dialogues per shift — when the flow is hundreds of contacts, coverage drops to fractions of a percent.

The rest remains unassessed, even though that is exactly where script violations, missed objections, and polite refusals instead of resolving the issue accumulate.

The score also depends on the reviewer: one deducts a point for a pause, another does not notice rudeness.

The discrepancy between two reviews of the same conversation easily reaches a third of the scale — and the final score affects the review and the bonus, so the operator disputes not the work but the reviewer’s mood.

Missed violations in dialogues almost always surface not from review but from a customer complaint, and they have to be dealt with after the fact, when the customer has already left.

On top of that comes the load on the team lead: they simultaneously close the shift, handle escalations, and manually fill out evaluation forms, leaving no time for analytics.

Automated service quality assessment removes exactly this bottleneck.

Speech recognition with speaker separation turns the conversation into annotated text, and checklist criteria are calculated from it — greeting, needs identification, compliance with the script, correct closing.

The reviewer receives not a flood of calls but a sample with deviations and disputed points, while coverage grows from a fraction of a percent to the entire sample.

Let's break down how this is handled.

What changes after implementing automated control

Complete review of dialogues

Manual listening physically covers a few percent of contacts — everything else remains unassessed. Automated analysis processes the entire flow: voice, chat and messenger correspondence. Every dialogue receives a score, not just the one that happened to be sampled.

Unified criteria for evaluating a conversation

The rubric is described formally: what counts as a greeting, where the scripted part is violated, at what point the operator lost the initiative. The same fragment receives the same score regardless of who reviews it. The spread between reviewers shrinks from one and a half to two points to tenths.

Time to violation detection

A deviation is visible within an hour after the conversation, not in the results of a weekly summary. The supervisor receives a notification with the recording fragment and a mark on the timeline while intervention is still possible. Quality work stops being an after-the-fact review.

Reduction in the share of disputed scores

Each score is supported by a quote and a link to its place in the dialogue, so it is harder to dispute. The share of returns for reconsideration falls — the reviewer spends time on real conflicts rather than re-review. The operator sees not "score reduced" but a specific fragment of the conversation.

Three implementation options: cloud, in-house model, hybrid

The option is chosen based on three parameters: how many hours of conversations arrive per month, how specific the vocabulary is — industry terms, product names, operator names — and where the recordings are required to reside.

A ready-made speech recognition model provides the first transcriptions in the launch week, an in-house model trained on your own data wins on a narrow vocabulary, and a hybrid recording processing option splits the flow by complexity and separates routine and disputed contacts.

A common point for all options: storing recordings on your side remains with the customer — audio does not leave their perimeter; only texts, metrics, and call aggregates go outside.

We compare on a single set of recordings: we run it through all three options and look at the share of correctly recognized utterances, robustness to background noise, and the amount of manual correction left for the quality control operator.

Implementation stages: from recording audit to handover

The order of work is structured so that the solution is tested on real data before it enters the daily work of the quality control department. Each stage ends with an artifact that allows a decision: continue, change the rubric, or stop.

1. Conversation recording audit. We export a dataset for the period, review channels, dialogue length, share of silence and speech overlap, and sound quality. The output is an understanding of what percentage of recordings are suitable for assessment at all and where preprocessing is needed.
2. Dialogue processing scheme. We lock down the flow: where the recording comes from, at what step speech is recognized, how the dialogue is split by speakers, where the transcript is stored, and how scores are tied to it. The scheme is agreed before configuration begins — otherwise the integration will have to be redone.
3. Rubric. We translate the quality control department's requirements into verifiable features: criterion wording, weight, behavior in ambiguous dialogues. Disputed criteria are taken to a calibration session with reviewers.
4. Configuration and calibration. We run the model and rules on the annotated set and analyze systematic biases — for example, underscoring on short dialogues.
5. Comparison of automated and manual scores. We compare on a sample where reviewer conclusions already exist and calculate discrepancies for each criterion separately. The target benchmark is agreement on the final score and no failures on critical criteria.
6. Pilot over a limited period. We enable it on one line or shift for several weeks: all dialogues receive a score instead of sampled percentages, while reviewers continue to check manually and compare results.
7. Team training and handover. We show how to review disputed scores, change criterion weights, and add new rules without us. We hand over documentation, access, and the rubric update procedure.

Rubric, procedures, and access: what we hand over to the team

We hand over not only the configured system but also everything the team will need to manage it further without us: criterion descriptions, data flow diagrams, access rules, and the procedure for disputed decisions.

The package is designed so that a new analyst or supervisor can figure it out from the document rather than from correspondence with the developer.

• Dialogue evaluation rubric — criteria with weights, scales, and trigger thresholds, divided into mandatory and contextual: script violation, tone, completeness of answer, correctness of escalation.
• Integration scheme with the recording system — where we pull audio and metadata from, how we match them to the dialogue, and what happens when a recording is delayed or a channel is missed.
• Recording access policies — roles of operator, supervisor, and auditor, retention periods, anonymization, and the procedure for export and deletion on request.
• Score review procedure — who disputes an automated score and within what period, how the decision is recorded, and how corrections return to criterion configuration.
• Reference dialogue sample with manual annotation — the basis for checking accuracy after any change to the rubric.
• Team instructions — how to add a criterion, change a weight, check the result on a control sample, and roll back an unsuccessful change.
• Rubric version log — we record what changed and when so that period comparisons remain correct.

Describe the task — we will come back with an estimate and scope of work for your recording system.

Case study: how the share of missed violations was reduced

A contact center with several dozen operators kept quality control on manual sampling: the team lead could listen to 6–8 calls per employee per month, that is, 2–3% of the flow.

The remaining conversations were not assessed, and systemic violations — interrupting the customer, lack of confirmation of the solution, incomplete identification — surfaced only in complaints, when it was too late to correct them.

What was changed

Speech assessment was enabled for the entire flow: the recording is transcribed, utterances are labeled by role, and rules check wording, pauses, and the order of mandatory steps.

Recognition runs on <cite>Whisper</cite> — it is robust to a telephone channel with noise and compression, and recordings do not leave the customer's perimeter.

What the results were

The share of reviewed dialogues grew from 3% to full coverage — every conversation receives a score, not a random sample.

The spread of scores between reviewers decreased by about five times: rules are applied to text identically, and discrepancies remain only at meaningful disputed points, not due to listening quality.

Reviewing a disputed case took minutes instead of half an hour: the reviewer opens the transcript with time stamps and looks at a specific fragment instead of re-listening to the entire recording.

The results of implementing speech assessment, based on re-checking the control sample — the share of missed violations is about four times lower, and they are recorded on the same day, not through a customer complaint.

What to do if conversation recordings are already being made?

If recordings of conversations are already accumulating, there is no need to build from scratch: we connect to the existing archive and to the flow of new calls.

We do not touch the telephony system — read access to recordings and the operator directory is enough; call routing remains exactly as it is now. Therefore, the first launch runs on historical data, without waiting for a new sample to accumulate.

The first step is an audit of the audio recording quality of the dialogue. We check whether operator and customer tracks are recorded separately, whether there is level overload, dropouts, long pauses, and queue music.

We assess speech intelligibility using the industry MOS scale: if the recording is "muffled" or voices are merged into one channel, no model will restore the meaning of the utterances.

The output is an archive map: what share of dialogues is suitable for automated service quality assessment, and where normalization or trimming of service sections is needed.

Next comes annotation of historical conversations. Supervisors annotate a sample according to your dialogue procedure — greeting, needs identification, objection handling, summary.

This sample becomes the reference: criteria are calibrated against it, and it shows how closely reviewers' scores agree with each other. Without such a reference, there is nothing to compare automated assessment against — it will disagree with people on every disputed call.

If part of the archive is incomplete — recordings are cut off, individual calls have no sound — this does not block implementation. We exclude such dialogues or supplement them with metadata from the accounting system: waiting time, transfers, repeat contacts. The model is calibrated on the suitable part, and coverage grows as new recordings arrive.

The implementation scheme without replacing telephony looks like this: archive audit, annotation of the reference sample, connection to the flow of new calls, and regular comparison with people. Move to automated assessment gradually, keeping manual checks alongside.

Describe what archive and telephony you have now — we will come back with an estimate and a work plan.

Common technical concerns before launch

Before a pilot, the same questions come up: what will happen on a poor connection, how assessment will handle a non-standard conversation, whether a given score can be checked, and what to do when the scenario changes.

How are non-standard dialogues assessed?

Assessment is based on the meaning of the utterances and the specified criteria, not on matching a template, so an atypical course of conversation does not break the result — only the score distribution changes.

Conversations that do not fall into any scenario are not discarded: they are collected into a separate sample and annotated manually.

When the conversation scenario changes, criteria and dictionaries are adjusted in a separate settings layer without retraining — the control sample confirms that previous scores have not shifted.

Can you check why a given score was assigned?

Every score has a trail: the recording, the specific utterance, and the criterion by which the score was reduced. The manager opens the conversation, hears the segment, and sees the rule that triggered — this is what makes dialogue scoring explainable.

A disputed score is challenged by the annotator, and the case goes into fine-tuning rather than manual report editing. The results go into familiar reporting — by operator, shift, and team.

Let's discuss the task: what to clarify at the first step

To come back with an estimate of implementation timelines and a work plan, rather than general thoughts, we need a minimum of initial data about your contact center.

Below is what is worth preparing for the first conversation: these points immediately show the scope of work, bottlenecks, and the order of stages.

• Volume of recordings: how many dialogues occur per month, what period the archive covers, and in what form it is stored.
• Channels: telephony, chats, email, messengers — what must be included in assessment and what can be postponed.
• Quality criteria: what rules are currently used to evaluate operators, and whether there are reference reviews to rely on.
• Systems in the perimeter: what telephony and CRM are used, and whether there is an export of dialogues and their metadata.
• Roles and procedures: who reviews disputed scores, whether a person is needed in the review loop, and at what step.
• Processing placement: whether cloud is allowed or data must remain in your perimeter.
• Target result: what effect you consider success — share of reviewed dialogues, review speed, unified criteria across sites.

Describe the dialogue assessment task in free form — that is enough to discuss the approach and come back with an estimate of timelines and an implementation work plan. If data for some points is not yet available, we will go over it on a call and outline what to prepare first.