Development of an AI system for automatic scoring of operator calls

Automatic call scoring: where manual control breaks down

Service quality control still relies on selective listening: a supervisor can physically review only a few percent of conversations per shift.

The rest of the flow is not evaluated at all, and the sample itself is biased — it includes the longest, most conflict-heavy, or already problematic calls opened due to a complaint. As a result, the metric describes not the department's work but what was convenient to open.

The second source of distortion is human.

Two reviewers score the same recording differently: a one- or two-point discrepancy on the scale is considered normal, because criteria live in the head and in verbal agreements, and the checklist is updated less often than the product changes. Operator scoring ceases to be comparable — neither between people nor between months.

The third problem is time. The review reaches the employee weeks later, when they remember neither the context, nor the client, nor their own wording. Feedback arrives at a moment when the behavior is no longer reproduced, and training is not tied to a specific utterance.

We explain how this is closed by continuous automatic call scoring: labeling conversations by typical scenarios, unified criteria for the entire stream of recordings, and feedback on the same day, while the conversation is still in progress.

What the system provides: rejection of sampling and a unified scale

Continuous review of conversations changes the very logic of control: the system processes the entire call flow, not a sample of a few percent that a manager can physically listen to.

Speech is recognized with Whisper and separated by speakers, then each utterance is checked against a criteria checklist — required wording, interruptions, correctness of conditions, objection handling.

Every recording gets a score, and the disputed fragment is marked with a timestamp: the review goes by quote, not by memory.

A unified scoring scale is the second effect. Criteria and weights live in the configuration, not in the reviewer's head, so the same conversation gets a similar score on repeated review.

Discrepancy between two reviewers shrinks from a point and a half to tenths, and disputed cases go into a separate sample for calibration.

Time to feedback is measured in minutes: the review comes to the operator on the same shift, while the call context is fresh, not a week later at a general meeting. This changes the meaning of control — the review becomes training, not a report.

Separately, the class of missed violations disappears. Critical criteria — conditions, amount, or deadlines not stated — are flagged regardless of the final score, and the manager sees a list of such calls, rather than hoping they did not make it into the sample.

Against this backdrop, the department's quality of work becomes measurable: the share of calls with critical violations, the shift in score distribution, and the dynamics for each criterion are visible.

Implementation options: ready-made service, own model, hybrid

Three modes differ in whose model produces the score and what data it was trained on.

A ready-made solution runs on a model trained on a general call flow; an own model — on your recordings and your regulations; a hybrid approach keeps the base model but extends it with industry vocabulary and conversation rules.

The choice comes down to three parameters: time to launch until the first scores, the amount of preparatory work on your side, and requirements for industry vocabulary — how important it is that the system understands internal names, abbreviations, and regulation wording.

In practice, the fork is rarely binary: teams usually start on a ready-made solution — to get the first scores and compare them with selective manual review — while accumulating labeled calls. After a few weeks, scoring is moved to an own or hybrid model without stopping quality control.

Work stages: from recording audit to system handover

We start not with code but with the archive of recordings: without understanding how conversations actually sound, any scale will be detached from practice.

Below are the implementation stages; we do not change the order, because each step relies on artifacts from the previous one.

1. Recording audit. We export a sample from the archive for several months, remove silence and service inserts, and check channel quality. The output is a conversation map: typical scenarios, share of anomalous calls, and places where the recording requires preprocessing.

2. Scale definition. We collect criteria together with department leads: what we consider a greeting, needs identification, objection handling, and compliance with regulations. Each criterion gets a weight and a description of edge cases, including "not applicable".

3. Labeling and model training. We label several hundred dialogues according to the approved scale and discuss disputed cases among annotators. This way, wording no longer allows double interpretation, and the system learns from agreed examples.

4. Validation on a holdout sample. We compare held-out dialogues that did not participate in training with manual labeling — separately for each criterion, not by average score. A discrepancy above the acceptable level means returning to the wording, not tweaking the threshold.

5. Pilot on live flow. We enable the system on part of the calls, compare scores with the shift manager's review, and configure notification thresholds. It shows how many conversations require attention and how often the score is disputed.

6. Team training. We show managers how to read the report and how to dispute a score for a specific call, and operators — how to review their own. We hand over the scale description and the regulation for its revision.

Configurations, scoring schemes, and labeling regulations

We hand over not a presentation but a working package: each document is tied to a file in the customer's repository, so scoring rules can be edited without our involvement. Below is what remains with the team after the automatic call scoring launch.

• Scoring configurations. Criterion weights, trigger thresholds, and rule sets by call type — as files versioned together with the code; edits go through the same validation run on the reference sample.
• Telephony integration scheme. Connection point to the PBX, recording export formats, behavior on segment break or delay, where the error signal goes, and how missed calls are reloaded.
• Scale description. What each score means, how the total is calculated, which criteria are mandatory and which only adjust the score, and where the boundaries of "satisfactory" and "requires review" are.
• Access policies. Roles — operator, supervisor, methodologist, analyst: who sees the recording and transcript, who changes the scale, retention periods, and anonymization rules.
• Disputed case regulations. How an operator disputes a score, who reviews the dispute and within what time, how the decision is recorded and enters system retraining.
• Instructions for reviewers. How to assemble a sample for quality control, what counts as a discrepancy with the model, and at what discrepancy threshold rules should be revised.

Separately, we maintain a change log: date, author, and reason for each scale edit — so that in six months it is clear why the score shifted.

Case: how selective review was removed in the sales department

Sales department — about 40 operators on two lines. Quality control relied on three senior managers: they listened to 2-3% of conversations, each by their own checklist.

Feedback reached the operator after one or two weeks, and score discrepancy between reviewers on the same call reached a third of the scale — there was nothing to dispute the score with, as the criteria were not at hand.

Automatic scoring was assembled: recordings are pulled from telephony and Bitrix24, speech is recognized, and the model labels the conversation by checklist — script and greeting, needs identification, objection handling, and fixed agreements.

Every score is tied to a recording fragment, so the operator sees what points were deducted for and what to replay in the next conversation.

Reviewers no longer listen to everything: they review disputed cases that the system flags itself — low recognition confidence, criterion conflict, or an objection without operator reaction.

Metrics before and after

• Share of reviewed calls: 2-3% → 100%.
• Time to operator feedback: 7-14 days → the same day, usually within an hour after the shift.
• Score discrepancy between reviewers on the same conversation: about 30% → within 5%.
• Disputed cases resolved on the day of request: 10-15% → 9 out of 10.
• Manual listening by senior managers decreased approximately threefold with full conversation coverage.

What to do if recordings lie in disparate systems

Different telephony systems almost always mean different data sources: a cloud PBX writes conversations to its own storage and provides them via API, recordings from the second system are exported as files to a server, and some calls come from messengers and managers' mobile apps.

Formats also differ — compressed and streaming, stereo and mono, with different channel lengths and different "operator/client" channel labeling. Until this is brought to one form, any call scoring starts with manual hassle, not analysis.

The first step is an inventory of sources: which systems record audio, where it is stored, and by what attribute each recording is linked to a deal.

Then a collection layer is built: connectors to telephony and CRM APIs, archive export, and reloading of files missed for the period. Each recording gets a unified technical passport — source, time, direction, participants, and channel. This is the foundation without which a unified labeling pipeline cannot be assembled.

The key problem in such projects is matching with deal history.

The client number, internal identifier in amoCRM, operator extension, and timestamp rarely match perfectly, so we do multi-step linking: exact match by identifier, then by number and time window, then a manual queue for disputed cases. The share of recordings linked to a deal is measured on every run — it shows where the link breaks.

On top of this, a unified labeling pipeline is assembled: a transcript with speaker separation, a link to the deal stage, automatic call scoring results, and the operator score.

After that, calls from all telephony systems can be compared with each other — by department, product, and operator tenure.

If some recordings historically remained unlinked, they go into a separate layer: they can be labeled by content, and the client link can be restored later when a deal appears.

First, we review your source map and show what share of calls is linked to deals automatically and what share requires refinement.

Complex cases: noise, interruptions, dialect, irony

Noise and interruptions break recognition not by themselves but at junctions: the system marks the section where both channels speak simultaneously and does not merge utterances into one.

Recordings come in separate channels — operator and client speech are not mixed, so overlaps are visible positionally, not by guess. Such sections are not discarded from scoring: they are marked as disputed, because that is where the key phrase is most often lost.

Dialect and a strong accent hit endings, function words, and names — the meaning of the phrase is guessed, but entity accuracy falls.

We expand the dictionary with recordings from a specific region and decide by intent: if the model is confident in the essence of the request but not in individual words, the case is scored by meaning, not discarded.

For figurative meaning, the rule is stricter: irony, sarcasm, and allegories create a conflict between tonality and the semantic tag, and such a conflict is not resolved automatically.

Where guessing begins, we consciously leave manual review. Model confidence is compared with a threshold configured for the specific flow: usually a few percent of calls go to listening — those where the score diverges from the expected one or where the signal is cut.

Every reviewed recording returns to labeling, and the rules are refined rather than fixed once and for all.

Can manual review be abandoned entirely?

Technically — yes, the threshold can be raised to zero, and all calls will be scored automatically. But then disputed sections with irony, overlaps, and dialect will get a formal score that looks convincing and creates high review overhead with the operator.

Hybrid mode gives stable quality: the machine covers the main volume, and the human — only where the decision is ambiguous.

Let's discuss the task and come back with a work plan

To discuss the task substantively rather than in general terms, it is enough to describe the process as it is currently arranged: who reviews calls, by what criteria, and how much time one sample takes.

There is no need to prepare a technical specification in advance — we will assemble it together from your source data, and a formal document at this stage would only lengthen the path.

If there are conversation recordings and scoring regulations at least in draft form, this is enough to move from words to scope assessment.

• description of current quality control: who scores, by what criteria, how often and with what frequency;
• anonymized sample of recordings for several weeks — necessarily including those conversations that are today assessed as disputed;
• current regulation or scoring checklist, even if it lives in a spreadsheet and is not approved anywhere;
• workload data: how many calls per day, what share goes into review, and how long one review takes;
• information about the systems where recordings and scoring results are already stored, and how they can be connected to;
• access and personal data storage restrictions that must be taken into account before the start, not after.

After the review, we come back with a scope estimate, a cooperation procedure, and a work plan by stages: data preparation and labeling of disputed cases, setup and calibration of the system on your sample, accuracy validation on conversations where opinions diverge, and launch in the current process. We agree on the plan before work starts, so that expectations for scope and result match.