Azure Speech Services: STT and Custom Speech integration

Why Azure Speech Services makes mistakes on live recordings

The general recognition model was trained on clean announcer speech, but in production it gets an archive of real calls.

A narrowband telephone channel, compression, speakerphone echo, office hum, and hold music reduce intelligibility even before the signal reaches the model — and Azure Speech Services confidently outputs incorrect words on such input instead of leaving a blank.

The second failure point is overlapping voices. When the customer and the agent interrupt each other, speaker separation makes mistakes, utterances merge into a single stream, and from the transcript it is no longer possible to understand who confirmed what.

For analyzing disputed interactions, such a transcription is useless: it does not reconstruct the course of the dialogue but retells it approximately.

The third layer is vocabulary. Item numbers, service names, legal wording, abbreviations, and surnames are substituted by the model based on phonetic similarity and frequency in the training set: a similar-sounding word displaces the correct one.

Without a domain dictionary and adaptation of the acoustic model to a specific channel, the accuracy of record transcription drops the more specific the industry is.

What this costs. Analysis of the interaction archive stops relying on text: analysts read through recordings manually, the share of dialogues covered by checks falls, and search across transcripts does not find what is needed.

Speech recognition errors also skew metrics — a topic that customers name identically breaks down in the report into dozens of spelling variants. Below we examine how this is addressed at the level of audio preparation, dictionaries, and configuration of the service itself.

Effect of tuning: accuracy, speed, archive analysis

Recognition tuning produces a predictable shift if metrics are fixed before work begins.

We calculate three: the share of incorrectly recognized words on a control sample, the processing time of the accumulated archive, and the share of records that go into work without manual correction. Below are benchmarks we see on projects after tuning Azure Speech Services for industry vocabulary.

The shift is not delivered by "tuning in general" but by two levers: the model is fine-tuned on your recordings, and the language model is trained on a dictionary of terms, abbreviations, and names. That is why quality improves precisely where the base model made mistakes most often.

• Terms and abbreviations go into the domain dictionary, not into text post-processing.
• Names, titles, and place names — separate lists for each recording stream.
• Speaker separation of utterances, pauses, and punctuation — meetings can be read without listening.
• Validation on a holdout sample: we calculate metrics on recordings the model did not see during training.

Result — transcription stops being a manual project: archive volume grows while the share of edits falls.

Azure Speech modes: stream, archive, model fine-tuning

Speech recognition operating modes are chosen for the task, not by the "power" of the service. Streaming speech recognition is needed where a response is required during the conversation: agent prompts, subtitles, voice bot.

Batch audio transcription is for what has already been accumulated: a call archive, meeting recordings, analysis of interactions over a period.

Fine-tuning the recognition model is not a third mode alongside them, but a layer on top of both.

An industry model with abbreviations, product names, and specific vocabulary serves both the live stream and the archive, so the options combine in one environment: Azure Speech Services provides streaming, batch, and custom models through a single access point, and switching between them does not require rewriting the integration.

In practice, you do not settle on one mode: first the archive is processed to collect a corpus, then the model is fine-tuned and the stream is enabled.

The transition is reversible — when rolling back to the base model, both stream and batch continue to work without changes to the integration.

How implementation is structured: from pilot to handover

Speech recognition implementation rarely hinges on the engine itself — difficulties begin at the intersection of audio, industry vocabulary, and business systems.

That is why we start from the customer's real recordings, not from a demo clip: first we look at what exactly will need to be transcribed.

1. Task and audio analysis. We combine sound sources (telephony, microphones, archives), noise level, number of speakers, and acceptable latency. The output is a list of scenarios the system must cover.

2. Integration design. We decide where the audio lives and where the text goes: streaming transcription during a call, batch archive processing, or a hybrid. We fix buffering and behavior during connection drops.

3. Pilot on real recordings. On an annotated sample from the archive, we measure baseline quality before any improvements. This is the reference point: thereafter every change is compared with it, not with impressions.

4. Tuning the model for the industry. On Azure Speech Services, we fine-tune the acoustics and language model on terms, abbreviations, and names that the general engine hears incorrectly; separately, we configure formatting of numbers and dates.

5. Validation on a control sample. We run the same set of recordings again and compare the share of correctly recognized words and accuracy on key entities. If there is no improvement, we return to the dictionaries rather than fixing by ear.

6. Handover and team training. We provide configurations, stream design, instructions for updating dictionaries and monitoring. We walk through them with your engineers so vocabulary can be changed without us.

The output is a working integration, a set of recordings for regression, and a model update procedure.

Configurations, dictionaries, and service access policies

Adaptation artifacts remain in the customer's environment: we hand over not a "configured service" but a reproducible set of configurations.

Azure Speech Services is the base layer, on top of which model profiles, dictionaries, and access rules live: they can be rebuilt, checked, and rolled back without our involvement.

• profiles for each scenario: tuning the speech recognition language model on domain data, confidence thresholds, and normalization rules — separately for telephony and meetings;
• domain dictionary: terms, abbreviations, product names, and surnames with pronunciation variants; versioned, edits are validated;
• audio source connection design: channels, formats, fragmentation, where source files are stored, and what is deleted immediately after transcription;
• access rights to the recognition service: separate accounts for applications, least-privilege roles, key rotation, no shared keys;
• processing policies: masking of sensitive fragments, retention periods for transcripts, audit log of audio access;
• regression set: reference recordings with verified transcripts, used to measure quality before and after any change;
• recognition documentation set: deployment from scratch, dictionary updates, log reading, actions for quality degradation.

The set makes the work reproducible: a new model or term is tested on reference recordings, not on live output. If the contractor changes, the customer hands over a ready set of configurations instead of reconstructing knowledge from correspondence.

Case study: transcribing an interaction archive in a week

A contact center came to us with an archive of interactions: about 40,000 hours of audio over several years, some recordings in low quality and with overlapping voices.

Baseline recognition with the general model produced a word error rate of around 28%: tariff names, agent names, and industry abbreviations turned into random strings of letters, and search across transcripts did not find the needed conversations. Selective manual verification took weeks and still covered only a small share of the archive.

We worked on Microsoft Azure: we assembled a corpus of 40 hours of audio with manual annotation, and separately annotated entities — order numbers, service codes, tariff names.

We trained a custom model on this corpus, added phrase lists for names and abbreviations, enabled channel separation, "who is speaking" labeling, loudness normalization, and pause-based segmentation before feeding the service. The configuration was versioned — at any step it is possible to return to the previous model.

On a holdout test sample, the word error rate fell from 28% to 11%, and for key entities — to 4%: order numbers and service names are recognized with almost no corrections.

The 40,000-hour archive was processed in a week with parallel jobs; the share of fragments that had to be resubmitted due to a failure was less than 1%. Search across transcripts began to find conversations by service mention, not by call date.

How does model fine-tuning differ from the base version?

The ready-made speech recognition model is trained on anonymized recordings across broad topics: it parses ordinary vocabulary confidently, but makes mistakes on terms, abbreviations, and surnames.

Fine-tuning the recognition model is not replacing the engine but adding your dictionary and your acoustics on top of the base version.

In Azure Speech Services, adaptation is divided into two independent layers. The language layer relies on a text corpus — transcripts, glossary, phrase list — and corrects vocabulary: compound terms, item numbers, names.

The acoustic layer is tuned on recordings — communication channel, accent, background noise — and improves parsing where the base hears a similar-sounding word.

The layers differ in data requirements: the language layer is assembled from hundreds of thousands of words and is enabled on the next iteration, while acoustics needs annotated recordings — usually tens of hours per dialect or complex channel.

The cycle "collected data — trained — measured" takes days, so accuracy is improved through iterations, not a single run.

Comparison of recognition approaches is valid only on a holdout sample from your recordings: the error rate is calculated before and after fine-tuning, separately for rare words — that is where the gap is largest.

The fine-tuned version is fixed at the time of training; when the nomenclature changes, the model is retrained, and a small dataset introduces bias and degrades parsing of common phrases.

That is why we run both versions on the same test and look at behavior on your dictionary, not at an average score.

Common technical concerns before project start

Objections at the start are almost always the same: the recording is noisy, audio cannot leave the perimeter, and quality must be proven on your own data, not on demo clips.

Below are specifics for each point, including the case when recognition is already deployed on another solution.

Audio must not leave the closed environment — is this feasible?

Yes: recognition is deployed in containers inside your infrastructure, and recordings and texts do not go to external services.

We configure network rules, storage and lifecycle for audio, access logging, and key rotation — so that the closed environment is confirmed by configuration, not by an agreement.

How can recognition quality be checked before implementation?

We assemble a representative sample — recordings from all channels and conditions where the service will operate. We annotate, calculate metrics, and break down errors by class: terminology, numbers, names, speech overlap.

The output is a measurement before implementation and the same measurement after, using the same methodology. This way accuracy is visible in numbers, not by ear.

Industry terminology changes — do we have to redo everything from scratch?

No. Phrase lists and dictionaries are updated separately from the integration, and model retraining is a routine procedure with the same validation on a control sample.

We work with Azure Speech Services and compatible engines, so replacing or fine-tuning the model does not break application logic or processing scenarios.

Let's discuss the task: what is needed to start work

To discuss a speech recognition task concretely, it is worth gathering several things for the conversation. From them it is immediately clear whether the base model is sufficient or Custom Speech training on your data is needed, and the scope of work takes shape.

• Recording examples — 20–30 files per scenario: a quiet meeting room, a noisy workshop, a telephone line. From them we calculate the real error rate before fine-tuning.
• Audio sources — telephony, meeting recordings, voice recorders, a stream from microphones. Noise suppression and speaker separation depend on this.
• Term dictionary — product names, job titles, abbreviations, names. It goes into pronunciation configuration and removes errors on narrow vocabulary.
• Accuracy requirements — where a draft transcription is sufficient and where character-by-character parsing is needed. This determines the volume of annotation and the number of training iterations.
• Load — how many hours of audio arrive per day and what the peak is. This determines the choice: batch processing or streaming recognition.
• Data handling rules — where to store audio and text, how long to retain it, who has access, whether anonymization is needed before training.
• Integration points — where to send the result: business system, storage, event bus — and in what format it is expected.

Next, we prepare an estimate for the recognition project: we calculate hours for data preparation, training, and quality validation, and present a stage-by-stage plan and acceptance metric. Work on recognition starts with a pilot on one scenario, after which we expand to the rest.

Describe the task — we will come back with an estimate and a plan.