Why ASR speech recognition breaks on real calls
ASR speech recognition passes confidently on recordings from a quiet meeting room and falls apart on live traffic. A noisy phone line cuts the band down to a narrowband channel; echo from speakerphone, headset clicks, and the neighboring operator’s speech overlay the voice.
A model tuned on clean studios loses word endings and confuses similar-sounding terms — the meaning of the utterance goes with them.
A separate class of failures is overlapping utterances.
When the customer and the operator speak at the same time, two voices end up in one channel, and the system either merges them into a single line or assigns the entire phrase to the wrong speaker: the share of correctly attributed utterances drops severalfold.
A dropped connection behaves differently — the stream arrives in broken fragments; on reconnection, a segment of the dialogue is lost, and a semantic gap remains in the transcript that cannot later be filled.
On long dialogues, errors accumulate. Timestamps gradually drift from the recording, and when investigating an incident, the text points not to where the agreement actually sounded.
Worst of all, it is precisely at overlaps and at the boundary of a drop that key terms are most often spoken — a pipeline without quality control loses them first.
We look at how this is addressed: voice activity detection, separate channel processing, robust speaker separation, and reassembly of the transcript after a break.
What changes after implementation: speed, accuracy, failures
The transition from disparate scripts to a managed recognition pipeline is visible not in a demo but in metrics. Below is what we review at acceptance and in the first week of operation on live traffic.
- Response latency. Text appears in the stream before the operator opens the ticket: the answer arrives during the conversation, not minutes after it ends.
- Recognition error rate. Dictionaries of terms, names, and SKUs plus post-processing remove most errors on domain vocabulary, not only on clean studio recordings.
- Review of long dialogues. Multi-hour recordings are split into segments with speaker separation: the reviewer reads a condensed summary and jumps to the needed minute instead of listening to everything in sequence.
- Operator workload reduction. Manual listening is replaced by spot-checking — fragments with low model confidence and disputed moments go to review.
- Resilience to drops. Segments are saved as they arrive; reprocessing does not create duplicates: a connection loss does not erase a piece of the recording or shift the queue.
- Quality control. The error rate and response latency live on a reporting dashboard and are run against a reference set of recordings with every model update.
The basis is Whisper and open models: we fine-tune them on your recordings, so quality grows with data volume. The output is a predictable pipeline: you can see where the error is and what caused it.
ASR deployment options: cloud, own server, hybrid
The recognition architecture is chosen not by the quality of the model as such, but by where audio may be stored, what stream must be maintained, and how narrow the vocabulary in the recordings is.
Cloud recognition provides a quick start: out-of-the-box streaming transcription, ready-made language models for general speech, and scaling for peak load without your own hardware.
The limitation is that the recording leaves the perimeter, and accuracy drops on industry terms, abbreviations, and names until the model is fine-tuned.
Deployment on your own servers resolves the data transfer issue: processing runs in a closed perimeter, and access, logs, and retention periods remain under your control.
The price is your own infrastructure, model version maintenance, and response latency that grows when compute resources are overloaded. A hybrid scheme separates the streams: sensitive channels go to the internal perimeter, while high-volume requests go to the external service.
| Option | When it fits | Limitations |
|---|---|---|
| External cloud service | quick launch, variable load, general vocabulary | audio leaves the perimeter, volume-based cost, weak customization for terminology |
| Open model on your own servers | personal data, regulatory requirements, stable stream | your own resources and model maintenance are needed, updates on your side |
| Hybrid scheme | mixed traffic: some channels are closed, some are public | more complex routing, two monitoring and versioning pipelines |
| Fine-tuning for terminology | industry vocabulary, abbreviations, names, narrow domains | a corpus of labeled recordings and revalidation on new data are needed |
We fix the scheme with measurements, not feelings: we run real recordings through the options and compare the error rate on terms and names, response time on the stream, and maintenance cost per hour of audio. Then we choose the pipeline — or combine them if requirements differ for different channels.
Engineering stages: from audit to handover
The order of work is structured so that each next stage relies on verified data, not assumptions. Below is what happens after the task is agreed and what result is recorded at the output of each step.
- Audit of recordings and metrics. We listen to real recordings from all channels — telephony, meeting room microphones, web conferences — and look at where errors come from bad sound and where from terminology: the share of speech overlaps, background noise level, pause length, and availability of labeling.
- Pipeline design. We determine where speech recognition runs, how streams are separated, where audio and transcripts are stored, and how the queue is organized under peak load. The scheme is agreed before configuration begins.
- Audio processing setup. Noise and echo suppression, loudness normalization, bringing the sample rate to a uniform format, and segmentation by pauses. Separately, the domain dictionary: product names, abbreviations, names, and job titles.
- Model and post-processing setup. Selecting a model for the language, accent, and channel type; punctuation and casing; formatting numbers and dates; speaker separation. We run every change through the same set of recordings so results can be compared fairly.
- Fault tolerance testing. Channel loss, a sharp load spike, unavailability of an external service: we look at what happens to the queue, whether recordings are lost, and whether retry triggers. Separately, we test recovery after a crash and behavior under degradation.
- Commissioning and handover. We deploy the pipeline and enable monitoring — failure rate, response latency, share of confident recognitions. We hand over to the customer’s team the documentation, access, and rollback procedure to the previous configuration.
Each stage is closed with an artifact that can be verified: a quality report on real recordings, a scheme, configurations, and a test protocol.
Artifacts: configurations, schemes, regulations, documentation
A speech recognition pipeline lives not in chat but in a repository. If after handover the customer’s team is left with only a working service and a couple of screenshots, any change turns into reverse engineering. That is why we prepare artifacts as part of the result, not as an appendix to it.
- Pipeline configurations — service declarations, model version pinning, stream limits, and timeouts. The environment comes up from scratch; environment differences are moved into variables.
- Data flow diagram — the path of audio from input to text and storage: where buffering happens, when a recording goes to a backup recognition node, and what data does not leave the perimeter.
- Access rules — roles, key lifetimes, environment separation; it separately states who may change the model and who only reads logs.
- Operations regulations — what is monitored and with what thresholds: recognition error rate, response latency, queue depth; how scheduled updates are performed.
- Common failure guide — a table of “symptom → cause → action”: stream drop, accuracy degradation after an update, recording storage overflow.
- Quality evaluation methodology — a labeled set of recordings with names, terms, and noise, recognition metrics, and the run procedure before rollout.
- Integration specification — exchange formats, limits, and behavior when neighboring systems are unavailable.
Artifacts live next to the code and are versioned: changes go through review, and every document has an owner. This is what allows updating the model or changing vendors without reverse-engineering the system from logs.
Case study: eliminating losses on long dialogues in a support service
Before
The support line received long dialogues — from seven minutes, with clarifications, pauses, and overlapping speech. Transcription was selective, about a tenth of the requests, and manual: a quality operator listened to the recording and wrote out fragments.
At utterance boundaries and when the speaker changed, some words were lost, so reports had missing fragments, and disputed requests were closed from memory rather than from the recording.
What we did
The analysis was built in two passes: speaker separation and segmentation of long utterances with overlapping segments, then speech recognition with a domain vocabulary — service names, tariffs, and internal abbreviations.
The core was the open model Whisper, fine-tuned on support line recordings. Low-confidence segments are marked and go to spot-checking rather than silently dropping out of the export.
What changed
The share of fully analyzed requests grew from 8% to 100%. Missing fragments in long dialogues decreased from 12% of utterances to 2% — the remainder is due to noise and simultaneous speech by two speakers.
Repeat requests on the same issue fell by a third: the operator sees the full history and does not ask again what the customer has already said. Manual analysis of one dialogue took about twenty minutes; now review takes two to three and only for borderline cases.
How does streaming audio processing differ from batch processing?
Streaming processing outputs text as the audio progresses: audio is cut into short frames, the model produces a hypothesis immediately, and the interface shows an intermediate result with sub-second latency.
Batch processing of recordings works differently — audio is fed in whole or in large blocks and recognized after the conversation. The difference is not in the model but in how much context is available at the moment of decision.
In streaming mode, the context window is limited by the buffer: the system does not hear the phrase to the end, so names, terms, and numerals are guessed from the beginning.
Words are lost at segment boundaries, speaker labels get confused during interruptions, and punctuation is based on a fragment. That is why hypothesis stabilization is mandatory — already displayed text is rewritten when the next frame arrives.
Batch mode allows a second pass: loudness normalization, noise suppression, speaker labeling across the entire file, and rechecking doubtful places with full context.
The word error rate on the same data drops by a factor of one and a half to two, but the result appears after a time comparable to the recording duration. This is the quality trade-off: accuracy is exchanged for response time.
The choice of speech recognition mode is dictated by the scenario, not by a desire to “make it more accurate.” Subtitles, a voice assistant, operator prompts in a dialogue — only streaming processing: latency of more than a second breaks the scenario.
Meeting transcription, call analysis, archives, and audits — batch: there is no rush, but the full context of the conversation is available.
In practice, a hybrid is often needed: a draft streaming text for real-time work and a final pass over the saved recording that replaces intermediate versions in storage.
We separate these two pipelines at the design stage — an attempt to catch up with batch quality inside the stream runs into resource and latency constraints.
Technical concerns: noise, accent, terminology, dialect
The quality of ASR speech recognition is determined not by the model but by the channel: background noise, room reverberation, narrowband telephony, packet loss.
That is why we start with the customer’s real recordings and determine at which stage meaning is lost — speech detection, acoustics, or the language model.
Background noise and connection quality are addressed at the input: noise suppression and a voice activity detector cut off pauses and extraneous sources, while buffered streaming mode smooths out dips.
Then domain adaptation comes in — a domain dictionary, priority term lists, normalization of numbers and abbreviations.
We take an open base engine, for example Whisper, and fine-tune it on your recordings: accent and diction stop being a verdict when the model has heard exactly these voices.
There are no industry terms in the dictionary — how to recognize them?
Terms and abbreviations are almost absent from general datasets, and the model hears similar everyday words in them. Priority term lists and template-based substitution restore correct spellings — provided the dictionary is updated along with the product.
What to do with poor connection and stream?
A telephone channel cuts high frequencies, and packet loss produces broken fragments. With this kind of input, processing with overlapping windows and a second pass over disputed sections helps: tuning is done for the stream, not for a finished file.
We verify each tuning step on labeled segments: errors are counted separately for terms, noisy recordings, and accented speech — you can see what produced an improvement, rather than a vague “it got better.”Let’s discuss the task and come back with an estimate and plan
Estimating speech projects depends on data more than on code, so a short introductory set is needed before calculation. Below is what to send on your side and what we will return in response.
On your side:
- 10–20 hours of real recordings — calls, meetings, noise, overlapping speech: recognition quality is measured on them, not on a studio sample.
- Usage scenario: conversation search, quality control, operator prompts — the processing mode depends on it.
- Where an error is critical: names, terms, numbers, quantities — and what noise level we consider acceptable.
- Integration points: telephony, meeting recording, real-time stream, or archive analysis.
- Data restrictions: where audio is stored, whether it can be taken outside the perimeter, and what access rules exist.
- Domain dictionary: product names, job titles, and abbreviations that the system must hear correctly.
We will return in response:
- Scope of work with boundaries: data preparation, domain tuning, integration, acceptance.
- Pipeline scheme: the path of audio from source to result, where what is stored, and what leaves your perimeter.
- Stage plan with timelines and points where your access or decisions are needed.
- Acceptance conditions: metric, measurement method on a holdout set, and a list of scenarios to be tested.
- Handover plan for the team: access, repositories, configurations, and maintenance instructions.
Describe the task — we will come back with an estimate and a plan.