STT Development: why the pilot doesn't survive to production
A pilot on clean recordings from a meeting room delivers speech recognition accuracy above 90%, but after a month on real traffic it drops to 65-70%.
Noisy channels, narrowband telephone path, echo, and overlapping speech of two speakers degrade acoustics more than any accent in a studio recording. Fixing this after the fact with filters is too late — the decision is made at the stage of data preparation and model selection.
Domain vocabulary delivers the final, targeted blow. SKUs, medication names, department acronyms, and surnames are confidently replaced by the base model with sound-alike words, and such text moves further through the process as valid.
This is solved not by "fine-tuning on everything indiscriminately" but by a pronunciation dictionary, a language model on a domain corpus, and annotation of exactly the entities the business needs.
Streaming latency is a separate failure point. If the engine waits for the file to end and only then outputs text, the dialogue with the operator breaks into pauses; engineers split the stream into windows and pay for it with higher errors at boundaries.
That is why real-time processing and archive processing are designed as two different modes with different requirements for quality and speed.
And lastly, archive reprocessing. Each new model version devalues previously transcribed material: to raise quality, accumulated hours have to be rerun, and this creates load on compute resources and timelines.
Planning this in advance means versioning models, storing source audio and features separately, and understanding which recordings really need reprocessing. We'll examine how these points are addressed at the design stage.
What a well-tuned speech recognition pipeline delivers
Recognition quality cannot be evaluated on a demo recording: real calls have noise, overlapping speech, and industry terms absent from the general vocabulary.
A well-tuned pipeline is distinguished by metrics measured on the same audio the client works with, not on a studio corpus.
The second layer is behavior under load: how long the first fragment waits, what happens on a connection drop, how participant voices are separated.
We fine-tune models for the domain, including open ones like Whisper, and deliver the result as a stream, not as a batch after file upload.
| Metric | Without pipeline tuning | After tuning |
|---|---|---|
| Error rate on real recordings | 15–25% on conversational calls with noise and accent | 4–8% on the same corpus, terms from the client dictionary are recognized consistently |
| Response time | recording accumulates and is transcribed entirely after upload | first fragment arrives within a second, the rest as speech proceeds |
| Manual review of transcripts | continuous proofreading of all recordings | selective verification only of fragments with low model confidence |
| Resilience to interruptions | session loss, recording must be started over | checkpoints and resume: session continues from the last confirmed fragment |
Overall, it is not one number that changes but an entire class of failures that disappears: transcription stops being an "overnight" task and does not fall apart on connection failure.
The operator looks not at the entire text but at marked places — hence a several-fold reduction in manual work.
STT Development: architecture options and their limitations
Choosing between a rented service, an in-house pipeline, and a hybrid scheme is not a debate about recognition accuracy.
On the same acoustics, the approaches produce similar numbers, but they diverge along three axes: where audio physically resides, what stream must be sustained at peak, and who is responsible for quality on domain vocabulary — medication names, SKUs, internal acronyms.
The main branching point is data constraints. If recordings contain clients along with medical or legal details, transferring the stream to a third party requires de-identification on the fly, and often this is what decides in favor of an in-house pipeline.
The opposite case is a rare language absent from the in-house model: here renting solves the task immediately, without months of annotation.
| Option | When it fits | Limitations |
|---|---|---|
| Rented service | Quick start, many languages, peak loads without own capacity | Audio leaves the perimeter, costs scale linearly with minutes, fine domain tuning is unavailable |
| In-house pipeline | Data protection requirements, stable volume, closed network | Long time to production, own servers with accelerators, quality depends on volume of labeled recordings |
| Hybrid scheme | Sensitive data is processed locally, rare languages and heavy tasks go to rented capacity | Two subsystems in monitoring, more complex routing and model versioning |
The practical selection criterion is not average accuracy but the error rate on the most expensive scenarios: order numbers, amounts, addresses. We assemble a test set from the client's real recordings, run it through all three options, and show exactly where the results diverge.
Work sequence: from recording audit to pipeline handover
Starting with the choice of recognition engine is a typical mistake: quality is determined not by the model but by the condition of recordings and how well the pipeline accounts for domain conditions. Therefore, the stages proceed sequentially, and each relies on the artifacts of the previous one.
- Recording audit. We take a sample from real channels — calls, meetings, dictation — and annotate problems: noise, echo, overlapping utterances, narrowband lines, spontaneous speech, share of domain terms. The output is a map of conditions and an understanding of which classes of recordings are even eligible for automation.
- Pipeline design. We fix the flow: source, preprocessing (noise suppression, speech detection, speaker separation), recognition, post-processing (punctuation, numbers, term dictionary), output to the target system. Separately, we describe failure points and behavior when external nodes are unavailable.
- Domain tuning. We collect dictionaries of terms, acronyms, and names, tune the language model on domain vocabulary, and if needed fine-tune the acoustic part on labeled fragments from the audit.
- Validation on a holdout set. We confirm quality on a corpus that did not participate in tuning, broken down by subgroups: communication channel, noise level, speech rate, term density. The average across the entire corpus hides failures on individual groups — and those are what matter in operation.
- Acceptance by metrics. We fix thresholds by subgroups, not an overall percentage: where automation returns the result immediately, where a second pass or a "review by operator" flag is needed.
- Pipeline handover and team training. We transfer the scheme, the retraining procedure when vocabulary changes, the metrics dashboard, and the incident resolution procedure; we train engineers and operators.
At the output, the client receives a working pipeline, documentation, and a team that can develop it without us. Describe your task — we'll return with an estimate and a stage-by-stage plan.
Deliverables: configurations, diagrams, and procedures
The engineering result of the project is not only a working speech recognition service but also a set of artifacts by which it can be deployed, tested, and developed by the in-house team.
We transfer everything in a repository together with instructions for deploying the pipeline on clean infrastructure.
- Pipeline configuration — a declarative description of recognition services, queues, load balancing, and autoscaling rules: the pipeline is brought up from configuration, not assembled manually.
- Data flow diagram — the audio path from ingestion to text output: buffering points, reprocessing locations, and the steps at which the recording is deleted.
- Access policies — service roles, keys, network restrictions, and recording handling rules: retention periods and masking of sensitive fields.
- Models and recognition profiles — fixed weight sets, domain dictionaries, and settings for different channels: telephone, microphone, meeting stream.
- Quality check suite — test fragments with reference transcripts and metrics: word error rate and named entity error rate for each scenario.
- Maintenance procedure — the order of model updates, load control, monitoring signals with thresholds, and conditions for rollback to the previous weight set.
- Team instructions — how to connect a new channel, add terms to the dictionary, resolve an incident, and provide feedback on errors.
Case study: call recognition in a contact center before and after
A contact center with several dozen operators handles thousands of calls per day, and each recording must be turned into a clear interaction card. Before restructuring, transcription ran as a single stream: without channel separation, audio normalization, and role labeling.
The operator received continuous text and corrected it manually, while the supervisor assessed quality by ear on a dozen recordings per week.
What was restructured in the pipeline
The pipeline was split into stages: audio conditioning, speech detection, recognition, role labeling, entity extraction.
We took Whisper in the current branch as a basis and fine-tuned it on the client's recordings, adding a dictionary of product names, addresses, and typical formulations. Recognition was made asynchronous: the transcript reaches the operator before they open the card.
The error rate on key entities dropped from 28% to 7%. The share of transcripts the operator accepts without edits grew from 59% to 88%, and interaction processing time decreased from nine minutes of manual filling to a minute and a half of review.
More than half of interactions are processed automatically and go into reporting without human involvement.
A side effect is quality control: the supervisor sees a sample across all calls, not ten per week, and a disputed interaction is reviewed based on the transcript. Models are versioned, so if accuracy drops on a new call segment, there is somewhere to roll back to.
How streaming processing differs from batch processing
The difference is not in the model but in the output contract. Streaming processing emits intermediate hypotheses as audio arrives and must finalize a phrase within fractions of a second after a pause; batch processing takes a ready file as a whole and can afford several passes.
Hence different resilience requirements: in a stream, channel failure interrupts the dialogue; in an archive, the recording simply goes to the end of the queue.
Real-time mode is needed where the result affects the current action: operator prompts, online conversation quality control, live broadcast subtitles, routing a call to the right department.
Here everything is subordinated to latency: audio is cut into short overlapping windows, intermediate text versions are refined as context accumulates, and the model computes by segments.
Accuracy on short phrases is lower than with a full pass, and this is a conscious trade-off, not a defect.
Batch processing runs over the archive: call recordings, meetings, voice memo notes. The material is available in full, so level normalization, speaker labeling, re-recognition of questionable segments, and a domain term dictionary are applied.
Latency is measured in minutes, but throughput is planned in advance — night windows, queue, reruns after a model update.
Both modes are combined on a common foundation: one set of models, a unified text normalization and dictionaries layer, but different input adapters and different queues.
Then an improvement to the dictionary or model is transferred immediately to both streams, and results can be cross-checked.
The choice is dictated by acceptable response time and the share of scenarios where an answer is needed at the moment of speech — everything else is cheaper and more accurate to run in batches.
Concerns about quality, data, and load
Before the start, the same five doubts almost always surface: will the system handle noisy recordings, can it be fine-tuned without moving data outside the perimeter, is it realistic to tune recognition to your own terminology, what will happen under peak load, and how to later change vendors without losses.
None of them is resolved by promises — each item is backed by a specific engineering mechanism that is measured on your recordings before production operation begins.
Below are short answers without marketing: what we do, how we measure, and where the limits of applicability lie.
Open models like Whisper set baseline quality, but on noisy channels the result is pulled up by data preparation and the result delivery scheme, not the engine itself.
Noisy recordings: call, office, street — will quality drop?
We bring audio to a unified format, cut by spectrum, and add controlled noise during training.
The share of confidently recognized phrases on recordings with echo and background conversation grows by tens of percent relative to the base model, and questionable segments are marked for selective operator review.
Data cannot be moved outside the perimeter — is that possible?
Yes: we deploy the entire recognition pipeline on your infrastructure, and training runs on anonymized samples inside the perimeter. Only aggregated quality metrics and error reports go outside — without the recordings and transcripts themselves.
Fine-tuning on your own terminology
We collect a dictionary of names, acronyms, and domain terms, and fine-tune the acoustic and language parts on your transcripts. This noticeably reduces errors on rare words, and the dictionary remains editable — new terms are added without retraining the entire system.
Peak load
Streaming requests go with priority, others enter the queue; compute resources scale by queue depth. A burst several times above average leads to delayed batch processing, not failures and lost recordings.
Vendor change — will we remain locked in?
Configurations, dictionaries, adapters, and labels are yours; they reside in your repository together with a deployment checklist. Transition to another team takes days: the same pipeline is deployed, metrics are reproduced, and work continues with accumulated data.
Let's discuss your speech recognition task
A project estimate for speech recognition is not assembled from general words — it rests on the inputs. In the first conversation, we clarify where the sound comes from: telephony, room microphones, meeting recordings, how many channels need to run in parallel.
And the domain with vocabulary: names, SKUs, terms that transcription must spell correctly, plus latency requirements if text is needed during the conversation.
Separately, we analyze noise, speech overlap, and the quality of the recording archive: these inputs show where fine-tuning on your data is needed and where tuning the language model and post-processing rules is sufficient.
Based on the conversation, we send:
- a stage-by-stage work plan — from recording preparation and labeling to holdout validation and production deployment;
- a flow diagram: audio ingestion, preprocessing, recognition, text normalization (punctuation, numbers, names), delivery of the result to your system;
- a project estimate: team composition, stage durations, and what is required from your side — access, recordings, approvals;
- target metrics: word error rate, accuracy on names and terms, acceptable output latency;
- deployment options: in your pipeline or on dedicated capacity, with load calculation for the number of concurrent channels.
Project start is after approval of the plan and metrics: we fix the stages, control points, and acceptance procedure. Let's discuss your task — we'll return with an estimate and a work plan.