Whisper fine-tuning: where domain speech recognition breaks down
The base model is trained on a broad array of everyday speech: dictation, correspondence, lectures, podcasts.
On industry material it does not "not hear" — it confidently substitutes similar-sounding common words instead of terms, and on abbreviations and Latin names it loses phrase structure entirely.
Errors in narrow terms are distributed unevenly: the rarer a word occurred in the training array, the higher the chance of substitution. But transcription is needed precisely for these words — part designations, SKUs, drug names, legal wording.
At the same time, the quality of transcribing recordings looks acceptable on average: the aggregate metric across the entire text is normal, while a specialist listens to critical fragments manually because they cannot be trusted.
The second layer is acoustics.
Recordings from noisy rooms, a distant microphone, several speakers, regional accents, and connected speech produce the same picture: the model smooths out word beginnings and endings, glues remarks together, and incorrectly places segment boundaries.
This is exactly where domain Whisper fine-tuning gives the greatest gain: domain speech recognition is tuned to real sound, not to a studio reference.
Let's break down how this is handled: a domain corpus of recordings, a replacement dictionary for terms and names, noise augmentation, and evaluation by the share of errors specifically in industry words, not by the average across the text.
Effect of fine-tuning: fewer errors in terms and noise
We measure the effect on a held-out sample — recordings that did not go into training. Otherwise the metric shows memorization of the corpus, not generalization.
We compare the error rate on everyday and domain speech: the gap between these figures is the part that speech recognition model fine-tuning covers.
The main gain is on industry vocabulary. The base Whisper on long compound terms, abbreviations, and Latin script inside a Russian phrase substitutes similar-sounding words.
After fine-tuning, such substitutions go away: the model assembles the term as a whole, and the reduction in error rate is uneven — on general text by a few percentage points, on specialized text noticeably more.
Robustness to noise and accent grows due to the diversity of recordings. If the corpus includes a workshop, street, conversation through a headset, and several speakers with different speaking styles, the model stops "falling apart" on one type of conditions.
This is more important than average accuracy: the system remains operational where the recording is initially far from studio quality.
The practical count is not only the error rate but also the volume of manual editing. We look at the share of segments that the editor leaves unchanged and the distribution of edits by type.
Most often, the class of term-substitution errors goes away — it is the one that consumes the most time during proofreading. What remains are punctuation and proper names that were not in the training data.
Local edits, hint dictionary, and fine-tuning: what to choose
Domain speech breaks the model unevenly: in some places the acoustics match, and only the spellings of terms and abbreviations diverge; in others the pattern itself changes — the dispatcher's connected tempo, Latin script inside a Russian phrase, a narrowband microphone.
Therefore adaptation is built in layers and chosen by the source of the error, not by the external solidity of the method.
The starting point is measurement: transcribe a control sample, label errors, and divide them into acoustic and orthographic. The ratio of these groups shows which layer will deliver returns.
| Option | When it fits | Limitations |
|---|---|---|
| Post-processing of transcripts | Errors are predictable and boil down to spelling: terms, SKUs, names | Does not fix acoustics: if a word is misheard, a rule is powerless. The number of rules and false positives grows |
| Dictionary of terms and hints | A long list of names and abbreviations that changes without rebuilding the model | Retains only similar-sounding words; an overloaded dictionary blurs attention and drops accuracy on the rest of the speech |
| Fine-tuning on a corpus of recordings | The domain acoustics are stable, and there are labeled hours of recordings with text | A corpus and labeling are needed; risk of overfitting to a narrow scenario; a repeatable training loop is required |
| Combination of layers | The domain is live: dictionary for new terms, fine-tuning for acoustics, post-processing for output format | More expensive to maintain: layers are versioned and tested together, and editing one breaks the metrics of another |
What is chosen in practice
Post-processing and a dictionary deliver returns when orthographic errors predominate: edits live in the pipeline and do not require rebuilding the model.
Whisper fine-tuning is chosen for persistent acoustic errors and an available labeled corpus: the weights are adjusted to the domain, but along with them comes the obligation to store the corpus, maintain a repeatable training loop, and keep metrics on a control set.
Without this, rolling back to the base version is impossible, and degradation on general speech manifests unnoticeably.
Project stages: from recording audit to model handover
The route from a recording archive to a model in operation is divided into stages with measurable acceptance criteria. If the metric on the control set does not improve, we return a step back — usually to the labeling scheme.
- Recording archive audit. We listen to a sample, count total duration, share of clean channels, types of noise, and speech overlaps. The result is a data map: which scenarios are missing for training.
- Labeling scheme and glossary. We fix transcription rules and the spelling of abbreviations and names so annotators do not diverge in disputed places.
- Corpus preparation. We bring recordings to a unified format, cut by pauses, remove duplicates, and split the array into training, validation, and held-out parts.
- Audio labeling. A second annotator rechecks a control share of transcripts; discrepancies go back into the glossary rather than remaining in the dataset.
- Fine-tuning Whisper. We compare partial and full fine-tuning, add noise and channel thinning, and monitor the metric at checkpoints.
- Validation on the control set. We count the error rate separately for domain terms: the overall metric may be high, while on the needed vocabulary the model errs most often.
- Model handover to the customer. We hand over weights and configuration, the launch script, a quality passport, and deployment instructions, and train the team on its recordings.
The held-out part is formed before training — otherwise the evaluation proves nothing. A new scenario in operation leads to a return to the labeling stage and fine-tuning on it, without rebuilding the corpus.
Artifacts, regulations, and access after project handover
Project handover does not end with a demonstration of a working model. The customer receives a package with which its engineers can run recognition themselves, add new domain recordings, and repeat fine-tuning — with us or without us. Below is what physically remains in the repository and internal documentation.
- Training and recognition configurations: splitting audio into segments, level normalization, data batch size, learning rate, number of epochs — the current model quality is reproduced from these files.
- Processing route scheme: the path of a recording from upload to finished text — where audio is stored, at which step preparation happens, who sees the transcript and when.
- Access policies for recordings: roles, retention periods, anonymization rules, log of access to audio and recognition results.
- Model documentation: domain dictionary and replacement rules, metrics on the held-out set, list of predictable errors and limitations by speech type.
- Retraining regulation: how to assemble a new corpus, how to make sure quality has not dropped on previous scenarios, and under what conditions an update is rolled back.
- Set of validation recordings and comparison script: a run takes minutes and shows quality drop before rollout to production.
- Operations manual: start, stop, dictionary update, response to typical failures — at the level of an on-duty engineer, without contacting the model authors.
Each item is verifiable: the package is accepted according to a checklist, and the same checklist can be used to conduct acceptance by the internal team.
Case study: support for recognition of narrow terms before and after
A contact center with dense domain vocabulary: tariff names, legal caveats, rare toponyms, and SKUs.
The base Whisper confidently transcribed general speech, but on terms it produced sound-based substitutions — a tariff turned into a similar-sounding word, and the transcript lost its legal meaning.
Editors listened to such fragments manually, and editing consumed more time than the handling of the request itself.
What was measured and what was compared
We assembled a control sample — 38 hours of recordings stratified by communication channel, speech rate, accent, and noise level.
These recordings went into neither training nor model tuning, so the before-and-after comparison ran on the same tracks and was not replaced by memorization of the material.
What changed after fine-tuning
- Share of incorrectly recognized words: 19.4 % → 6.1 %; on terms from the corporate reference — 34 % → 8 %.
- Transcripts sent to the customer without manual editing: 12 % → 58 %.
- Editing time per hour of recording: 26 minutes → 9 minutes.
- Repeated listenings of disputed fragments decreased roughly threefold.
The basis of the result is a domain corpus: about 120 hours of transcribed dialogues brought to a unified labeling scheme, plus a dictionary of rare names and rules for number normalization.
Training preserved the general branch of the model, so quality did not drop on conversational speech without terms — regression was checked with a separate set of everyday recordings.
How does Whisper fine-tuning differ from a hint dictionary?
The hint dictionary works at the decoding stage: a list of terms and contextual phrases shifts probabilities when choosing the next token. It helps when the model hears a word correctly but writes it differently — in Latin script, merged, with the wrong case.
A hint does not change acoustics: if the sound is misparsed, it will only make the model more confidently choose the wrong variant.
The boundaries of post-processing lie where an error can be described by a rule on text: replacing stable combinations, normalizing numbers, dates, and units of measurement, labeling utterances by pauses. As long as the failure is systematic and reproducible in text, this path is cheaper and faster.
But when there are many rules, they begin to conflict with one another, false positives break correct fragments, and homophones are distinguished only by meaning — which a rule does not have.
This is the limitation of the hint dictionary: it does not distinguish sound, it guesses the spelling.
Fine-tuning is needed when errors are acoustic: domain vocabulary, unusual pronunciation, speaker accent, narrow recording channel, names and titles that the model consistently confuses with similar-sounding words.
Here the model itself changes — it is shown "audio — exact transcript" pairs on recordings from your domain, and it rebuilds its internal representations of sound.
In practice, the choice is made by measurement: on a domain set, the word error rate is counted and one looks at what share falls to spelling and what share to recognition. The first is covered by a dictionary and rules, the second only by Whisper fine-tuning.
The usual order is: first squeeze out post-processing, then fine-tune the remainder — otherwise the model will learn what is already treated by text.
Technical concerns: failures, volumes, data security
Engineering concerns around fine-tuning a speech recognition model usually come down to four: how much data is really needed, what to do with an already deployed pipeline, how to protect audio materials, and whether quality degrades over time. Below is how we answer each.
How many hours of recordings are needed for the model to learn domain vocabulary?
The threshold depends on the distance between general speech and your domain. For terminology and accent, 10–40 hours of clean speech with precise labeling is usually enough: beyond that, returns drop and the risk of overfitting grows.
If there are fewer recordings, we start with augmentation and mixing in synthetic data, and measure quality on a held-out set — we do not use it in training in any run.
We already have a deployed model — will it have to be retrained from scratch?
No. Adapters are trained on top of the current model; the base weights remain untouched — this is cheaper and does not break what already works.
The new version first goes into shadow mode: it processes the same stream in parallel, we compare the error rate on the reference set, and only then do we switch the traffic. Rollback is a return to the previous checkpoint.
How is the customer's audio material protected?
Recordings are anonymized before training: personal data is cut out, and if necessary the timbre is changed while preserving phonetics. Data is stored in your environment, access is role-based and only for engineers on the task, and the retention period is fixed by regulation. After handover, the corpus is deleted by act if this is a customer condition.
What about degradation of recognition quality over time?
Speech changes: new terms, products, and names appear. We maintain monitoring of the share of confident recognitions and a monthly run on the reference set — a discrepancy signals drift before users notice it. Fine-tuning on fresh recordings restores metrics without a full rebuild of the pipeline.
Describe your scenario — we will come back with an estimate of the corpus size and a work plan.
Let's discuss your task: what is needed from you to start
To come back with an estimate and project plan rather than general considerations, we need a short technical context.
Fine-tuning Whisper for domain speech (fine-tuning) does not hinge on the choice of architecture but on the representativeness of the data: a studio read-aloud recording will not show how the model behaves on a noisy channel.
The earlier we see real speech, the more accurately we can name the scope of work and the acceptance metric.
It is more convenient to discuss the recognition task using the following set:
- Examples of recordings from the working channel — 20–60 minutes per scenario, as is: with noise, overlaps, varying volume, without preliminary cleanup or editing.
- Transcripts for part of the recordings — a reference by which we will calculate the error rate before and after fine-tuning and show the effect in numbers.
- Customer's dictionary of terms: abbreviations, SKUs, names, toponyms, and how they are pronounced aloud — often these are exactly what account for most errors.
- Usage scenarios: who speaks, in what environment, and what is done with the text afterward — search, minutes, export to an accounting system.
- Environment constraints: a closed environment without external access, batch transcription of archives or processing during conversation, requirements for storing recordings.
- Current quality and what has already been tried: exactly where recognition fails — names, numbers, professional jargon, language mixing.
- The person responsible on your side: who confirms the dictionary and accepts the result according to the agreed metric.
With this set, we will come back with a plan: the volume and composition of data, the fine-tuning scheme, the method of validation on a held-out set, and the implementation order. Describe the task — we will clarify the details and provide an estimate.