Fine-tuning Whisper: where a ready-made model fails on your domain
A ready-made recognition model is trained on averaged speech: clean studio recordings, neutral vocabulary, standard literary language. In the customer's domain, everything is different — terms, abbreviations, product names, and SKUs that were absent from the training set.
The language part of the model confidently "fills in" a familiar word instead of the one it heard, and the text retains a plausible but incorrect wording.
Failure classes repeat again and again. An industry term is replaced by an everyday one, an abbreviation is expanded at random, a Latin product name turns into a set of letters.
A separate case is language mixing: one English phrase inside a Russian utterance switches the model, and it translates the rest.
Noisy recordings add a second layer of errors. On material from a workshop, showroom, or open office, the model segments utterances worse, loses endings, and merges words from adjacent speakers.
Here fine-tuning Whisper gives the greatest gain: the model gains a domain vocabulary and robustness to imperfect sound.
The cost of errors is not in one line. An incorrect term breaks archive search, disrupts automatic export to a report, and forces an editor to listen to the recording again. Therefore, the share of lines edited manually grows with the density of industry vocabulary.
Next — why exactly the universal model stumbles on the domain and how recognition of industry vocabulary is addressed: a custom vocabulary, fine-tuning on your recordings, and measuring transcription errors before and after.
What model adaptation delivers: measurable changes
The base model is trained on general speech and makes predictable errors on domain vocabulary: it replaces abbreviations, product names, SKUs, and surnames with similar-sounding words.
The errors are not spread across the text — they are concentrated in one class of terms, and they consume most of the editing time. After fine-tuning Whisper on the customer's annotated recordings, measurable values change, not "quality in general".
What we record before and after — on a holdout set from the same recordings:
- Share of correctly recognized terms — grows from 70–80% to 95%+ on the vocabulary that was previously edited manually.
- Transcription editing time — drops from 30–40 to 5–10 minutes per hour of recording: the editor checks meaning rather than guessing words.
- Error rate on rare vocabulary — disappears as a class: similar-sounding terms stop being confused with one another.
- Errors in numbers and SKUs — fall to a level where spot checks are sufficient.
- Volume of manual proofreading — is reduced severalfold, and it can be assigned to a proofreader rather than a domain specialist.
- Robustness on poor tracks — recordings with noise, accent, and interruptions give the same accuracy as clean ones.
We measure metrics on a set the model did not see during training: otherwise the gain remains in the report rather than in actual work. Separately, we count how many terms from the customer's vocabulary remain problematic — this list becomes the plan for the next iteration.
Approaches to fine-tuning Whisper: comparison of modes
Adapting a recognizer to domain vocabulary rarely starts with retraining. First, decoding settings and hint dictionaries are tested: they do not change the model weights and can be disabled with one configuration line.
If the word error rate remains high on accented dictation or a recording from a noisy room, a different lever is needed — and it already requires annotated domain hours and its own validation loop.
| Mode | When it fits | Limitations |
|---|---|---|
| Decoding settings, term dictionary, and hints | Narrow vocabulary: abbreviations, names, place names; few recordings, clean channel | Does not correct acoustic errors; dictionary is maintained manually; hint length is limited |
| Acoustic part adaptation | Several hours of annotated recordings, stable channel and microphone | Will not learn unfamiliar word forms; sensitive to annotation quality |
| Retraining on your own data | Tens of hours of annotation, narrow domain, high cost of error | Long cycle; risk of overfitting and loss of base quality; requires your own validation |
| Hybrid: dictionary plus acoustic tuning | Domain is growing, terms are constantly added | Two support loops and the link between them |
How we choose a mode
The decision is determined by the structure of errors, not the model size. If the recognizer hears a word correctly but substitutes the wrong term, decoding is enough; if it hears incorrectly across the entire domain, work on acoustics is needed.
A mixed picture, typical of vocabulary from several languages, is addressed by a hybrid: the dictionary handles rare forms, and acoustic tuning pulls up high-frequency ones.
Work stages: from audio audit to model handover
We build the process around one principle: any model change is confirmed by numbers on recordings it has not seen.
That is why fine-tuning Whisper begins not with launching training but with analyzing current quality and preparing the corpus — otherwise it is unclear whether things got better and due to what.
- Analysis of current quality. We run the customer's recordings through the base model, calculate the error rate on meaningful terms, and label typical failures: slang, abbreviations, product names, background noise, blended speech.
- Audio corpus collection. We select recordings by scenario — phone line, room microphone, meetings — and fill in what is missing: short utterances, rare words, different speakers.
- Audio corpus annotation. Transcription with quality control: a unified term glossary, agreed rules for numbers, names, and Latin script, and selective re-checking of disputed fragments by a second engineer.
- Adaptation scheme. We determine what we are tuning — domain vocabulary, channel acoustics, or both directions at once — and lock the term dictionary so the model does not "drift" on familiar words.
- Training and iterations. We run series on the annotated corpus, keep general-domain recordings in the mix to prevent overfitting, and measure the result after each series.
- Validation on a holdout set. A set of recordings that did not participate in training shows real gains on terms and typical phrases, not on familiar examples.
- Quality acceptance and handover. We agree on acceptance criteria for the customer's scenarios and hand over configurations, launch instructions, and the procedure for fine-tuning on new recordings.
The acceptance criterion is not average intelligibility but the share of correctly recognized terms and commands on holdout data. Such a measurement is visible in specific examples: previously a term was recognized as a similar word, now it is correct, even on a noisy recording.
What we hand over: artifacts and regulations
Handover is not an archive with weights but a working bundle: the model, domain dictionary, and normalization rules by which it prepares text. Below is what exactly remains in your environment and which documents support it.
- A tuned model after fine-tuning on Whisper with a quality report: error rate on the test set of recordings before and after, broken down by vocabulary type — terms, abbreviations, proper names.
- A domain dictionary in an editable format: terms, abbreviations, product and job titles, transliteration rules. Your specialist can edit it without contacting us.
- Normalization rules: how numbers, dates, units of measurement, and forms of address are formatted, and what is intentionally kept as spoken — so that transcripts read consistently across all systems.
- Deployment schemes: where the model lives, what access channels exist, how a new build is rolled out without stopping the service, and how to roll back to the previous one.
- Data access policies: which employees can see recordings and transcripts, retention periods, anonymization before training, and an audit log of archive access.
- Model documentation: model card, scope of applicability, known weaknesses, instructions for launch and diagnosis of typical failures.
- Dictionary update procedure: who brings new terms, how often, and how we verify that the update did not degrade quality on the test set.
We write the regulations for your processes, not for team convenience: you can update the dictionary and retrain on your own, while we remain on change support. Describe the task — we will come back with a list of artifacts and a handover plan.
Case study: how recognition of industry terms changed
The customer was a service company with a distributed support service: conversations with clients had been recorded for years, but there was no way to extract meaning from the archive.
Industry vocabulary — SKUs, abbreviations, names of components and equipment models — was difficult for the base Whisper model: instead of the term, it produced a similar-sounding common word, and the utterance lost its meaning.
What measurements showed before fine-tuning
On a test set from the archive, the share of correctly recognized industry terms stayed around 62%, and the errors were systematic: the same component names were recognized incorrectly in nine out of ten cases.
Processing the recording archive was slow — transcribing 24 hours of audio took almost two days, so ticket review lagged by weeks, and operators handled disputed fragments manually.
What we changed and what we achieved
We assembled a dictionary of industry terms and their pronunciation variants, selected recordings with verified transcripts, and fine-tuned the model on this corpus.
Dictionary hints were connected at the recognition stage, and quality was measured on the holdout part of the archive that did not participate in training — so that the metric reflected real domain accuracy rather than fitting to training examples.
The share of correct terms on the same test set grew to 94%, and on recordings with strong background noise — to 88%: the model stopped confusing component names with one another.
Transcription of 24 hours of audio dropped to several hours, and the share of recordings with manual edits fell from one third to one tenth — operators focus on handling tickets rather than proofreading text.
How does model adaptation differ from dictionaries and post-processing?
External edits work on top of the recognition result: hint dictionaries bias the model's choice at the decoding stage, while text post-processing changes an already finished string — rule-based replacement, lookup in a term reference.
The model itself does not change: what is tuned is not how it hears, but what we do with its answer.
The limits of external edits are visible where the error depends on the meaning of the phrase.
A rule works with a string and does not know what is being discussed: a term that is correct in one sentence is wrong in the next, so rule-based replacement either always fires or never does. The cases can be distinguished only by context, which the rule does not have.
The second limit is the accumulation of exceptions. Rules are added point by point, then begin to intersect, the order of application becomes significant, and false positives appear where the text was recognized correctly. Maintaining such a list grows faster than the term dictionary.
The third limit is lost information. If similar-sounding terms are not acoustically distinguishable, a hint only increases the probability of the desired variant but does not recover what was missed.
Post-processing is more dangerous here: it can replace a correct word with an incorrect one — unnoticed.
Fine-tuning Whisper on domain vocabulary shifts the distribution itself: the model distinguishes terms by sound and context, not by a list of exceptions, and works confidently on formulations that were not in the rules. External edits remain, but for format normalization rather than as the main accuracy mechanism.
Technical concerns: data, quality, support
The main concern about fine-tuning Whisper is "we don't have thousands of hours of recordings." Domain adaptation does not need that many: 3–5 hours of clean speech with accurate transcription work, where exactly the terms, abbreviations, and product names are annotated, plus a text vocabulary dictionary.
Annotation quality weighs more than volume — an hour of verified text gives a greater gain on domain words than ten hours of "approximate" transcription.
The second risk is that the model loses its understanding of ordinary speech. That is why we do not rebuild it entirely but fine-tune a small add-on layer on top of the base model.
Every run includes a test set of general speech: if its error rate exceeds the agreed threshold, the update is not accepted, even when domain metrics have improved.
Vocabulary updates are a regular procedure, not a one-time job. New terms appear: we collect recordings where recognition made an error, annotate them further, run a short fine-tuning, and compare against test sets.
Everything stays inside your closed environment — recordings, annotations, and training do not leave the infrastructure; only weights and quality metrics go outside.
Can you train without exporting data outside?
Yes. The environment is deployed on your infrastructure or in your isolated segment, engineer access is through agreed accounts and is logged. Transfer of audio recordings and transcripts to third parties is not required: you receive updated weights and a metrics report.
What if quality on general speech still degrades?
We keep the base model untouched, so rollback is switching to it, without retraining. Every release is accompanied by comparison on general and domain sets, so degradation is visible before deployment, not after.
How often should the model be updated for new vocabulary?
Usually once a quarter or after a noticeable number of errors on new terms accumulate. The cycle is short: collect disputed recordings, annotate further, fine-tune the layer, validate on test sets, and release.
Let's discuss your task and come back with an adaptation plan
The first review takes one meeting and is based on your data, not general presentations. You describe the task: where the recordings come from, how many hours of audio, which terms, abbreviations, and product names the model currently distorts.
Separately, we look at how your specialists edit transcripts manually — these edits become the annotation by which quality is calculated.
On the customer's side, we need samples of real recordings with verified texts, a list of domain vocabulary, and access to the environment where the fine-tuned model will run. After that, we provide a document that allows a decision to be made without further clarification.
- Project estimate: how many hours of audio are needed for fine-tuning Whisper, how many iterations we plan, and what reduction in word error rate on the test set we consider a result.
- Data preparation scheme: recording cleanup, segmentation alignment, term annotation, selection of difficult fragments into a separate validation set.
- Work order: stages, what we do at each, where your validation is needed, and at which step you can stop without losing what has already been done.
- Pilot run plan: we fine-tune on your vocabulary, show transcription of test recordings alongside current quality, and only after that launch the full cycle.
- Requirements for access and infrastructure: where audio is stored, how it is transferred, and how it is deleted after training.
- Acceptance criteria: a set of test recordings, metrics, and the method for recalculating them by your engineer.
Describe the task and send a couple of representative recordings — we will come back with an estimate, scheme, and work plan, including the boundaries of the pilot run.