Custom STT dictionary: where recognition breaks
Speech recognition makes errors unevenly: on everyday phrases transcription quality stays high, but on highly specialized terminology it predictably drops.
Models like Whisper are trained on general speech, so when ambiguity arises they choose the more frequent variant — the surname "Kurilo" turns into "kurila" (the Russian word for "she smoked"), an abbreviation is read letter by letter, and a part number in an invoice becomes a random set of digits.
Three classes of entities break systematically. Proper names and toponyms: employee surnames, branch names, rare references. Abbreviations — they sound like a single word that is not in the general dictionary.
And compound terms: legal wording, classifier names, medical and technical names, where the error occurs at word boundaries and the model starts "guessing" by sound.
General settings do not fix this. Confidence threshold and auto-editing only mask the problem by substituting a rare word with a frequent one, and the language model of the general corpus does not know your domain and will not learn it from a single recording.
Corrections in the finished text do not scale: the next file arrives with the same errors, and distorted entities go into archive search, reports, and summaries — the share of correctly recognized entities remains low.
This is solved at the decoding level: a custom STT dictionary tells the system the expected words and their pronunciation before the recording is parsed, not after.
Next, we break down how this is covered — where to get the dictionary, how to describe pronunciations, and how to measure that speech recognition errors have actually gone away.
Effect of tuning: fewer errors on terms and names
We take the first measurement before any edits: we transcribe 30–50 minutes of real recordings and manually mark discrepancies.
The baseline is calculated on the same model that will go into production — usually Whisper or its fine-tuned branch — otherwise the before-and-after comparison proves nothing.
We calculate not overall accuracy but the share of errors specifically on domain words: terms, abbreviations, product names, and surnames.
After connecting the custom dictionary, we repeat the same measurement on the same set of recordings and on new ones. We look not at the average number but at the spread across speakers and recording conditions — it shows where the dictionary works and where it pulls in extra.
- Error rate on domain-specific vocabulary drops by 2–4 times; we separately check terms that were previously always recognized incorrectly.
- The volume of manual corrections after transcription decreases severalfold, and on typical recordings within the domain it reaches zero.
- Robustness to speakers: the term is correctly recognized across 8–10 different voices, including female and low male ones, without tuning to a single speaker.
- Robustness to conditions: we test recordings from a phone, headset, and meeting room — the drop should not exceed a few percent.
- False positives: the dictionary does not pull in similar common words — we monitor this on a set without the domain.
- Regression when updating the model: re-measure after each update, otherwise accuracy quietly slides back.
Recognition tuning methods: comparison of approaches
Tuning to the customer's vocabulary is handled in five ways — from external rules to adapting the model on domain recordings.
They differ in the point of intervention: acoustics, language model, or finished text — hence different timelines, reversibility, and risk of regression on ordinary speech.
In projects we combine two or three layers: domain language model tuning, a replacement dictionary for abbreviations, and post-processing for part numbers and units of measurement.
We make the decision based on the error rate on domain-specific terms, not on an overall metric — gains on vocabulary can easily degrade recognition of everyday speech.
| Method | When it fits | Limitations |
|---|---|---|
| Replacement dictionary | stable abbreviations, names, part numbers | does not help if the model hears the word differently; false positives with dozens of rules |
| Phonetic hints | rare names, toponyms, nonstandard pronunciation | requires pronunciation analysis, confusion with homophones |
| Adaptation on a recording corpus | many labeled hours, stable domain | requires data and labeling, risk of overfitting, long validation cycle |
| Transcript post-processing | formats, punctuation, normalization of numbers and dates | does not restore what the model did not hear |
| Domain language model tuning | industry vocabulary in general | requires a text corpus, does not change acoustics |
We fix the choice based on measurements on a blind sample of the customer's recordings: error rate on terms, number of false replacements, time to add a word. We start with reversible layers; we connect corpus adaptation when there remain errors that cannot be closed by rules.
Implementation stages: from terminology collection to handover
Editing the dictionary without measurements turns into guessing: a term was added, the error rate did not change. So we go in reverse order — first we count where recognition breaks, then we touch the vocabulary.
Steps and checks
- Recognition error audit. We label transcripts from real channels by type: unfamiliar term, abbreviation, Latin script inside a phrase, homonymy. The output is a frequency table by class.
- Collection of the customer's terminology. Interviews with domain experts, exports from reference books. For each term — pronunciation, acceptable variants, scope of applicability.
- Test sample of recordings. We set aside a labeled set that is not used in tuning: different speakers, channels, noise. Otherwise, fitting to examples cannot be distinguished from improvement.
- Dictionary tuning scheme. We decide where to enter the term — pronunciation dictionary, output replacement, language model — and with what weight.
- Dictionary tuning and validation. After each iteration, we run the held-out recordings: we look at the overall error rate and new false positives.
- Handover of the solution. We provide the dictionary configuration, edit regulations, labeling, and reports, and train the team to add terms without regressions.
After handover, the dictionary lives in your environment: edits follow the regulations, metrics are reproducible, and rollback to the previous version takes minutes.
Implementation artifacts: term dictionaries, replacement rules, regulations
After implementation, the customer is left not with a "black-box model" but with a set of artifacts they can work with without us: edits are made in-house, system behavior is reproducible, and disputed recognitions are resolved by document, not from the developer's memory.
- Dictionary configuration — a file connected to the recognition engine: term weights, case, word forms, acceptable spelling variants. One edit changes the output across the entire stream of recordings.
- List of terms — a business-approved list: product names, internal system designations, job titles, Latin script that is dictated aloud.
- Replacement and pronunciation rules — a "how it sounds → how we write it" table, homophones, and persistent distortions of specific speakers; applied on top of recognition and does not corrupt the original transcript.
- Control set of recordings — audio fragments with reference transcripts, including noisy tracks and fast speech. Running it shows that the edit did not break anything else.
- Validation scripts and metrics — calculation of the error rate by terms and a report before and after changing the dictionary.
- Documentation — how to add a term, which fields are required, what to do in case of a rule conflict.
- Update regulations — who initiates an edit, who confirms it, when the change goes into production, and how it is rolled back.
The set turns dictionary support from a one-time setup into a managed process: a new term passes validation on the control set and reaches the production environment without our involvement.
Case study: how a class of errors on highly specialized terms was eliminated
The task came from a design institute: technical councils and acceptance meetings are recorded, but transcripts were still prepared manually. The speech contains abbreviations, equipment brands, standard numbers, and contractor surnames.
The base open model Whisper confidently handled general language but stumbled specifically on this vocabulary.
First, we fixed the baseline: a control set of recordings with manual labeling of terms.
The error rate on domain-specific vocabulary was 17–19%, the editor spent about 40 minutes per hour of audio, and the finished text for an hour of recording appeared only by the end of the day.
The dictionary was not collected "by ear": we exported frequent terms from the corpus and added pronunciation variants — Latin and Cyrillic, inflections, continuous and separate spelling, abbreviations that are dictated letter by letter.
We separately configured term weights so that a rare but important name would not be displaced by a frequent common word, and added normalization after recognition: a single glossary instead of three spellings of one brand.
After tuning, the error rate on domain-specific vocabulary dropped to 3%, manual edits decreased by about four times, and an hour of recording turns into a finished transcript in 40 minutes instead of four hours.
We also checked the opposite — that quality on general language did not drop: the dictionary extends the model rather than replacing it with forced substitutions.
The class of errors with distorted equipment and standard names disappeared, and the editor corrects meaning rather than guessing terms.
The dictionary lives on: new words enter it from the editor's corrections, and the regression set prevents improvements on one topic from breaking the others.
How does recognition tuning differ from model fine-tuning?
These are two different classes of work, and confusing them is costly. Vocabulary tuning changes only the word recognition layer: terms, abbreviations, and product names are added to the dictionary, and weights and normalization rules are set. The acoustic model weights are not touched — the result is predictable and reversible.
Fine-tuning changes the model itself on a labeled speech corpus. It is justified where the mismatch is not vocabulary but pronunciation: a strong accent, a narrow subject area with persistent jargon, recording through a specific microphone or communication channel.
| Approach | When it fits | Limitations |
|---|---|---|
| Custom STT dictionary and weights | New terms, abbreviations, names — the word is known, pronunciation is standard | Does not help if the word sounds unintelligible |
| Replacement rules and normalization | Post-processing: a unified format for numbers, dates, part numbers | Works on top of recognition, does not improve the sound |
| Model fine-tuning | Accent, noisy channel, persistent jargon, a lot of same-type speech | Requires a labeled corpus, possible degradation on previous topics |
Support determines the choice. The dictionary is configuration: edits are made by a domain specialist, changes are versioned, rollback takes one revision, and validation runs against a set of reference phrases.
A fine-tuned model cannot be rolled back that way: it requires a saved corpus, a repeated validation run, and retraining with every domain shift.
Practical rule: first we solve the task with the dictionary and weights, and only if the error rate on the reference sample does not drop do we move to fine-tuning. The solution remains manageable, and ownership of it is predictable.
Technical doubts: scope of work, timelines, support
Objections about tuning a recognition dictionary repeat from project to project: the scope of work, implementation timelines, quality of source recordings, and what happens after launch are unclear.
Below are answers with the sequence of actions, measurements, and boundaries beyond which we do not guarantee the result.
How much work is actually involved in tuning the dictionary?
The scope is determined not by the dictionary size but by the domain structure. With 200–300 same-type terms, most of the time goes to pronunciation variants rather than labeling, while abbreviations, Latin script, and compound names add work on text normalization.
How long until results?
We start with a run on part of the archive: measuring the error rate before and after, and reviewing terms that are still missed. Then we process the rest of the corpus. The timeline depends on the volume and quality of recordings, not on the number of words in the dictionary.
What if the recordings are noisy?
The quality of the source sets the ceiling: on overlapping speech and a telephone channel, the dictionary will pull out terminology but will not restore what was lost.
Therefore, we first evaluate the recordings and, if necessary, set up preprocessing — noise reduction and speech segment separation.
Do we have to rebuild everything with every terminology edit, and what about support?
No: the dictionary and language model are separated from the acoustic part. Adding a term or editing a pronunciation is a dictionary update and a run on the control set, without full retraining; acoustics are touched only when the domain or communication channel changes.
After launch, we work on request: a regression set of recordings, dictionary versioning, error rate measurement, and rollback if things get worse.
Let's discuss the task: what to send for evaluation and a work plan
To come back with an estimate and work plan rather than clarifying questions, we need just one set of source data. The more accurately the domain is described, the fewer iterations will be needed to agree on the solution scheme.
- Audio from the real channel. 30–60 minutes of recordings per source: telephony, microphone, video call — with typical noise, multiple speakers, and accents.
- Reference transcripts. 10–20 minutes of verified texts with notes on disputed places — we use them to calculate the baseline error rate and see where the dictionary will provide the main gain.
- List of terms. Product names, abbreviations, job titles, full names, toponyms — with pronunciation variants and how they should appear in the output.
- Normalization rules. Case, punctuation, numbers, units of measurement, Latin and Cyrillic in one word, formatting of repetitions and filler words.
- Flow diagram. Where the audio comes from and in what format, where it is stored, where the text goes, and whether speaker labeling is needed.
- Constraints. Requirements for output latency, where data processing is allowed, and the acceptable share of manual editing.
- Error examples. 20–50 real phrases that are currently recognized incorrectly — this is enough to estimate the scope of work with the dictionary.
It is not necessary to send everything at once: we will start with the audio and the list of terms, and clarify the rest along the way. Describe the task — we will come back with an estimate, solution scheme, and work plan.