Why hotword boosting does not provide gains on real recordings
Hotword boosting adds weight to the desired word at the decoding stage, and on studio dictation this looks like an increase in accuracy.
On real recordings, the gain disappears: the acoustic model does not distinguish the phonemic composition of a rare term, and the boost only redistributes probability between similar hypotheses. The term remains incorrect — the model just makes mistakes more confidently. Toolkits like NVIDIA NeMo provide ready-made biasing, but quality does not rest on it.
The first bottleneck is the pronunciation dictionary.
The lexicon is usually assembled from a single spelling variant, whereas in speech a rare term sounds in cases, merges with a preposition, is pronounced in Latin script inside a Russian phrase, or is distorted by the speaker's accent.
If none of the variants is described, the boost amplifies a non-existent phoneme sequence — and instead of an omission, a substitution appears.
The second is signal quality. A distant microphone, a conference room with reverberation, a call with packet loss, and the merged speech of two speakers simultaneously reduce intelligibility even before biasing kicks in.
Weights at the decoder level do not restore what is lost in the spectrum: transcription errors in such fragments remain, and rare terms in them are the first candidates for distortion.
The third reason is the metric. The share of rare terms in the stream is a few percent, so the aggregate speech recognition error barely moves, and the tuning looks useless.
Plus, an inflated weight causes false positives: the term is inserted where it was not present and spoils already correct sections.
This is solved not by tweaking the global weight, but by a pronunciation dictionary, contextual limitation of biasing, and validation on domain recordings.
We will break down how this is arranged: what we add to the lexicon, how we limit activations, and what we use to measure the result.
What controlled keyword boosting provides in recognition
Controlled keyword boosting (hotword boosting) works at the decoding stage: a list of domain lexemes and their weights shift probabilities during graph search, so a part number, drug name, or toponym is recognized with equal confidence — regardless of diction, accent, and channel quality.
The list lives in a separate configuration, not in code: the dictionary is edited without rebuilding the model or an unscheduled release.
The effect is measurable on a dedicated set of phrases.
On several hundred domain terms, the recognition error rate drops from 18–25% to 4–6%, and manual transcript edits are reduced by a factor of three: the operator checks the meaning of what was said rather than reconstructing words by sound.
Recognition stability is maintained both on recordings with background noise and on narrowband channels — the metric does not degrade on the 'tail' of the audio.
The key weight is capped and checked on a control sample without keywords. If you raise it without such a check, the model starts substituting the term where it was not spoken: the false positive rate grows faster than the miss rate falls.
Therefore, we select weights on labeled validation, not 'by ear', and keep them in the configuration next to the replacement dictionary.
When domain terms stop being lost, the process scheme also changes. The share of requests closed without an operator grows because transcript search finds the needed object by its exact name, not a distorted one.
We take the base from the open-source Whisper, and we write our own boosting layer and post-processing — normalization of numbers and dates, replacement dictionary, speaker diarization — so that quality on domain vocabulary does not depend on third-party updates.
Formats of hotword boosting: dictionary, weights, and model fine-tuning
All three methods address one pain point: rare domain words — drug names, part numbers, toponyms, abbreviations — are heard by the model as similar frequent words and substituted.
The difference is where exactly knowledge about terminology goes: into the lexicon at the recognition stage, into weights when producing the result, or into the model itself. Hence different data requirements, implementation time, and cost of error.
Where knowledge about terms is introduced
The choice is usually dictated by the size and variability of the dictionary.
While there are dozens of terms and they change once a quarter, tuning hotword boosting at the inference stage is enough; when it is thousands of words and an entire industry language, fine-tuning is indispensable. Below is a comparison by applicability and limitations.
| Approach | When it fits | Limitations |
|---|---|---|
| Term dictionary | Up to several hundred words, updates without retraining | Will not override a confident model error, false positives on similarly sounding words |
| Keyword weights | Terms are known in advance, there are primary and secondary ones | Calibrated for a specific stream and recording quality, changing the channel requires re-tuning |
| Model fine-tuning | Thousands of terms, a narrow industry, there are labeled recordings and texts | Longer implementation, need data and retraining when terminology changes, risk of degradation on general vocabulary |
The approaches are not mutually exclusive: the lexicon and weights cover the majority of cases and allow terminology to be updated even weekly; fine-tuning adds robustness where the dictionary hits acoustic limits.
We start by measuring the error rate on your set of terms — it shows whether inference tuning is enough or data work is needed.
How we implement keyword boosting: stages of work
Keyword boosting does not start with editing the dictionary, but with measuring what the system hears on your recordings: without a baseline result on a real corpus, tuning turns into guesswork. Then the work proceeds from audit to acceptance, and each step has a measurable criterion.
- Recognition audit. We export recordings from your channel and calculate the error rate by entity types: names, part numbers, industry terms. The output is a list of words the model consistently fails to recognize.
- Recording labeling. We label transcripts manually: where the reference is, where an error is, where an ambiguous case is. From this we assemble two non-overlapping sets — working and held-out.
- Boosting tuning. We select the form of hints: phrase dictionary, phonemic spelling variants, weights, activation thresholds. We ensure that growth on target words does not cause false positives on similarly sounding words.
- Quality check on the held-out set. We run the configuration on recordings that were not present during tuning and compare with the baseline measurement by two values: the share of correct activations and the share of false ones.
- Iterations to plateau. We change parameters one at a time, record each contribution, and stop when the next step yields an improvement within the margin of error.
- Solution acceptance. We agree on criteria in advance: for which words and on which set the task is considered solved. Acceptance is performed on recordings that did not participate in tuning.
- Update procedure. We hand over to the team the procedure: how to add a word, what to check, by what thresholds to roll back the configuration, and how often to review the dictionary.
Handover is not an archive of files, but a working procedure by which your team can further expand the dictionary itself. Without it, boosting degrades on the very first update.
Delivery composition: configurations, dictionaries, and procedures
Solution handover is not an archive with a model, but a working kit: the service is deployed from the same files as on our test bench, without manual 'on-site' edits. Below is what exactly goes to the customer and in what form.
- Service configuration — recognition parameters, boost weights, and activation thresholds per domain separately; profiles for streaming and file modes, default values fixed in one place.
- Term dictionary — base and domain, with pronunciation variants, word forms, and priority marks; the table format is agreed with your team so edits can be made without our involvement.
- Request processing scheme — the audio path from reception to final hypothesis: where hotword boosting turns on, where re-ranking occurs, what timeouts exist, and what happens when the dictionary is unavailable.
- Update procedure — triggers (new product, operator complaints, accuracy drop on the control set), verification order, and rollback to the previous dictionary version.
- Control sets — recordings with labels and reference answers; they show that the boost raised the needed words and did not break general ones.
- Support documentation — how to add a word, how to read logs, which deviations are fixed by tuning and which require engineer intervention.
- Solution handover — access, deployment in your environment, walkthrough with the team, and an escalation point with a fixed response time.
Case study: how recognition quality improved in a contact center
The contact center «Ладога Телеком» receives about 25,000 incoming calls per month at 40 operator workstations.
Transcripts were used for quality control and search by requests, but domain entities — tariff names, part numbers, abbreviations — were recognized inconsistently, and analysts manually proofread dialogues.
The implementation case is built on before-and-after metrics on the same dataset: a held-out set of recordings — 4,000 calls with manual labeling that did not participate in tuning.
Before the work, the error rate in key entities was 22%, the overall word-level error rate was 19%, and product names were recognized correctly in only 61% of cases.
What was changed: we assembled a dictionary of 1,200 domain entities and assigned weights — high priority to product names, contract names, and abbreviations, zero to function words to avoid growing false positives.
For each entity, we added phonetic variants and spellings in two alphabets: operators dictate and pronounce them differently.
After tuning, the error rate in key entities decreased from 22% to 5%, the overall error rate dropped to 11%, and product names are recognized correctly in 93% of cases. Transcript proofreading was halved, and search began finding calls by tariff name, not only by request number.
The result was confirmed by an external measurement on a second, independently labeled sample: the discrepancy with the original metrics was no more than two percentage points.
What to do if you already have your own recognition service deployed?
An already running recognition service does not need to be rebuilt from scratch: hotword boosting is embedded at the decoding stage, on top of the current acoustic model.
We do not touch the acoustics and do not retrain the network — we add a contextual list of domain terms with weights that increase the probability of the desired hypotheses and reduce competition from similarly sounding words.
We check compatibility with your environment first. Additional tuning does not require external services: dictionaries and rules live locally, recognition runs on the same compute resources as now, and lists are loaded dynamically for a domain, client, or specific session. In a closed perimeter, audio and text do not leave your environment.
The implementation order is built around measurability. First, we collect a corpus of real recordings with labels and record baseline metrics: the share of correctly recognized terms and the number of false insertions.
Then we select boost weights and activation thresholds for each word — excessive boost confidently catches terms, but starts inserting them where they were not present.
Next is validation on a held-out set that the model has not seen, and rollout to part of the traffic with line-by-line comparison against current quality.
The configuration is versioned: if the false positive rate grows, we revert to the previous set of weights with one switch, without rebuilding or service downtime. This is exactly careful model tuning — reversible and verifiable at every step.
Describe which existing recognition service you have running and which terms it loses — we will come back with an assessment and a tuning plan for your environment.
Doubts and limitations: engineers' questions
Hotword boosting addresses a narrow class of tasks: rare names, part numbers, abbreviations, drug names that the general model hears as similar frequent words.
The approach has limitations, and it is better to know about them before starting — otherwise some expectations will have to be rebuilt during the project.
Below are four questions most often heard at the evaluation stage: the amount of labeling, behavior on unfamiliar words, the method's boundaries, and self-maintenance of the dictionary.
How much labeling is needed for the boost to work?
For the basic mode — zero: the list of terms with weights is set manually and works at the decoding stage. Labeling is needed only for fine-tuning acoustics for an accent or noise — a few dozen occurrences per term and negative examples where the word is absent are enough.
What happens to a term that was not in the training set?
The term is added to the boost dictionary and receives an increased weight during hypothesis search — model retraining is not required. Only the pronunciation for non-standard reading is set manually: abbreviations, Latin script in Cyrillic speech, compound part numbers.
Where are the method's boundaries and what breaks with greedy tuning?
The main risk is false positives: an inflated weight pulls the term in where a similar ordinary word sounds. The weight is set with a cap, checked on recordings without the target vocabulary, and revised when new terms compete for the same phonetic image.
Can the dictionary be maintained by your own team after launch?
Yes: the dictionary is a table of terms with weights and pronunciation variants; it is stored in the repository next to the recognition configuration.
Adding a word takes minutes and does not require rebuilding the model; once a quarter it is useful to run regression on the control set and compare the false positive rate.
Let's discuss the task: what to send for assessment
An assessment based on the wording 'we need hotword boosting' always gives a wide range — it is determined by the quality of your recordings and the number of domain words that need to be retained.
To come back with an implementation plan rather than a random range, we need a snapshot of current recognition; some items can be covered approximately.
- Sample recordings — 20–50 fragments of real audio across different scenarios: telephone channel, conference room microphone, noisy room.
- Reference transcripts for at least part of the recordings: without them, it is impossible to calculate how the error rate on target words changes after boosting.
- Entity dictionary — nomenclature, names, abbreviations, part numbers, including pronunciation variants and frequent distortions.
- Current metrics — error rate on domain words, share of recordings with recognition failure, request response time.
- Operating environment — streaming recognition or file processing, cloud or closed perimeter, latency constraints.
- Data requirements — what can be moved to a test environment, what remains internal only, who has access to recordings.
- Priorities and acceptance — what is more important: accuracy on nomenclature or response speed, and who on your side will confirm the result.
Based on this set, we will come back with a work estimate and implementation plan: on what data we calibrate the boost, how we verify the effect on your recordings, and what steps lead to a production environment. Describe the task — we will discuss it substantively and propose an order of work.