TTS fine-tuning: where a ready-made voice breaks
The base voice reads everyday phrases evenly and predictably breaks where your domain begins. Abbreviations fall into letter-by-letter reading, service names and product codes sound like a random set of syllables, and numbers and units are read in the wrong grammatical case.
Long enumerations lose intonation by the third item, and repeated lines are dictated with the same emphasis. It is on these phrases that the share of support requests grows: “the robot mispronounces the service” is heard noticeably more often than a complaint about timbre.
Stress errors are a separate category. The general model places them according to frequency in everyday speech, so industry terms are read at random.
The pronunciation of terms also breaks at junctions: Latin script inside Russian text, abbreviations made of consonants only, compound words. The customer hears not a minor defect but incompetence in their own product.
The reason is technical. The phonetic dictionary and text normalization rules of the base voice are assembled on general vocabulary.
Domain-specific speech synthesis requires something else: its own pronunciation and stress dictionary, rules for reading numbers, ranges, units, and abbreviations, and stress marking in the training corpus.
TTS fine-tuning closes this in a targeted way — the voice is further trained on domain texts, receives an exception dictionary and normalization rules, and the intonation of enumerations is configured separately.
We check naturalness not by ear: blind listening on a set of real utterances from your domain plus an intelligibility assessment. Next, we break down how this is closed.
Measurable effect: intelligibility, failures, response
We check pronunciation quality on an annotated corpus: names, toponyms, abbreviations, numerals, rare inflectional forms. We calculate the share of phrases where stress, pause, or vocalization deviated from the reference — that is the pronunciation error rate.
TTS fine-tuning on customer recordings transfers the domain's phonetics into the model, and errors on atypical words fall by multiples, not just “sounds better overall.”
We measure speech intelligibility in the channels where the solution actually works. A studio signal, a narrowband telephone channel, and a channel with background noise produce different scores, so blind listening is conducted on the MOS scale across all three.
The key indicator is the gap between studio and channel scores: if it stays within tenths, intelligibility does not fall apart for a subscriber with a poor connection.
We assess synthesis response time by the 95th and 99th percentile, not the average — the average hides the tail. A separate measurement is taken under simultaneous load: we hold a constant queue of requests and see whether latency grows linearly or hits a ceiling.
Streaming delivery of the first sound reduces the perceived pause before the response begins, even when the full phrase takes longer to assemble.
We calculate repeat requests from logs: the share of sessions with a clarification request, the share of transfers to an agent, and the share of dialogues where the phrase had to be repeated differently.
If these shares do not move after fine-tuning, the cause is not the voice but the scenario — and that is visible from the same numbers. We take measurements on the same set of phrases before and after, so the shift is attributed to a specific change, not traffic seasonality.
Voice adaptation methods: comparison of approaches
The choice is simple: either the model already sounds almost right and needs to be “patched up” on specific words, or you need a timbre it does not have.
Speech synthesis adaptation is rarely a single action — in practice it is a choice between three paths, and each hits its own ceiling. The selection criterion: how recognizably yours the voice must be and how much clean speaker speech you are ready to give for training.
| Option | When it fits | Limitations |
|---|---|---|
| Pronunciation dictionary | Terms, abbreviations, product names, and numbers are read incorrectly; the timbre otherwise works | Changes only pronunciation: intonation, rhythm, and timbre remain the same. It does not help with connected speech and homonymy — contextual parsing is needed |
| Voice adaptation (fine-tuning on top of a ready-made model) | There are 20–60 minutes of studio recording, and a close timbre is needed within a tight timeframe | The voice remains recognizably “not yours”; in a narrow subject area, quality drops are possible. Testing on domain texts is mandatory, not just on a control phrase |
| Training on your own corpus | A unique speaker voice and stable results on your vocabulary are needed | Requires 5–20 hours of annotated speech, a consistent microphone, and a uniform reading style. Timelines and the volume of checks are higher; quality depends directly on corpus quality |
We do not choose an option “by eye”: first we run your text through the base model, record exactly where it breaks — pronunciation, intonation, or timbre — and only then propose a path.
We verify the result with an intelligibility assessment and blind comparison on your domain; the target is MOS 4.3+.
Stages of engineering work on a custom voice
The order of stages here is strict: a data error cannot be fixed by architecture or by the number of training steps. Therefore most of the work is about recordings and annotation, and only then about the model itself.
From corpus to model in production
- Corpus audit. We listen to everything available: studio takes, call recordings, old broadcasts. We count how many hours are actually usable, and mark noise, clipping, and extraneous speech.
- Normalization and data schema. One format, segments of 5–12 seconds, no overlaps, with leveled loudness. Each fragment has speaker, text, and quality label.
- Recording annotation. We proofread transcripts manually: numbers, names, and abbreviations are written as they should be pronounced. Disputed fragments go to separate listening.
- Speech model training. We fine-tune from the base model, changing one parameter per iteration — otherwise it is unclear what produced the gain. We catch overfitting on the validation set and on speech the model has not heard.
- Blind listening and acceptance. Native speakers rate naturalness without knowing which is our synthesis and which is a studio recording. The threshold is MOS 4.3+; we separately check questions and intonation changes.
- Difficult cases. Numbers, dates, rare names, long words, fast speech — we run scenarios where synthesis breaks most often.
- Handover to production. Model weights, synthesis settings, pronunciation dictionary, and a set of checks — so the voice can be updated without us.
If the blind test does not reach the threshold, we return to annotation rather than “catching up” on quality with training steps: the voice will not become more even from this, but there will be a risk of losing the source speech.
Project artifacts: model, data, regulations
After the project, you retain not only the synthesized voice but also everything needed to operate it independently: data, build scripts, synthesis settings, and update rules.
We hand over artifacts so that repeated fine-tuning on a new batch of recordings does not require returning to us and does not stop a running service.
- Fine-tuned model and intermediate states of each run — there is somewhere to roll back if the new voice version sounds different from expected.
- Training corpus with annotation and checksums — records what data produced the result and makes repeated runs reproducible.
- Scripts for preparing recordings and launching training — a new batch of phrases is processed and sent to training without manual assembly.
- Synthesis configuration — tempo, timbre, pause duration, pronunciation dictionary for names and terms; edited separately from the model, no rebuild needed.
- Voice update procedure — the order of edits and release of a new version without stopping the service: shadow testing on part of the traffic, metric comparison, switching.
- Quality report — naturalness scores before and after, blind listening results, and a set of control phrases for re-checking.
- Voice documentation — where the data and model are stored, what access rights are needed, what to do if the sound degrades, and where to look first.
Case study: voice for a support line, metrics before and after
The support line for the service department used an auto-attendant with speech synthesis: the bot answered the call, clarified the category of the request, and transferred to an agent.
On typical phrases, the voice sounded even, but it broke on domain vocabulary — product codes, contract numbers, abbreviations from the internal reference book.
Every third call began with “please repeat,” and after reviewing the recording, some customers called back to the general line.
For the project, six hours of clean recordings of two support agents were made: without background music or overlaps, with the same speech tempo.
The recordings were annotated by phrase, a pronunciation dictionary was compiled separately for abbreviations and product codes, and numeric identifiers were assigned digit-by-digit reading rules — the model does not “guess” them.
The synthesis was connected to the existing telephony on Amazon Connect with streaming playback so that the utterance begins before the full response is generated.
Measurements before and after
Blind listening was conducted by a panel of twelve employees not involved in the project: they were given shuffled recordings of the old and new voice without labels.
The MOS intelligibility score rose from 3.1 to 4.3, the share of incorrect readings of product codes fell from 18% to 2%, and calls where the customer asked to repeat the utterance decreased threefold. There were no errors in contract numbers in the stream.
One limitation remained: the voice is stable on the vocabulary that was in the recordings. For new product lines, additional recordings for the pronunciation dictionary are needed — this is hours of work, not a repeated model training cycle.
How does TTS fine-tuning differ from tuning a ready-made voice?
The difference is in which layer of the model the engineer works with.
Tuning a ready-made voice is a layer on top of the pronunciation dictionary: phonetic transcriptions for names, abbreviations, and terms; stress placement; rules for reading numbers and dates; speed, loudness, pause duration.
Such a layer edits how a phrase is pronounced but does not touch who pronounces it: timbre, delivery, and character remain the off-the-shelf voice.
TTS model fine-tuning changes the acoustics.
The model learns the “text → spectrum” mapping on a specific speaker's speech and adopts what cannot be set by a dictionary: the density and softness of plosives, aspiration, micro-pauses between words, and even loudness on long phrases.
This is also where intonation appears in synthesis — the melody of questions and statements, logical accents in enumerations, and shifts in emotional register within a paragraph.
A ready-made voice imitates intonation in an averaged way because it is trained on a mixture of many speakers.
The boundary between the approaches runs along the data. Dictionary tuning requires not a second of recording and is reversible in one step — rules can be rolled back or rewritten.
Fine-tuning relies on a corpus: tens of minutes of clean speech with even diction and identical recording conditions are needed.
If the material is heterogeneous in microphone, room, or manner, the model will average the defects and transfer them into synthesis — fixing this at the inference stage is harder than assembling a normal corpus.
Practical conclusion: if the problem is in the pronunciation of specific words, we solve it with a dictionary; fine-tuning is excessive. If you need your speaker's timbre and lively intonation, only acoustic tuning will do.
We start with an audit of the task and speech samples: we show what can be achieved from the existing recordings and where additional recording will be needed — and only after that do we estimate timelines.
Technical concerns about the project: answering questions
Before the start, the same set of questions almost always comes up: how many recordings are needed, what will happen with rare words and names, how to update the voice when the service is already running for users, and who owns the rights to the speaker's voice. Below are brief answers on how we handle this in practice.
How much corpus is needed for fine-tuning?
For a stable timbre, 3–5 hours of clean studio speech from one speaker with even intonation is enough; for expressive delivery, 8–12 hours. More important than volume is homogeneity: one microphone, one room, the same tempo.
Recordings first undergo review — we remove noise, clipping, and duplicates — and only then is the corpus considered sufficient.
Will quality drop on rare words and names?
It will drop if rare tokens are left to the base model. We add them to the dictionary and phonetic rules, supplement contexts with domain terms, and separately check phrases with abbreviations, numbers, and names. We record the result with the MOS intelligibility metric — keeping it no lower than 4.3 out of 5.
How is the voice updated if the service is already running?
The new version is trained in a separate environment, passes checks on a reference set, and goes into a limited release on a share of traffic. If metrics do not decline, traffic is switched over fully, and the previous version remains for instant rollback. Users do not notice the replacement — the service does not stop.
Who owns the rights to the speaker's voice?
The speaker signs consent for speech synthesis with a specified term and scope of use; the rights to the trained model remain with the customer. This is fixed in the contract and transferred along with the model — further improvements are not tied to us.
Let's discuss the task: start and first steps on voice
Timelines for voice fine-tuning are determined not by the volume of recordings but by the state of the materials and how early the pronunciation dictionary is agreed upon.
That is why we conduct the first conversation as a technical review: we go through scenarios, listen to source materials, and immediately say where quality will hit the data and where it will hit annotation and stress placement.
To come back with a timeline estimate and work plan rather than general wording, it is enough to prepare for the first call:
- voice samples — recordings of the speaker in the channel where they will sound: studio microphone, headset, or telephone channel;
- texts for voiceover: voice menu scenarios, outbound calling scripts, interface utterances;
- a dictionary of terms, names, and abbreviations with correct stress and letter-by-letter reading;
- delivery requirements: tempo, pause duration, neutral or emotional manner;
- integration points — telephony, application, call center: they determine the audio delivery format;
- restrictions on voice rights and recording storage: who is the owner, where we store, who has access;
- a contact person who accepts the result and signs off on stages.
At the meeting, we analyze the acoustics of the source materials, show which recordings are missing for a stable result, and fix a set of phrases on which we will check synthesis quality.
After that, the materials go into work: we return with a stage-by-stage plan, timelines for each step, acceptance points, and a list of what you receive at the output — a trained model, a pronunciation dictionary, a test set, and voice update instructions.