Custom voice for TTS: training on VITS, XTTS, YourTTS

Why TTS model training fails on a real voice

A demo on ten minutes of studio recording sounds convincing, and after it TTS model training seems like a matter of a few days. Reality is tested by the first long phrase: the client turns on the result and does not recognize themselves.

Speech synthesis in your own voice breaks not on architecture, but on data — and each such breakdown costs weeks of retraining and delays launch deadlines.

The main reason is the quality of the training corpus. Real recordings come from calls and meeting rooms: echo, air conditioner noise, varying loudness of takes, clipped phrase beginnings, background music in the intro.

Three hours of clean material from one microphone in one acoustic environment produce a recognizable timbre, whereas thirty hours of "as is" blur it into an average speaker.

Audio annotation matters no less: if phrase boundaries are marked inconsistently, the transcript diverges from speech, and some tracks contain pauses and breaths, the model learns to draw out sound and swallow endings.

Bringing the corpus to a uniform form — level normalization, segmentation into phrases, text proofreading with punctuation — usually takes longer than training itself.

The second group of failures is intonation and terms. On short utterances the voice sounds lively, but on long phrases prosody flattens: the phrase is delivered as a flat stream without semantic accents, and to the ear this reveals synthesis faster than any artifact.

Proper names, abbreviations and Latin script fail separately: the dictionary reads them letter by letter or places stress in the wrong place, and mispronunciations of terms in industry text are immediately audible.

This is cured not by "tweaking" a finished model, but by a pronunciation dictionary and additional examples in the corpus.

That is why work starts not with code, but with an audit of recordings and annotation — we analyze how this is addressed and which checks filter out defects before training.

What a custom voice delivers: effect in measurable quantities

Your own voice is fine-tuning a model on a speaker's recordings: several hours of clean studio speech set the timbre, and architectures like XTTS transfer it to any text without re-recording. We measure the effect not "by ear," but in quantities.

Intelligibility of synthesis and naturalness of intonation

We check intelligibility by blind listening and reverse recognition: if the transcription of the synthesized track matches the source text, articulation is not "eaten." Intonation stops being flat — the model places pauses and semantic stresses where the listener expects them. The proportion of words with incorrect stress in terms and proper names drops severalfold.

Reduction of manual voiceover edits

Previously, each wrong accent meant re-recording a take or splicing in an editor. Now edits are reduced to the pronunciation dictionary: abbreviations, Latin script and narrow terms are specified once, and the model reads them identically across all materials. Manual work shrinks to proofreading exceptions instead of continuous control.

Speed of audio material production

A long text turns into a track in minutes, so short, full and advertising versions are made from one source instead of being recorded anew. An edit during approval no longer pulls in the studio — only the changed fragment is re-voiced. A unified brand voice is preserved across all media, from videos to auto-attendant.

Approaches to voice training: comparison and limitations

The choice of option is determined not by the desire to get an "own" voice, but by the volume of clean speaker data and how strictly term pronunciation must be maintained.

Adaptation of someone else's voice and fine-tuning a ready model give results on short recordings; training from scratch on your own corpus requires many times more data, but removes license and timbre restrictions.

Separately, we fix requirements for speaker recording: even tone, absence of echo and overlaps, consistent reading manner in all sessions. Different recording quality yields "drifting" timbre audible on long phrases.

The pronunciation dictionary applies to any option: abbreviations, toponyms and product names are set manually, otherwise the model reads them by general rules.

Limitations here are more important than advantages: they determine which class of defects will remain in the finished voice. Before training a TTS model, we review your recordings and say which option fits them and which will require additional recording.

How work is structured: from data audit to finished voice

Voice training is not a single run, but a chain where each step relies on the result of the previous one.

An error at input — noise in the recording, unclear rights to the speaker's voice, incorrect annotation — will not surface immediately, but will show up in the finished voice as artifacts and fluctuating intonation. Therefore we run the project as a pipeline with acceptance at each stage.

1. Audit of audio materials and rights. We listen to recordings, assess noise level, duplicate takes, pauses, timbre consistency; check documents for voice usage. Output — a report: how much clean data exists and what needs to be additionally recorded.
2. Corpus preparation. We normalize loudness, remove clipping and extraneous noise, cut long pauses, bring files to a unified format and segment into utterances.
3. Annotation and transcription. We compare text with the recording, correct discrepancies, mark stresses and pauses. An error in the transcript results in incorrect pronunciation, so we perform this step with proofreading.
4. Voice training plan. We choose synthesis architecture for the task, fix the number of steps, control metrics and checkpoints — so that the result is reproducible and does not depend on a lucky run.
5. Training and intermediate listening. We listen to a fixed set of phrases at equal intervals, compare options with each other and stop at the best one, not the last one.
6. Listening and acceptance. We assemble a test set: numbers, abbreviations, long sentences, intonation changes. The client listens; we turn comments into edits and fine-tuning.
7. Handover and support regulations. We deliver the model, parameter description and instructions for adding new phrases, agree on the retraining procedure when the text domain changes.

Project artifacts: corpus, configurations, documentation, regulations

The client retains not only a file with weights, but a complete package by which TTS model training can be repeated, and the voice can be updated and transferred to another team without us. Below is what exactly is included in the handover and in what form it is documented.

• Prepared corpus. Audio after cleaning and normalization, transcripts with pause and stress annotation, split into training and validation parts, information about rights to source recordings.
• Configuration files. Training and synthesis configurations with fixed parameters: speed, fragment length, number of steps, generation seed. A repeated run gives a comparable result, not "whatever happens."
• Model checkpoints. Intermediate and final versions, synthesis launch scripts, environment requirements — how much memory and what acceleration are needed for real-time operation.
• Voice documentation. Voice passport: pitch range, comfortable tempo, permissible emotions and places where synthesis sounds unnatural. From it, an editor understands voice boundaries without listening to hours of recordings.
• Voice usage rules. Speaker consents, term and territory of validity, list of topics and formats where the voice must not be used. This protects against claims and against recognizability in an inappropriate context.
• Voice update procedure. Regulations: how to add new recordings, when retraining is justified, by which metrics to compare the new version with the current one and at which values to roll back.
• Instructions for the content team. How to prepare text for synthesis: numbers, abbreviations, stresses, pauses and markup, plus typical errors and "before/after" examples on real phrases.

Case study: custom voice for an information service

An information service kept its auto-attendant on studio recordings of a speaker: the voice was lively, but changing a rate plan or route dragged along re-recording of all related phrases.

Edits accumulated until a scheduled session, and splices from separate files sounded choppy — intonation at junctions did not match, callers asked for repetition.

We assembled a corpus on the same speaker: about 12 hours of clean speech in a consistent reading manner, with level alignment and room noise suppression.

Annotation was done in Praat — segmentation down to phoneme level, marked stresses, a separate pronunciation dictionary for abbreviations, numerals and names.

Before acceptance, we ran a test set from real inquiries and added to the corpus the phrases that synthesis broke most often.

Measurements before and after

The voice stopped being a consumable: text is edited in the panel, while pronunciation remains recognizable.

The share of re-recordings dropped to zero, intelligibility on the control sample increased, and disputed words are resolved by adding a line to the dictionary, not by a new studio session.

What to do if voice recordings exist but there is no annotation

Raw speaker recordings without annotation are a typical input, not an exception.

First we assess acoustics: whether the level is even, whether there is overload on loud syllables, whether it is one room and one microphone position in all files, how many different voices there actually are, whether there are speech overlaps.

Suitability assessment is based on these features, not duration: three hours with noise and microphone changes are less useful than forty minutes of clean monologue.

Most can be restored automatically. Speech recognition together with forced alignment yields a transcript and sound boundaries, and by model confidence, places where text or boundaries are unreliable are marked: connected speech, laughter, filler words, overlaps.

Such automatic annotation covers the main volume of the corpus, while disputed fragments go as a separate list for listening.

Then manual work begins, and it cannot be avoided. Re-recording of disputed fragments is done in the same microphone position and same manner: the speaker repeats words and phrases with neutral intonation, supplements what was recognized uncertainly, and covers gaps.

Here it is important not to "improve" the voice — a change in tempo and timbre makes the fragment foreign to the future model.

The additional data collection plan is built from coverage: which combinations of sounds, tempos and pauses are missing in the material, where one speech style ends and another begins.

As a result, a corpus remains with transparent origin of each utterance — it is visible what was recorded in the studio and what was cut from a long recording. On such a set, TTS model training proceeds according to the actual content of the files, not assumptions about it.

Technical doubts: quality, timelines, voice rights

Doubts before TTS model training are almost always the same — from volume of recordings to rights to the finished voice. Short answers below usually resolve them before work starts.

What to do if source files are noisy?

We remove hum, clicks, overload and room echo, align level, cut by pauses. Recordings from different microphones are separated into groups and taken into account during training — otherwise timbre "drifts."

Will speech manner and intonations be preserved?

Yes, if the set contains varied delivery and natural pauses. On short recordings the result is an averaged voice; over several hours, tempo, accents and characteristic phrase endings are preserved.

How is synthesis acceptance carried out?

We synthesize a control set: numbers, abbreviations, names, long sentences. We listen together, assess intelligibility, intonation and phrase junctions; the threshold for the share of successful phrases is agreed in advance.

Who owns the trained voice?

Rights are secured by the speaker's written consent: territory, term and methods of use. The contract determines who owns the model, whether it can be transferred to third parties and what happens if consent is withdrawn.

Let's discuss the task: what we need from you for assessment and plan

To come back with a work plan and timelines for speech synthesis model training, you do not need to prepare a technical specification or annotate audio in advance. A few inputs about the scenario and what voice recordings you already have are enough. Below is the minimum by which we estimate the scope of work.

• Target scenario: where synthesis will sound — video voiceover, telephony, app interface, voice assistant — and typical phrase length.
• Speaker reference: 2–3 recordings of 10–30 minutes in a quiet room; a studio archive will also work. Plus the speaker's consent to use the voice.
• Materials you already have: studio takes, draft transcripts, archived conversation recordings — they show how much data needs to be collected additionally.
• Requirements for the result: language and dialect, whether emotion and tempo changes are needed, how critical correct pronunciation of names, terms and numbers is.
• Deployment conditions: where the model runs — in an external cloud or in the client's closed environment, what load is expected during peak hours.
• Approval points: who accepts the result and at what stage access to your test phrases is needed.

Then we come back with a work plan, timelines and training scheme. We show the first result as a trial listening on your phrases — the voice can be heard before the project starts, not after. Describe the task — we will prepare an estimate for your scenario.