Fine-tuning intonation and pauses in TTS with SSML markup

SSML Markup: Why Speech Synthesis Sounds Robotic

Speech synthesis breaks not at the sound level but at prosody. The engine splits a phrase into syntagms using punctuation and its own dictionary, and where a human derives intonation from meaning, the machine has nothing to go on.

This leads to choppy phrases, stress in voiceover on the wrong syllable ("zámok" instead of "zamók"), words running together at the boundary between preposition and noun, and pauses in speech where a live person would not make them.

Ambiguous text is the worst: abbreviations, numbers, Latin script inside a Russian phrase, surnames, toponyms, tariff names. Without explicit rules, the engine will read the abbreviation for "year" as the full word, and a rare surname with default stress.

The listener notices this instantly because speech is processed faster than it is consciously understood.

SSML markup provides controls exactly for these places: tags set pause and its duration, stress, rate, volume, and timbre of an individual phrase; they hint at how to read a number or abbreviation, while a pronunciation dictionary locks in terms and proper names.

But markup for markup's sake sounds cartoonish—it is built around a speech scenario: greeting, confirmation, refusal, and help each receive different speech synthesis intonation.

In a finished product, the cost of error is visible in people's behavior. Where naturalness of sound is lacking, the share of clarification requests grows, listeners abandon the recording midway and do not reach the needed item in the voice menu. Below we break down how this is addressed at the level of text and synthesis settings.

What Controlled Intonation and Pause Placement Provide

Intonation control turns voiceover from "reading text" into "speaking by meaning": SSML markup sets rate, volume, and emphasis on an individual word.

A change to one parameter is audible in a specific line, so the share of fragments going to manual re-recording drops severalfold.

Pause placement is separate work, not commas from the script. A short pause within an enumeration, a long one at a change of semantic block, complete silence before a key number.

The line fits the timing without cutting or stretching the sound, and editing stops being a mandatory stage.

Pronunciation dictionary fixes what synthesis reads incorrectly by default: company names, abbreviations, toponyms, numbers in oblique cases.

One dictionary per project gives a unified voiceover sound—the narrator in the main video, the voice in the app, and the version in another language pronounce key terms identically, without inconsistency between episodes.

At voiceover acceptance, we check not "like it or not" but control points: stress matching the semantic parse, pause duration at junctions, dictionary accuracy on a set of complex terms.

Script acceptance speeds up because edits go as a list of parameters rather than re-recording the entire track: approval takes one iteration instead of several.

SSML Markup: Modes for Controlling Intonation and Pauses

Speech Synthesis Markup Language (SSML) is a markup standard for speech synthesis that engines support only partially. The same file on different voices either produces even pauses and correct stress, or is ignored and breaks phrase intonation.

Therefore, we take the minimum level of intervention sufficient for the task and know each one's limitations in advance.

We check synthesis capabilities before launch: run reference phrases on all voices in the pool and listen to sentence junctions. We store markup next to the texts—you can roll back a version or change a voice without re-recording the script.

Such runs take hours, not days, and remove the main risk—"it sounded good on the test voice."

Implementation Stages: From Script Audit to Handover of Rules

The work order is built so that each stage leaves a verifiable artifact: a parsed script, a delivery scheme, a set of rules, and a reference recording.

We do not "tweak" the voice in production—all changes go through versions of the markup file, so reverting to a previous intonation takes minutes.

1. Script audit. We read the text together with the director's task: where the semantic emphasis is, where an enumeration is, where a question is. We mark places where even reading breaks the meaning—run-together items, abbreviations, numbers and dates that sound incorrect.
2. Text delivery scheme. We compile the script into a "fragment → role → voice behavior" table: rate, pause before and after, emphasis strength. This is the map used to write rules; without it, markup turns into a set of random insertions.
3. Voiceover rule setup. We assemble SSML templates for each fragment type: pauses, stress, rate, pronunciation of numbers and abbreviations. We test on several voices and speeds so the rule does not fall apart when the speaker changes.
4. Listening on real channels. Phone line, speaker, headphones: a narrowband channel eats part of the pauses, and this is audible only on the device. We fix where meaning is lost.
5. Auditory acceptance. The client listens to reference fragments and makes edits by voice—"this is rushed," "this sounds like a question." We turn each into a specific markup change.
6. Handover of the regulation. We deliver a document with rules, templates, instructions for adding new texts, and brief training for the client's team—after that, scripts are voiced without us.

Prosody Schemes, Templates, and Regulations in the Project

After project handover, the client is left not with a "configured voice" but with a working loop: rule files by which voiceover can be changed in-house.

Everything verified in SSML markup sits next to the texts and is versioned with them—when the script is updated, it is clear which edit led to which result.

• Rule library — typical markup constructs for intonation roles: question, enumeration, emphasis on a number, warning, pause before a key word.
• Line templates — blanks for recurring scenes: greeting, confirmation, clarification, error message, conversation ending.
• Pronunciation dictionary — a "written / sounds like" table: abbreviations, Latin script, units of measurement, client product names that the synthesizer reads incorrectly.
• Prosody map — boundaries of rate, pitch, and pause duration for each scenario; going beyond them is caught at acceptance, not in the finished product.
• Voiceover update regulation — who makes the edit, which checks are run, how the new version is compared with the previous one, and how to roll back.
• Voiceover documentation — file structure, purpose of each rule, and exception logic: a new engineer can figure it out without messaging us.
• Acceptance checklist — what to listen to before release: numbers, abbreviations, phrase junctions, the beginning and end of the script.

The kit removes the main risk: voiceover does not freeze after release and does not require us for every new phrase.

Case: Eliminated Choppy Phrases in Instruction Voiceover

The client is a service network that voices instructions for equipment installation and maintenance.

Synthesis read technical blocks as an even stream: enumerations merged into one phrase, the pause before a warning did not differ from the pause within a sentence, and abbreviations and units of measurement were pronounced any which way.

Every second line was returned by the editor for re-recording—edits went into the text, not the markup, and on the next run the defect surfaced again.

We assembled the voice on Amazon Polly: the platform understands SSML markup, but writing it manually for hundreds of lines is pointless. We built a template library—a separate one for a step, a warning, an enumeration, a section heading.

Inside, we set the rate: warnings are read more slowly than the main text, pauses between semantic blocks are noticeably longer than within a phrase; terms and abbreviations go through the pronunciation dictionary, not through hints in the text.

Before generation, a check runs the markup: an unclosed tag or invalid value stops the build instead of surfacing in the finished audio.

What Changed

Line consistency no longer depended on who edited the text: intonation and pauses are set by the template, not by ear. The instruction package now goes out in a single run, and changing wording does not require rebuilding the entire voiceover.

What to Do If Synthesis Is Already Deployed and Pauses Do Not Work

Inherited synthesis is rarely empty: voiceover is already deployed, lines are recorded, the script is written.

The problem is different—such a system silently ignores part of the SSML markup: a command passes without error, but the output is even speech without pauses and accents. The first step is not to rewrite texts but to determine the actual support map.

We run the same phrase with different tags and compare the result by ear and by sound duration. This reveals that pauses and pitch changes work, acceleration and word emphasis are cut, and nested constructs break neighboring markup. Partial markup support is a working situation; intonation control can still be built with it.

Next, we build a transformation layer on top of the existing recording: the script stays as is, and before synthesis a unified notation is decomposed into the commands the system actually understands.

We replace unsupported commands with workarounds—punctuation, splitting a long line into separate phrases, local rate adjustment at the beginning and end of a sentence.

Backward compatibility is a mandatory condition here: a line without markup must sound the same as before the modification, otherwise all already-approved voiceover will shift.

Therefore, we make changes pointwise, keep the previous version for each text, and check that the new sound differs only where it is explicitly set. This way, modifying existing voiceover provides controllable intonation without rewriting the script.

Common Technical Doubts About Speech Synthesis

SSML markup works on top of a specific synthesis engine: pauses, stress, and rate are set by tags, but the engine determines the sound.

Hence typical engineering questions—what carries over when changing the engine, how to catch unnaturalness by ear, and where synthesis capabilities end.

How can naturalness be checked by ear rather than by documentation?

By blind listening: the same phrases through different markup versions, evaluated against a checklist—stress, word connectedness, rhythm, pauses at punctuation, homographs. Disputed fragments go into pronunciation dictionary edits, not manual re-recording.

How should dialogue be voiced so lines do not sound like one voice?

Lines are marked by role: each role gets its own voice and rate, with controlled pauses between them, and volume is leveled to a common level. This preserves the character's timbre and the rhythm of conversation, including interruptions and short replies.

What changes in multilingual voiceover?

The language is set by a separate tag per fragment; phonetics and stress rules switch with it. One voice rarely sounds credible in all languages—for each we take a suitable voice and its own pronunciation dictionary for names and terms.

Where does synthesis still fall short of live recording?

Shouting, strained whispering, sarcasm, simultaneous speech of several characters, and nonstandard names remain weak spots. We mark such fragments during markup and cover them hybridly—with narrator recording over the synthesized array.

Let's Discuss the Task: We'll Build Sound for Your Script

To make the assessment concrete rather than "by eye," an input set is needed—the same one the system will work with. Setting a task without example lines almost always produces a mismatch in expectations, so we start with materials, not with discussing deadlines.

• Example lines — 15–30 phrases as the user will hear them: greeting, refusal, numbers, names, emotional peaks.
• List of languages and dialects, including mixed phrases where an abbreviation or Latin script appears inside Russian speech.
• Delivery requirements — tone, rate, timbre, gender and age of voice, narrator reference in audio, if available.
• Pronunciation dictionary — abbreviations, names, units of measurement, numbers and dates that synthesis reads incorrectly.
• Playback channel — phone line, app, video: pause length and acceptable response delay depend on it.
• Current SSML markup — if it is already used, show the file: it reveals which constructs work and which the synthesizer ignores.

From this, a work plan is assembled: dictionary, manual markup of pauses and stress, runs on your lines, acceptance by agreed criteria. A timeline estimate appears after the first run on real examples—it relies on dictionary volume and number of voices, not text length.

The collaboration format is short iterations with listening to each version: edits go into markup, not into the finished recording. Describe the task and send materials—we will return with an estimate and plan.