Implementing prosodic speech control (rate, pitch, loudness)

Prosodic speech control: where intonation breaks

Prosodic speech control is managing tempo, tone, and loudness at the phrase level rather than the individual sound level. An off-the-shelf synthesis model reads text evenly by default: words are pronounced correctly, but the meaning is not emphasized.

The user catches the problem not from a word error but from their reaction—they ask again or drop out of the dialogue after a few turns.

The first typical failure is unnatural pauses in synthesis. A pause is placed where punctuation appears, not where a semantic group closes: the result is choppy delivery with a stop in the middle of a phrase, while two connected blocks are glued together without a breath. In long instructions and enumerations, the response has to be listened to again.

The second is stress failures. The model resolves homographs and terms by the most frequent variant, while the context requires another; phrase stress often falls on a function word instead of new information. The listener loses the thread and asks to repeat—the repetition sounds the same.

The third and fourth are monotone delivery and fluctuating loudness. The pitch contour is averaged across the dictated material: there is no rise on enumeration and no fall at completion, so the voice assistant's intonation does not distinguish a question from a statement. The level jumps at segment boundaries.

All of this is treated not by replacing the model but by a control layer: markup of syntagms and intonation centers, pause durations matched to boundary type, and level alignment across the entire utterance. Below is how it is structured.

What controlled speech intonation delivers: measurable effects

Prosody is three separate contours that are tuned independently: tempo (syllables per second and pause duration), tone (range and position of phrase boundaries), and loudness (signal level and semantic accents).

An error in any of them breaks perception as a whole: the text is correct, diction is clean, but the listener asks again because the intonation contradicts the meaning of the phrase.

Below are the metrics by which we verify the result, and we calculate them on the client's recordings, not on test examples.

• Speech tempo control — in sections with numbers, addresses, and SKUs, tempo decreases; on function-word connectives it returns to the baseline. Observable metric: the share of follow-up questions per dialogue and the number of requests completed only on the second confirmation.
• Tone control — intonation sets the boundaries of enumerations, questions, and caveats. Metric: the share of incorrect interpretations where an utterance contains a list of three or more items.
• Loudness control — the level is aligned across the channel, while accents remain on content words. We test intelligibility in noise on a recording from a real location and on a telephone channel—we calculate the share of correctly recognized words.
• Synthesis naturalness — in a protocol close to ITU-T P.800, we look at how the score changes and what share of utterances listeners take for live speech.

These metrics pull in different directions: speeding up speech for intelligibility in noise and preserving synthesis naturalness at the same time does not work.

That is why we tune the contours together and record a baseline measurement before edits—otherwise a shift cannot be distinguished from an impression from listening.

Approaches to intonation control: comparison of options

Intonation is controlled at three levels: markup in the text, synthesis acoustics, and post-processing of the finished recording.

None of them works alone, and the choice between options is determined by two factors—the volume of clean speaker recordings and exactly where intonation breaks in the output.

In practice, the table rows do not compete but add up: pause placement rules cover syntax, fine-tuning on speaker recordings transfers voice manner, intonation markup fixes accents in disputed places, and post-processing aligns tempo and loudness at boundaries.

The hybrid approach produces a predictable result when text changes: the rules remain readable and are edited manually, while the model is responsible for naturalness of sound.

How we implement intonation control: stages of work

Intonation is a set of parameters that must be fixed before synthesis: tempo, phrase melody, and placement of pauses and stresses. We start with an audit: we listen to current recordings, mark where the voice “drifts,” and assemble a reference sound for the working scenario. Then the work proceeds in steps, and each ends with a verifiable result.

1. Intonation audit. We analyze recording samples: where tempo is above normal, where a pause is not at a semantic boundary, where tone falls into monotone. The output is a list of deviations tied to specific phrases.
2. Intonation rule schema. We formalize the reference: phrase type (question, statement, enumeration), position of the stressed syllable, pause length, and pitch change boundaries. The schema becomes a single source of truth for all scenarios.
3. Voice profile setup. For each scenario, we define its own profile—baseline tempo, tone range, and loudness on accents. Profiles are versioned: editing one does not break already approved recordings.
4. Checking on control phrases. We run a fixed set—long enumerations, questions, numbers, abbreviations. We analyze discrepancies at the level of rules, not individual files.
5. Acceptance and handover. You listen to the control set and confirm the result. We hand over the rule schema, profiles, and edit regulations so that new texts can be onboarded without our involvement.

Project artifacts: configurations, voice profiles, regulations

We do not hand over the solution as a black box: along with the tuned model, the client receives a set of artifacts that the team can use to continue working independently—without us.

These are configurations, markup references, and regulations describing what was configured this way and why.

• Intonation profile — reference values of speed, tone, and loudness by utterance and scenario: it is used to verify a new recording and reproduce the previous sound when the speaker or material is replaced.
• Pause and stress rules — a table of pause durations, allowable shifts of the stressed syllable, and phrase boundaries; both text editing and synthesis setup rely on it.
• Annotated corpus — recordings with word-level labels and scores of compliance with the profile: the reference against which everything that comes next is compared.
• Speech processing scheme — an end-to-end map of the pipeline from input file to finished material, with control points and areas where human intervention is mandatory.
• Markup edit regulation — the change procedure: who approves, how the version is recorded, how a disputed edit is rolled back, and how it is checked that it did not break neighboring scenarios.
• Intonation documentation — a plain-language description of the decisions made: which deviations are considered normal and which are sent for rework.
• Check suite — an automated run over the corpus that shows whether the new version stays within the specified boundaries.

Everything listed is versioned and handed over together with the markup sources, so that six months later no one has to reconstruct the logic from memory. At acceptance, we run the control set and confirm that the result is reproducible on the client's side.

Case study: eliminating choppy intonation in a voice assistant

Starting point

The client is the voice assistant service Sfera with a continuous stream of incoming conversations. Individual phrases sounded fine, but in a live utterance the intonation broke: stress fell on the wrong syllable, loudness jumped between neighboring phrases, and tempo fluctuated. The listener asked again, and the conversation descended into repeating the same request.

We measured three metrics: the share of phrases with stress failure, loudness spread between phrases, and preparation time for a new speech block.

At the start, stress failed in every seventh phrase, loudness varied by a factor of four, and a new utterance took two days to assemble—each line was edited by hand and re-listened to.

What we changed and what we achieved

Between the text and synthesis, we placed a prosodic layer: markup of stresses and syntagm boundaries, a single melodic contour for the entire utterance instead of a contour for each phrase, loudness normalization to a reference level, and tempo limiting on long enumerations.

Error analysis stopped being manual—the rules are applied to new text automatically; the engineer checks the result rather than tuning each line.

After implementation, stress failure is less than one phrase per hundred, and loudness spread dropped below the threshold a listener can distinguish by ear.

Delivery stability holds across the entire conversation, including branches with long numbers and addresses, and a new speech block is prepared in an hour to an hour and a half instead of two days.

What to do if synthesis is already deployed but intonation drifts?

“Synthesis is already in operation, but intonation drifts”—this is usually not a breakdown but an accumulation of errors: the model predicts the pitch and duration contour from averaged diction, and on a long phrase deviations from semantic stress add up.

The listener catches it as “robot” before understanding the cause. Then there is one question: where is the boundary of the edit—markup on top of the finished layer or a rework of prosody.

If the system is generally intelligible and the failures are local, rules on top of markup are enough. We explicitly set stressed syllables, pauses at punctuation and semantic block boundaries, and limit phrase length and tempo on introductory constructions.

Such edits are made quickly and rolled back with a single revision without affecting the acoustic model.

It is another matter when the prosodic layer itself breaks: a stress shift changes meaning, a question sounds like a statement, and emotional registers are not distinguished.

Here prosody control cannot be treated with rules—a corpus with markup of tone, energy, and durations is needed, along with fine-tuning the model on it, with mandatory validation on a held-out sample. We do not touch the acoustics or vocabulary in this case.

Audio post-processing covers the third class of tasks: loudness alignment between syllables, pause normalization at splices, and tempo adjustment across a phrase.

It is cheaper than fine-tuning and does not require new data, but it treats the symptom: if the error is in the contour, filters only mask it until the next text.

We update a running system without downtime.

The new version processes a share of traffic, the old one remains a rollback point, and we compare quality by measurable attributes—the share of phrases with lost stress, tempo spread, and blind naturalness evaluation on identical texts. When the metrics converge, routing switches over entirely.

Common technical doubts about intonation control

Doubts about intonation control are usually not about the model but about data and acceptance: how many speaker recordings are needed, who owns the voice, and how to prove that the result is suitable. Below are short answers to what most often comes up at the evaluation stage.

We fix markup and channel requirements before training starts—otherwise the dispute at acceptance is about tastes, not metrics.

Who owns the voice data, and what is recorded in the rights?

The rights remain with the client, and the speaker signs a consent form with an explicit list: term, channels of use, and permission for synthesis.

Separately, we describe storage of reference samples and the revocation procedure—if the speaker revokes consent, it is clear which artifacts are taken out of use.

How is the result verified, beyond “we listened and liked it”?

Blind comparison of “before/after” pairs over a set of phrases, the share of correctly reproduced intonation contours, and separately deviations in tempo and loudness. We collect scores on a held-out sample that did not participate in training.

How is the markup structured, and can it be extended in-house?

A table with time labels: phrase boundaries, stressed syllables, pauses, and an intonation label. We document the format so that your team can add recordings without us—the markup remains reproducible and is versioned together with the audio.

What happens when the channel changes—headset, telephone path, recording in noise?

We check channels separately: a narrow path and extraneous noise change the spectrum, and metrics on one channel do not transfer to another. We build robustness through distortions at the data preparation stage, and at acceptance we listen to each scene separately.

Let's discuss the task: send examples of recordings and markup

The prosodic layer is not evaluated from a description: tone, tempo, pauses, and stresses depend on the material and scenario.

That is why task evaluation begins with analysis of your recording and markup examples—they show which intonation events the system must distinguish and where it currently makes mistakes. What is useful to send:

• 20–40 minutes of recordings in the form in which they enter the system: without manual cleanup or post-processing.
• The same sample with reference markup—syntagm boundaries, pauses, stresses, and types of intonation constructions.
• 10–15 examples of clear errors: what the system produced and what should have been output.
• Target scenario: diction assessment, synthesis control, or corpus markup—the metric depends on it.
• Constraints: where the solution operates, allowable response latency, and requirements for storing recordings.
• Success criterion: the share of correctly identified events on your set or an expert scale.

Based on this set, we come back with a task estimate and work plan: the order of stages, what we do at each one, what data is needed from you, and at what point.

We agree on acceptance before the start—it is tied to a metric on your own sample, not to the wording “it got better.” We analyze disputed cases there as well, using your examples.