Why AI dubbing breaks on articulation and timing
AI dubbing fails not on translation but at the junction of sound and image.
The synthesized track comes in clean and even, but the character's lips continue to live by the rhythm of the source language — lip-sync and audio desync is visible within a couple of frames, and the viewer registers the substitution before understanding what is wrong. Then the scene falls apart: the intonation is convincing, but trust in the shot is lost.
The second bottleneck is line timing. The same thought in another language takes longer: where the original fit into lip movement, the translation adds an extra 15–25% of duration.
Speeding up to fit turns dialogue into a tongue twister, while refusing to fit pushes the phrase beyond the shot boundary — the line runs into the next scene and breaks the editing rhythm.
A separate issue is on-screen articulation in close-ups. The model replaces the mouth but preserves the original facial expression and micro-movements of the jaw, producing "rubber" facial animation that the eye catches instantly.
In a wide shot it is unnoticeable; in a close-up it is a defect that turns the scene into lip-sync rerecording with manual touch-up of every frame.
The cost of the error is not cosmetic. Distribution gets a version where some close-ups are rerecorded from scratch, editing reassembles line timing for the new phrase length, and release shifts to a stage no one planned. We break down how this is closed.
What synchronous rerecording delivers: the effect in measurable terms
The main metric of synchronous rerecording is the share of scenes that pass acceptance on the first run. On a dialogue corpus, it stays at around eight scenes out of ten: articulation, line duration, and timbre align without edits.
Not long ago, such fragments went to a second and third round, and each round meant a newly assembled dubbed track and a repeat review.
Manual phonetic editing is reduced by an order of magnitude. The engineer no longer fits line length to lip movement frame by frame — they check only the segments where the model showed low confidence.
Instead of hours per scene, minutes of pinpoint edits remain, and they do not break the rhythm of neighboring lines.
Preserving voice timbre is the second half of the result.
Cloning is based on a clean reference track of the actor, and in our work we rely on the current branch of ElevenLabs: the timbre holds up through changes in emotion and volume, so the viewer does not hear a "different person" in the middle of a scene.
The "lips did not match" failure class disappears as a category. Articulation and voice are computed jointly rather than in two independent passes, so desync does not accumulate toward the end of a long line.
Scene acceptance comes down to a formal criterion: frame discrepancy and speech intelligibility are within norm — the scene is accepted rather than sent back for revision.
AI dubbing formats: segment-based and streaming modes
The rerecording mode is chosen not by preference but by what is in the source tracks and what should come out at the end.
Segment-based processing cuts the material into individual lines, translates and synthesizes each one anew — you can shift a phrase in time, replace the voice on a specific line, and remix the track with music and noise.
Streaming dubbing mode goes through the audio forward without preliminary markup and delivers the result with a small delay, but an already delivered fragment cannot be corrected.
| Option | When it fits | Material constraints |
|---|---|---|
| Segment-based processing | Full-length films, series, advertising — anything that is edited and goes through acceptance | Requires a clear voiceover track: lines over music and noise are recognized less accurately, and some go to manual correction |
| Streaming dubbing mode | Live broadcasts, webinars, simultaneous screenings | No second pass: a change of speaker or a slip of the tongue cannot be corrected retroactively |
| Multi-voice rerecording | Dialogues where it is important to separate characters by timbre | Requires stable speaker separation; when lines overlap, voices blend and manual markup is needed |
Material constraints are often more important than the mode itself: short one- or two-word lines and phrases overlapping each other are the main source of lip desync.
In practice, the working scheme is hybrid: a rough pass in streaming mode, then segment-based refinement of the places where articulation diverged from the frame.
Stages of work on a dubbed track
Lip-sync dubbing is not text translation but a reassembly of the audio track: lines are rerecorded, and articulation is fitted to the source video. The order of steps is fixed because an edit at a late stage pulls a full reassembly of the entire track with it.
- Track audit. We break down the master into dialogue, music, and noise, assess line clarity, and flag scenes where speech is overlapped by effects or sounds off-screen. The result is a track diagram and a list of problem areas.
- Script preparation. We align lines with time codes and select wording that matches the original in length and rhythm: the phrase must fit the lip movement rather than be trimmed in editing.
- Voice model setup. We assign a separate voice to each character — timbre, tempo, intonation — and tune it to the role's emotional range: without this, the voice "drifts" during shouting.
- Track assembly. We mix synthesized lines with music and noise, level the volumes, restore pauses, and check the duration of each phrase relative to the original.
- Check on reference scenes. We run close-ups, fast dialogues, and emotional scenes — that is where desync is most noticeable. We agree on the scene set in advance, before mixing the entire track.
- Project delivery and team training. We hand over master tracks, the project diagram, voice model parameters, and short instructions so your specialists can update lines without us.
Reference scenes are approved before final mixing, so edits remain local rather than turning into a new assembly. This is what saves the project's main resource — cycle time.
What we hand over: projects, presets, regulations
We hand over not "source files on disk" but a kit that the editor, sound engineer, and producer can work with without us. Each artifact covers a specific stage: from regenerating a line to accepting the final master and making edits a month after delivery.
- Working track projects — separate sessions by line with saved timecode markup, so a single scene can be reassembled without running the whole film again.
- Character voice presets — a fixed timbre, tempo, and intonation profile for each role: a new line sounds in the same voice as in the first episode.
- Audio routing diagram — a map of tracks: dialogue, music, noise, international soundtrack. It shows what is mixed and what goes into the master as is.
- Project passport — source language and dubbing language, frame rate, resolution, scene list, and a character — voice — voice actor correspondence table.
- Acceptance regulations — a step-by-step checklist: articulation, lip-sync in close-ups, absence of desync, volume level, pauses for lines.
- Editing instructions — where to put the tracks, in what order to enable them, how to replace a line if the client requested a wording edit.
- Decision log — what was rejected and why: scenes with overlapping speech, places where phrase length had to be changed to match articulation.
The track diagram and project passport remove the main post-production risk — edits are made according to a document, not from memory. The acceptance regulations fix the criteria in advance, so the "like / dislike" argument turns into a point-by-point check.
Let's discuss your task — we will come back with the kit composition for a specific format and amount of material.
Case study: lip-sync for a series without articulation defects
Seasonal material differs from a standalone film in that episodes are interconnected: character voices, speech tempo, familiar shots.
An error in one episode pulls reassembly of neighboring ones, so season localization almost always runs into the throughput of manual articulation fitting.
Where it broke
The input had a ready translation and voiceover; the output required lip-sync that does not destroy facial expressions. About 40% of scenes required manual re-editing — close-ups, fast speech, dialogues with overlapping lines.
An episode took 8–10 days to deliver, and returns for reassembly accumulated at 4–5 per season: consonant mismatches were found already at the final screening, and the work went to a second round.
What was changed
The episode was split into segments by lines and phonetic markers, and synchronization was computed segment by segment rather than across the entire material.
The result was checked against the dubbing requirements of the Netflix platform, and articulation was verified on key frames automatically — before a human viewed the scene. The editor reviewed only disputed segments with borderline check results.
What became
The share of scenes with manual re-editing dropped from ~40% to 7%, returns for reassembly — from 4–5 per season to one, and episode delivery time decreased from 8–10 to 4–5 days.
The class of defect that previously surfaced only at acceptance also disappeared: consonant desync in close-ups. The season is delivered on a steady schedule, not in bursts before final screenings.
What to do if the material has already been rerecorded and edited
It is possible to integrate lip synchronization into already rerecorded material, but everything depends on what has been preserved in the project.
If the speech track is separate from music and effects, the work comes down to reassembling only the speech layer: articulation is rebuilt for the existing translation, and the backing remains untouched.
If only the final mix remains, the track has to be taken apart again — separated into speech, music, and effects. The separation is never clean: traces of voice remain in the music layer, and hall reverberation and noise remain in the speech layer.
That is why we first listen to fragments where speech goes over music, and to pauses: if a "ghost" of the speaker is audible in silence, we clean the layer manually rather than trusting automation.
Next is timing. Synchronization relies on phoneme recognition and lip position, and when the length of the translated phrase does not match the original articulation, there are two paths: adjust the video or rerecord the phrase while preserving meaning and length. The second path is preferable — it does not tear the edit and does not change scene lengths.
Mixing with music is the riskiest stage: music and effects cannot simply be laid on top. We preserve their levels, phase, and dynamics, and fit the speech track so that it does not stick out on loud accents.
We compare the finished master with the original in blind fragments and run it on different playback systems — from headphones to a television mix — otherwise discrepancies will surface with the viewer.
Technical concerns around AI dubbing
Before a project starts, clients ask the same questions — about voice rights, noisy scenes, short lines, and multilingual versions. Below are substantive answers: how this works in processing and where the boundaries of applicability lie.
How are voice rights cleared?
We build synthesis on the voice of a dubbing actor with written consent that specifies the scope of use and term. We use cloning only where rights are confirmed by contract — otherwise the track is assembled from scratch for the specific role.
What should be done with noisy and overlapping scenes?
We break the mix into a dialogue track, music, and noise, and work specifically with speech rather than the final mix — otherwise articulation "drifts" under the hum.
We handle overlaps by sections where lines are separated in time rather than layered on top of each other.
Does this work on short lines?
"Yes," "no," sighs — the hardest class: there are few anchor sounds, and it is difficult to set the lips from them. Such lines go through a separate pass with manual articulation checking, not a general run alongside long phrases.
How are multi-language versions structured?
Phonetic markup and the time grid are calculated once and reused for all language versions — only speech synthesis changes. Adding a language becomes a repeat pass over the lines rather than a new scene processing.
Let's discuss the task: we will come back with an estimate and plan
To come back with an estimate and work plan, we need inputs on the material: the earlier we see a real scene, the more accurately we can state timelines and the labor intensity of stages. Everything listed can be gathered in one go; no separate preparation on your side is needed.
- source in the highest available quality and, if preserved in the project, a separate track with clean speech of the original actors;
- a ready translation or a rough transcript of lines with time codes; if there is no translation, we will estimate its preparation as a separate stage;
- glossary: names, titles, terms, stresses, and how to pronounce them on screen;
- language pair and voice requirements: character type, age, timbre, examples of dubbing actors you like;
- material structure: number of lines, share of off-screen text, songs, and scenes with dense voice overlap;
- delivery requirements: separate tracks, containers, frame rate, whether a final mix is needed;
- priority episodes and deadlines: what is needed earlier in order to parallelize work by scenes.
On a test scene two to three minutes long, we run the entire route: speech recognition, rough AI dubbing, voice selection for character types, articulation synchronization, and track assembly.
You get a fragment that shows the quality on your material, and we get an understanding of the real labor intensity. After that, we come back with a project estimate, a stage-by-stage work plan, and honest boundaries of applicability by scene.