Why talking head generation breaks on articulation
Discrepancy between audio and video almost never shows on short phrases — it accumulates. The model drives articulation by an averaged rhythm, while live speech breathes: pauses, accelerations, inhalations, semantic emphases.
The eye catches plosive and bilabial sounds — "p", "b", "m" — precisely there articulation defects read as "someone else's mouth".
On a long line, error compounds: a small lag or lead in each phrase sums up, and by the third or fourth minute audio-video discrepancy becomes noticeable without frame-by-frame comparison.
Lip sync tuned on a short clip falls apart on dubbing long lines — especially where the speaker changes tempo and pauses within a sentence.
Talking head generation when releasing a series of videos multiplies this effect by the number of episodes. The same defect is reproduced in every episode, and fixing becomes a manual pass through timecodes: shift the track, regenerate the segment, reassemble the scene. Some takes go to waste, and re-recording eats time planned for editing and publication.
The root is that articulation is built independently of the phonetics of a specific track.
Below we break down how this is closed: binding facial expressions to annotated phonetics, controlling drift accumulation on long segments, and acceptance by articulation, not by the first frame.
What precise lip sync gives: time, quality, volume
Lip sync stops being a separate post-production stage and becomes part of the pipeline. Previously, the track was re-recorded, then articulation was manually adjusted by key frames: a ten-minute episode took a day of work, and any line edit restarted the cycle.
Open solutions like Wav2Lip showed that hitting articulation is achievable automatically. But in practice the result is determined not by the model, but by data preparation: aligning the track to lines, stable face tracking, handling occlusions and head turns. These steps decide whether an episode goes to publication or returns for revision.
The effect is measured by three numbers. Episode release time: a ten-minute video goes through the cycle in one run instead of sequential fragment-by-fragment editing.
Share of manual refinement: sync is adjusted by hand only where the speaker covers their face or leaves the frame. Stability: articulation does not diverge on fast speech and when changing angle between takes.
Another metric — how many episodes go to publication without a single edit. When dubbing automation is tuned to a specific shooting format, such episodes become the majority, and freed time goes to editing and meaning, not lip adjustment. We break down how this is arranged in production.
Approaches to generation: video clone, avatar, re-voicing
Lip sync is not solved by one algorithm: the method is chosen by what material is available at input, and changing this mid-project is expensive. Therefore at the start we fix three working options and test each on a real shooting fragment, not on a showcase demo.
| Option | When suitable | Limitations |
|---|---|---|
| Speaker video clone | There is 15–30 minutes of clean material: even light, one angle, speech without interruptions | Reproduces source physicality — changing pose, background or wardrobe requires additional shooting |
| Photo avatar | Start needed in a matter of days, no filming — a front-facing portrait suffices | Facial expressions are poorer; close-up and strong emotions reveal synthetic quality |
| Re-voicing | Video already shot, language or line text changes | Shots with hands near face, profile, beard lose articulation accuracy |
Options combine: train a clone on an interview archive, and build language versions with re-voicing over the finished edit — the original voice is preserved, only the articulatory sequence changes.
Language limitation rests on phonetics: for languages with articulation unusual for the model, we add an exception dictionary and gather control phrases at acceptance.
We work with HeyGen and open models among others, but the tool is secondary: 30–60 seconds of test run on your material show exactly where sync will drift, before the project goes into production.
Describe the task — we'll return with an assessment of which of the three approaches to use and what will be needed from your side.
How we run the project: from recording audit to handover
The project starts not with code, but with a source audit: recording quality determines how much manual refinement talking head generation will require. Below is the order of work we follow from first listen to solution handover.
- Audio recording audit. Listen to the material in full: assess noise and reverberation, find dropouts, overlapping lines, different tempo between takes. Output — a map of problem fragments and scope of edits before start.
- Solution scheme. Fix the pipeline design: splitting into phrases by pauses, bringing the track to uniform tempo, binding articulation to frames. Agree before implementation — rework at this step costs least.
- Reference data. Select 10–15 short fragments with different articulation and annotate where lips must match phoneme and where smoothing is acceptable. The reference then serves as a ruler for all checks.
- Sync setup. Calibrate the alignment window of audio and frames, smoothness of transitions between lip positions, compensation for drift at the start of phrases. Parameters selected by reference, not by eye.
- Run and measurements. Calculate share of frames with noticeable discrepancy and number of places that must be edited by hand. Edits go into settings, not into separate files.
- Result acceptance. Go through the checklist together with the client: silence in pauses, absence of "floating" lips, even tempo. Rejection criteria stated in advance.
- Handover. Give configurations, pipeline scheme and restart instructions — the next video is assembled without us.
Handover package: configurations, regulations, accesses
The project is considered delivered not when the avatar speaks in a demo, but when your team conducts a new recording without us.
Therefore the handover contour is assembled in advance: configurations, schemes and regulations are fixed in your repository and on your infrastructure, not left in the contractor's working notes.
- Processing pipeline configuration — as annotated files: audio preparation parameters, phoneme annotation, lip mapping, post-processing and value boundaries beyond which articulation noticeably degrades.
- Data flow scheme: from source recording to finished video, indicating where intermediate artifacts are stored and by what rule they are deleted.
- Access regulation: roles and rights for the model, voice profile, source files and results catalog, separately — procedure for granting and revoking access when an employee changes.
- Voice rights: confirmed consents of the speaker and rights holder, terms of use of the synthesized voice and prohibition on transferring the profile to third parties.
- Solution documentation: architecture, settings reference, typical failures and how to fix them, change log.
- Editors' memo: how to record audio and annotate text so lip sync does not require manual refinement.
- Recovery plan: how to deploy the pipeline on a clean environment and in what time it again produces a result.
Acceptance goes not by sending an archive: an engineer on your side independently goes through the scenario — loading a new track, editing settings, releasing video, rolling back to the previous configuration version.
We observe at this meeting and refine places where the documentation was not sufficiently unambiguous.
Case: dubbing a training course without re-recording the speaker
Input — a training course of 46 episodes: lecturer on camera, clean sound, but the author physically could not return to the studio, and the course was needed in a second language.
Text translation was ready, all that remained was to "animate" source facial expressions and articulation to the new soundtrack without losing speaker recognition.
How the process was arranged
Lines were cut by pauses with an amplitude threshold, and the length of each segment was fitted to the source articulation interval — otherwise lips and sound diverge on the last syllables.
Numerals, abbreviations and terms were normalized into phonetic notation, proper names run through a pronunciation dictionary.
Talking head synthesis went in HeyGen using reference video, then segments were glued with the source sequence, and subtitles generated from the same timecode grid.
Metrics before and after
| Indicator | Before | After |
|---|---|---|
| Time per episode | 4–5 hours | 35–45 minutes |
| Share of lines with manual editing | ~35% | 6–8% |
| Acceptance iterations per episode | 3–4 | 1 |
The main gain came not from synthesis itself, but from preparation: aligning line lengths to source articulation and the phonetic dictionary removed most re-voicing. Remaining edits are pinpoint — terms with non-standard stress, they are fixed in one pass.
How does re-voicing differ from transferring facial expressions from video?
Re-voicing and facial expression transfer give a similar result, but are structured differently. In the first case the video remains source, and only articulation is rebuilt to the new track: the model matches speech phonemes with visemes and redraws the lip contour.
In the second, facial expressions, head tilt and micro-movements are taken from reference video, and the face or speaker is replaced — discrepancy with emotion and speech rhythm is visible immediately.
Hence the difference in requirements for the source. Re-voicing tolerates neutral delivery and short lines — the narrow area around the mouth is edited.
Articulation transfer requires a frontal face without occlusions and a stable angle: if a hand or microphone enters the frame, reference points disappear, and identity begins to "float" between frames.
The failure class also differs. In re-voicing the error is localized: labial and plosive consonants blur, lips lag on fast speech. In transfer it spreads across the whole frame — the jaw line trembles, cheek contour tears, gaze does not match head turn. Therefore we cut long monologues into short segments and check joints frame by frame.
The source chooses the approach. There is video of the needed person — transfer articulation and preserve their facial expressions. There is only a portrait and text — synthesize movement from scratch.
Need another language on a finished video — re-voicing is enough: models like Wav2Lip give a clean result on static delivery.
Technical doubts worth removing before start
Doubts are usually caused not by the lips themselves, but by what surrounds them: noisy sound, line edit, rights to appearance, multilingual versions and model behavior on long recordings. We address each point before start — so contested issues do not surface at acceptance.
What happens if the recording is noisy?
Articulation is built on recognized phonemes, so noise and level fluctuations affect accuracy. We run the track through noise reduction and leveling, and if speech boundaries are lost — we say so immediately, not mask with artifacts.
Do we need to rebuild the entire video because of one phrase?
No. Render is split into segments and linked to source audio timecodes: an edit affects only the changed fragment, neighboring ones are taken from cache. Replacing a phrase does not restart the full cycle.
How are rights to appearance and voice handled?
For a real speaker we take written consent for use of image and voice — fix channels, term and territory. For a synthetic image the question is removed, but the layout is agreed with you before render.
Can we get a version in another language?
Yes, but it is a separate render: articulation is tied to phonemes of a specific language, so simple track replacement gives discrepancy. Translated phrases are fitted by length so speech tempo remains natural.
What happens on long recordings?
On long segments drift accumulates between sound and picture, so we render in segments and join by reference points. Articulation remains even throughout the recording, not only at the beginning.
Let's discuss the lip sync task on your material
So the assessment is not "by eye", we analyze your material before work starts: look at what audio and video already exist, how the voice is recorded, how many episodes are in the series and what their editing rhythm is.
After analysis we return with a work plan, stages and list of what will be needed from you — without assumptions that later turn into rework at the acceptance stage.
- Task format: where talking head works — advertising, training course, social media — what line length is and what is more important: articulation accuracy in close-ups or speed of releasing the series.
- Source analysis: audio tracks, video, previous generation versions, if any — to understand what exactly did not suit you in them.
- Recording conditions: language and accent of the speaker, background noise and hum, pauses, voice overlaps. This is what most strongly affects lip sync quality.
- Trial episode: take one short fragment and run the full cycle — sync, facial expressions, final video assembly. You see the result before assessing the full volume.
- Assessment and work plan: number of episodes, stages, our area of responsibility and yours — texts, script edits, acceptance.
- Accesses and placement: where the solution will work, whether integration with your publishing system is needed, who owns materials and how rights are divided.
- Source handover: give audio, video, sync settings, launch scripts and instructions so the episode can be repeated without us.
Let's fix what we consider a finished episode and how many rounds of edits are included — then acceptance goes by rules, not by feelings. Describe the task and send one or two fragments: we'll return with an assessment, plan and trial episode.