Lip sync, dubbing or a voice clone: which one do you actually need?

Change the words, change the language, change the voice: three different jobs, three different tools, and a search box that returns all three for every one of them. Here is which one you want, by what you are actually trying to do.

Three different microphones in a row

Three tools live near each other in every AI video product, and they are confused for one another daily, because all three involve a voice and most involve a face. They do different things, they take different inputs, and choosing the wrong one costs a generation and a puzzled look at the result. The quickest way to sort them is by what goes in and what comes out.

What each one is

Lip sync takes a face and an audio track and reshapes the mouth so the face appears to say the audio. The voice is not changed; it is whatever was in the track. The words are not changed; they are whatever was in the track. Only the picture changes. Input: a video clip or a still portrait, plus audio. Output: a clip of that face saying that audio.

Dubbing takes speech and translates it into another language, spoken aloud, keeping the timing and the delivery and on some models the original speaker's voice. Attach a video clip and the mouth is re-synced to the new speech as part of the job; attach a bare audio track and only the audio comes back. Input: a clip or a track with speech in it, plus a target language. Output: the same thing in the other language.

A voice clone takes thirty seconds of someone's speech and makes a voice that can say anything you type. It produces audio only, has no idea about faces, and does not translate anything; it reads the text it is given in the language it is given. Input: a recording, once, then text forever after. Output: that voice saying that text.

Put differently: lip sync changes the picture, dubbing changes the language, cloning changes the speaker. Each leaves the other two alone.

The six jobs, and which tool each wants

"The presenter said the wrong thing in one line." You want the same person, the same language, different words. Generate the corrected line in a cloned version of their voice (or re-record it), then lip-sync the shot to the new audio. Two tools, in that order: the voice first, because it is cheap to re-roll, and the picture once the audio is right.

"I need this video in Spanish." Dub it. Attach the clip and you get Spanish speech with the mouth re-synced in one job. If the speaker is off camera, attach the audio only; it is cheaper and faster because no frames are rendered. You do not need a clone and you do not need a separate lip-sync pass; dubbing includes it. Generate captions from the dubbed track afterwards if you want Spanish subtitles too.

"I want to narrate my videos without recording every time." A clone, and nothing else. Record the thirty-second passage once, then type each narration and generate it in your voice. There is no face in this job, so lip sync is irrelevant, and no translation, so dubbing is too. The narration guide is the long version.

"I have a portrait and a script and no camera." Lip sync, fed a still. One clear, front-on face plus a voice track (generated from a stock voice or a clone) becomes a talking portrait. This is the cheapest talking-head there is, and it is a single job: you do not animate the photo into a clip first. The lip sync guide covers what makes a face work for it.

"The dub is good but the mouth still looks dubbed." This usually means the dub was audio-only and laid over the original picture. Run the dub again with the clip attached, or lip-sync the clip to the dubbed track. Either way the mouth follows the new speech. Dubbing on its own, with no picture attached, leaves the original mouth movements, which is fine for an off-camera narrator and wrong for a face.

"I want my own voice in Spanish." This one is two tools, and the order is not obvious. A clone reads text in whatever language you give it, so you could translate the script and generate the Spanish in your clone. But dubbing already does this on models that carry the speaker's voice through, with the timing preserved, from the original audio. Try the dub first; reach for the clone only if the carried-over voice does not sound like you, or if the script needs rewriting rather than translating.

The things none of them do

None of them write. A clone says the text you give it, and if the text is a document rather than speech it will sound like one; the voiceover guide is about that. Dubbing translates, which is not the same as localising: an idiom, a joke, a reference to a local shop crosses the language and loses the point.

None of them fix a bad source. A dub of mumbled, noisy speech is a confident translation of the wrong words. Lip sync on a face in profile, or a face the size of a coin in a wide shot, has no mouth to find. A clone from a recording with music under it is refused rather than cloned into nonsense. Settle the source first.

And none of them grant permission. A clone of someone else's voice needs their explicit consent, and the set-up asks for it. A lip sync of a real person saying words they did not say is the same question with a face on it. The tools are for your own work and the people who have agreed to be in it.

The decision in one line

If the picture is wrong, lip sync. If the language is wrong, dub. If the speaker is wrong, or absent, clone. If more than one is wrong, fix the audio first and the picture last, because audio is cheap to retry and picture is not.

The order that works

  1. Name what is wrong: the words, the language, the speaker, or the mouth.
  2. Fix the audio first: clone, re-record or dub, and listen before anything visual is generated.
  3. Attach the clip to a dub when a face is on screen; audio-only when it is not.
  4. Lip-sync last, to final audio, on a clear front-on face.
  5. Captions from the final track, in the final language.