How to remove the music from a video and keep the voice

A video with music baked into it used to be finished, for better or worse. Now the music comes out. Here is how the separation works, where it fails, and what to do with the voice you get back.

The reasons to want the music out of a video are all familiar. A clip you shot at an event with a song playing that will get the upload claimed. A screen recording you made with Spotify running in the background. An old promo where the voiceover is fine but the track is dated, licensed for a term that ended, or just wrong. A video you want to dub into another language without dubbing the song.

For most of the history of recorded sound this was not fixable. A mixed track is one waveform, and there was no undo. That is no longer true, and the change is recent enough that most people still assume it is.

What separation actually does

Stem separation is a model trained on a very large number of songs where it could see both the mix and the individual parts, learning what a voice looks like in a mixed signal against what everything else looks like. Given a new mix, it estimates the parts. The output is an approximation, but on well-recorded material it is a startlingly good one.

The rework-a-track task exposes it in two forms that matter for this job:

Isolate the voice returns the speech and drops the rest. For a talking-head clip with music under it, this is the one you want: the result is the voice with the bed gone.

Split out a stem returns one named part of a song (vocals, drums, bass, guitar, piano) or "everything else", the backing without the lead vocal. That is the reverse job: keeping the music and losing the singer, which is what you want when you have a song and need an instrumental to put narration over.

Both take an audio asset from your library. If the music is baked into a video clip, split its embedded sound onto its own lane first, so there is an audio track to hand to the model.

What comes back

Set expectations correctly and the tool will not disappoint you.

On a clean case, a voice recorded on its own microphone with music mixed under it afterwards, the separation is close to perfect. The voice comes back with its full tone and the music is gone, leaving something that sounds like the raw recording did before the bed went on.

On a harder case, a voice recorded in a room where music was playing, the separation is good but not clean. The music is gone, but so is some of the room around the voice, because the model cannot tell reverb-of-the-room from reverb-of-the-song. The voice sounds a little drier and a little more present than it did. Most people prefer it; some notice.

On the hardest case, music with a singing voice in it under a speaking voice, expect residue: the singer is a voice too, and the model will keep some of them. Isolate the voice first, then run the result through a denoise pass, and accept that a faint ghost may remain in the pauses. A new ambience bed at a low level under the whole clip covers the ghost, which is the fix in practice.

Two artefacts to listen for on any result. A slight watery or phasey quality on sustained vowels, which is the model's uncertainty made audible; it usually disappears once anything else is in the mix. And gaps in the very quiet passages where the model has taken everything out, which read as dead air; the same ambience bed fixes those too.

Do not leave the silence

This is the step people skip. Music that has been removed leaves a hole, and a voice over a hole sounds like an answerphone. Something has to go under it.

If the video needs music, put a new bed in: a generated track in the style you actually want, ducked under the speech. If it does not, put a quiet ambience under it, the room, the street, whatever the picture shows. The audience will never notice it is there and will absolutely notice if it is not.

Then treat the isolated voice the way you would treat any recording: fades at the edges, level set as the reference, everything else under it.

The dubbing case

The most useful application is the one people think of last. A video in one language with music under the speech, going to a second language.

A dub replaces the speech and should keep everything else, and separation is how to guarantee it does. Lift the audio off the clip, isolate the voice, and send that track to the dubbing task as audio: what comes back is the translation in a matching voice, on its own, with nothing under it. Then put the original backing (the "everything else" stem from the same separation) back on a lane beneath it. The music and the room are exactly what they were, the speech is in the new language, and the two never went through a model together. It is a few more steps than dubbing the mixed track, and it is the difference between a dub that sounds like the original and one that sounds like a phone call over it.

Removing a song from a video does not change whether you had the right to use it, but it does change what happens next, and it is worth being precise.

Platform claims are triggered by the audio they detect. Remove the song and the claim goes with it; the video is no longer matched against it. That is the honest use of the tool, and it is the common one: you are removing something you were never trying to use.

What it does not do is make a song yours. Splitting out an instrumental of a commercial track and putting narration over it is using the song, and a fingerprint match on the backing will find it. If you want a bed you can publish freely, the generated music route is the one that gets you there, and a generated instrumental in the style of the track you liked is usually as good in the mix as the original was.

When to give up on separation

Sometimes the right answer is to re-record. If the recording is a phone in a loud bar and the voice is barely above the music, what separation returns will be a phone in a loud bar with the music replaced by uncertainty. Ten minutes with a microphone in a quiet room beats an hour of trying to rescue it.

The same is true if you have the original. If a video was rendered from a project, go back to the project and change the music; do not separate the render. Separation is for when the mix is all you have.

The order that works

  1. Get the audio onto its own lane if it is baked into a video clip.
  2. Run isolate-the-voice to keep speech, or split a stem to keep music.
  3. Listen for residue on the vowels and in the pauses; denoise if there is singing under the speech.
  4. Fill the hole: a new bed, ducked, or a quiet ambience.
  5. Mix the voice as the reference and everything else under it.
  6. Do not build on a song you do not own; generate the bed instead.