AI stem separation works in three steps: it converts a finished mix into a spectrogram, has a neural network predict a mask for each instrument, then rebuilds those masked pieces back into audio. That's the whole trick, and the rest of this post is about what happens inside each step.
Quick terminology note before we go further. What these tools actually pull out are individual instrument tracks, so calling them multitracks would be more accurate. "Stems" technically means submixes. The industry settled on "stems" anyway, so that's the word everyone uses, but it's worth knowing the difference.
TABLE OF CONTENTS
The basic pipeline, start to finish
Here's the sequence, plainly. There are four moves.
- Convert to a spectrogram. A Short-Time Fourier Transform (STFT) turns the waveform into a 2D time-frequency picture — a grid where each little cell holds how much energy sits at a given frequency at a given moment.
- Recognize patterns. The network has heard thousands of examples of isolated instruments during training, so it knows roughly what a kick or a vocal looks like in that picture.
- Predict a mask. For each instrument, it outputs a mask that, when laid over the mix, leaves only that instrument behind.
- Convert back to audio. An inverse STFT rebuilds each masked spectrogram into a listenable file.
Why bother with the spectrogram at all? Because a raw waveform tells you when something happens and how loud it is, but a spectrogram tells you what frequencies are present. That second view is way easier for a network to work with.
Picture it like this. A kick drum shows up as a cluster of bright pixels down in the low frequencies, popping on each beat. A vocal shows up as a shifting pattern in the midrange that moves around as the singer changes notes. The network is basically learning to spot those shapes.
What a spectrogram mask actually does
This is the heart of it, so let's slow down. A mask is a grid of weights between zero and one, and there's one value for every time-frequency cell in the spectrogram. That value decides how much of that cell belongs to a given instrument.
A one means "this cell is all vocal." A zero means "none of it." A 0.4 means "about forty percent vocal, the rest belongs elsewhere."
That in-between part matters. Older approaches used hard masks that made a flat yes-or-no call on each cell, which tends to sound chopped up. Soft masks let energy get shared between instruments in a more nuanced way, and that's why they sound more natural.
One more detail worth knowing: the masks for all the instruments are designed to roughly sum to one across each cell. So all the energy in the original mix gets accounted for and divided up, rather than invented or thrown away.
The phase problem nobody mentions
Here's the catch most explainers skip. The mask only works on magnitude — how much energy is in each cell. But audio also has phase, and phase is a separate thing entirely.
Older spectrogram models like Spleeter and OpenUnmix don't estimate the phase for the separated instrument. They just borrow the phase from the original mix and paste it onto the estimated magnitude before converting back to audio. It's a shortcut, and it works well enough — but that borrowed phase is a genuine source of the weird sounds you'll sometimes hear. It's a known limitation, not a secret.
Spectrogram, waveform, and hybrid models
A little context helps here, without turning this into a tool roundup. Around 2017, neural nets started blowing past everything that came before. A big reason was training data — datasets like MUSDB18, which packs 150 full songs with isolated vocals, drums, bass, and "other" that models could learn from.
From there, two camps formed:
- Spectrogram-domain, like Spleeter. It works exactly how we described above — mask the time-frequency picture, then rebuild.
- Waveform-domain, like Demucs. It skips the picture and processes the raw waveform directly, which helps it hold onto detail that masking tends to smear.
The field eventually met in the middle. HTDemucs is a hybrid — it runs both the waveform and the spectrogram paths at once, with a transformer letting information flow between them. That's why it beats models that only used one view.
Now, a correction you'll see a lot. Plenty of consumer articles still call HTDemucs "the current state of the art." That's out of date. Band-split transformer models like BS-RoFormer and Mel-Band RoFormer now sit at the top of the benchmarks. HTDemucs is still very good and very widely used — it's just not the leader anymore. If you want a deeper read on the general concept, EducBA has a solid overview of audio source separation.
Why vocals and drums separate cleaner than mid-range instruments
This is the part most people actually feel when they run a track through one of these tools. Vocals and drums come out clean. The synths and guitars come out iffy. There are two reasons.
First, distinct fingerprints. Vocals have a clear harmonic structure, vibrato, breath sounds. Drums have sharp transient hits with recognizable frequency signatures. Bass lives in a defined low range with its own envelope. The network has seen these shapes thousands of times, so it grabs them confidently. This is the same reason vocal isolation works as well as it does, and if that's your goal, we've got a full walkthrough on how to isolate vocals from a song with AI.
Second, the overlap problem. A lead synth can sit in the exact same frequency range as a vocal, with harmonics that smear into the drum attacks. The model has no clean fingerprint to lock onto, so you get bleed and smearing. And the "other" stem is a catch-all — on dense arrangements it turns into a muddy pile of things that don't really belong together.
Drums specifically got a boost from waveform and hybrid handling. Because those models keep the raw waveform in play, they preserve the sharp transients that spectrogram-only masking tends to soften. That's why recent models sound noticeably tighter on snares and hi-hats.
It's also genre-dependent. Clean studio pop and well-produced hip-hop split beautifully. Dense orchestral music, live recordings with room bleed, and reverb-soaked genres are genuinely hard — there's just too much overlap for the model to untangle.
The low-end overlap: 808s, bass, and kick
The low end is its own headache, and it's worth spelling out because producers run into it constantly. Below roughly 200 Hz, bass guitar and kick drum start sharing a lot of energy. Below about 80 Hz, everything blurs into one rumbling mass in the spectrogram.
808-style kicks make this worse. They extend way down into sub-bass and overlap with bass lines, so the model can't cleanly tell them apart. The result is exactly what you'd expect — you'll hear the 808 bleeding into both the drums output and the bass output. It's not a bug in the tool. The information just isn't separable.
The artifacts you'll actually hear
Let's be honest up front: you will get artifacts. Every tool does, because they're baked into the problem itself, not into one company's software. There are two main kinds.
Bleed, or crosstalk. This is bits of one instrument ghosting into another stem. A hi-hat faintly haunting the vocal, or a little bass leaking into the drums. It happens most where instruments overlap in frequency.
Musical noise, sometimes called "birdies." These are little chirps and flutters that come from isolated holes and peaks the time-frequency processing leaves behind. They tend to show up in quiet moments and on reverb tails.
Make sure you audition your results on headphones. A lot of this stuff is subtle, and it hides in the noise on laptop speakers. Let's give it a listen closely, then decide for yourself what's actually usable for your project — for some jobs a little bleed is fine, for others it's a dealbreaker. If you're mainly after removing a vocal rather than keeping stems clean, our piece on how AI vocal removers work gets into how good these really are.
Quick recap: how it works
- Spectrogram plus mask is the core process — convert the mix to a time-frequency picture, predict a mask per instrument, rebuild the audio.
- Vocals and drums separate best because they have distinct spectral and temporal fingerprints the model recognizes easily.
- Low-end and midrange overlap causes bleed, which is why 808s, bass, kick, synths, and guitars are the hard cases.
- Artifacts are inherent to the problem, not a defect of one tool, so use your ears and audition everything on headphones.
Frequently Asked Questions (FAQs)
Is AI stem separation lossless?
No, AI stem separation is not lossless — some artifacts are always present. The process estimates each instrument rather than recovering a true original recording, so you'll get some bleed and musical noise. How noticeable it is depends on the track, the model, and how dense the arrangement is.
Why do vocals separate so well?
Vocals separate well because they have a distinct fingerprint the model recognizes easily — clear harmonic structure, vibrato, and breath sounds that stand out from other instruments. The network has trained on thousands of vocal examples, so it can grab that pattern confidently even inside a busy mix.
Why do 808s bleed into the drums and bass stems?
808s bleed into both drums and bass because they extend deep into sub-bass and overlap spectrally with the bass line. Below about 80 Hz, kick, 808, and bass all blur into one mass in the spectrogram, so the model can't cleanly assign that energy to a single stem.
What are birdies or musical noise in stem separation?
Birdies, also called musical noise, are little chirping or fluttering artifacts left behind by time-frequency processing. They come from isolated holes and peaks in the spectrum after masking, and they tend to show up in quiet passages and reverb tails. Headphones make them much easier to catch.
Are stems and multitracks the same thing?
No, stems and multitracks aren't the same. A multitrack is a single instrument or source on its own track, while a stem is technically a submix of several sources grouped together. AI tools actually output multitracks, but the industry calls them stems anyway — so the two words get used interchangeably.
Final Thoughts
Once you know it's spectrograms, masks, and pattern recognition under the hood, the results make a lot more sense. Vocals and drums come out clean because they're easy to spot. The low end and the midrange bleed because that energy genuinely overlaps and no amount of clever math fully separates what was never separate to begin with.
So go run a few tracks through, and trust your ears on what's good enough for the job. The tech is genuinely impressive now — just don't expect it to hand you back the original session.
Some of the links within this article are affiliate links. These links are from various companies such as Amazon. This means if you click on any of these links and purchase the item or service, I will receive an affiliate commission. This is at no cost to you and the money gets invested back into Audio Sorcerer LLC.