Inconsistent instrument separation in long tracks with local inference

[Cactooz]

I have two problems when running the sam-audio-large model locally for instrument segmentation in music tracks, using just text prompts and audio input. I'm running SAM Audio with songs between 2 and 5 minutes, splitting each song into 10-second crops.

1. As the splitting of the song into crops, an instrument might be absent from some of the crops. When the instrument in the prompt is absent, the model can output random audio from the track instead of silence. Compared to when segmenting music on the website (https://aidemos.meta.com/segment-anything/editor/segment-audio), if the prompted instrument isn't present in the inputted audio, the target is silent or quiet noise.

2. When the instrument exists in the crop, I also have issues where ~10-20% of the crop segmentations fail in one of the following ways
   a. fails to segment (fully misses the instrument)
   b. segments the wrong instrument
   c. switching between segmenting a single and sometimes multiple instruments
   No number of rerankings or crop durations has been able to fully fix these issues. Having rerank 2-3 makes it better than 1, but I don't see much improvement after that.

• Are there any parameters or different prompts I can use to mitigate these issues?
• Is there a specific recommended strategy for maintaining consistency for longer songs? (I'm now using multi-diffusion on the chunks as well, without much difference)
• How is the website demo version configured, does it have some specific thresholds or post-processing? (as it seems not to have the same issues)
• Is the model able to understand plural inputs like guitars? Or is it more sensitive to some instruments? (e.g. in my current tries, for the prompt "guitars" some crop segments only the acoustic guitar, and some both the acoustic and electric guitar.)

Model and settings I've tried:

• sam-audio-large and sam-audio-large-tv
• Text prompt only for audio files (visual model disabled)
• Music durations are split into crops between 10 and 80 seconds
• Batches of crops between 1 and 8
• predict_spans=False
• reranking_candidates between 1-12
  • reranking with only CLAP
  • reranking with both CLAP and SAJ
  • changed the CLAP vs SAJ ratios

Prompts I've tried:

• guitar
• guitars
• electric guitar
• piano
• synth
• synthesizer
• saxophone
• drums

Hardware & OS:

• NVIDIA RTX 6000 Ada (48GB)
• Ubuntu 24.04

Inference code:

inputs = list(torch.split(source, num_samples, dim=-1))
if inputs[-1].shape[-1] < num_samples:
    inputs[-1] = torch.nn.functional.pad(inputs[-1], (0, num_samples - inputs[-1].shape[-1]))

batches = [inputs[i:i + BATCH_SIZE] for i in range(0, len(inputs), BATCH_SIZE)]

targets = []
residuals = []
with torch.inference_mode():
    for batch_inputs in tqdm(batches, desc="Processing batches", unit="batch"):
        batch = processor(
            audios=batch_inputs,
            descriptions=[DESCRIPTION] * len(batch_inputs),
        ).to(device)

        result = model.separate(batch, predict_spans=False, reranking_candidates=5)
        
        targets.extend(result.target)
        residuals.extend(result.residual)

full_length = source.shape[-1]
target = torch.cat(targets, dim=-1).clamp(-1, 1).cpu()[..., :full_length]
residual = torch.cat(residuals, dim=-1).clamp(-1, 1).cpu()[..., :full_length]

[chevalierNoir]

@Cactooz

As the splitting of the song into crops, an instrument might be absent from some of the crops.

When the instrument is absent, the model behaves undesirably. It would be better to first run audio event detection and only provide the prompt when the instrument is present.

Are there any parameters or different prompts I can use to mitigate these issues?

Generally reranking + span predicting would help. Also one common issue we noticed is that some instrument are easier to separate than the others in a mixture. Say in an audio with guitar + violin and guitar is easier to separate, you can try prompt with guitar first and get violin in the residual. By iteratively doing so, you can get all instruments out.

Is there a specific recommended strategy for maintaining consistency for longer songs? (I'm now using multi-diffusion on the chunks as well, without much difference)

For consistency, do you mean the artifacts in the boundary of chunks? Multi-diffusion should produce quite consistent results per our observation. Do you follow the same chunk size and stride as the paper?

How is the website demo version configured, does it have some specific thresholds or post-processing? (as it seems not to have the same issues)

The demo is using sam-audio-large, and use span prediction + reranking size =8

Is the model able to understand plural inputs like guitars? Or is it more sensitive to some instruments? (e.g. in my current tries, for the prompt "guitars" some crop segments only the acoustic guitar, and some both the acoustic and electric guitar.)

It typically doesn't understand plurals. Separating different guitars is a bit challenging. You could prob try using span that might help.

[Cactooz]

Thanks for your response @chevalierNoir

When the instrument is absent, the model behaves undesirably. It would be better to first run audio event detection and only provide the prompt when the instrument is present.
Generally reranking + span predicting would help. Also one common issue we noticed is that some instrument are easier to separate than the others in a mixture. Say in an audio with guitar + violin and guitar is easier to separate, you can try prompt with guitar first and get violin in the residual. By iteratively doing so, you can get all instruments out.

I tried to do detection checking for every chunk, though for similar instruments detection can still sometimes fail. Running SAM then outputs something unrelated. Maybe span prediction would help for these chunks as well.

For consistency, do you mean the artifacts in the boundary of chunks? Multi-diffusion should produce quite consistent results per our observation. Do you follow the same chunk size and stride as the paper?

Overall, I didn't find many artifacts at the boundaries for my test, but I switched to multi-diffusion to fully remove them.

With "consistency" in multi diffusion, I mean that I still observe the model sometimes struggling to identify the prompted instrument throughout the whole track. For example, if there is both a guitar and a piano, it might jump between them across different chunks. If prompted "piano," most of the output would be piano, but sometimes it outputs a piano-sounding guitar.

I tried a bunch of different chunk sizes and strides in the multi diffusion, finally ending with 20-second chunks and a 16-second stride, just 1 second more in stride than the paper.

The demo is using sam-audio-large, and use span prediction + reranking size=8

Perfect, then I'll try to reproduce everything fully :)

It typically doesn't understand plurals. Separating different guitars is a bit challenging. You could prob try using span that might help.

So it would be best to just use simple singular-word prompts like "guitar", "piano", "bass", "drum", or similar?

Thanks for your input!

On a song-by-song basis, I think everything works quite well overall if the prompted instruments are present throughout, and adding span predictions will probably make it even better, fixing some chunks that fail. I need to tinker some more to improve consistency for large-batch work of multiple songs automatically.

[Preet-Sojitra]

I have two problems when running the sam-audio-large model locally for instrument segmentation in music tracks, using just text prompts and audio input. I'm running SAM Audio with songs between 2 and 5 minutes, splitting each song into 10-second crops.

1. As the splitting of the song into crops, an instrument might be absent from some of the crops. When the instrument in the prompt is absent, the model can output random audio from the track instead of silence. Compared to when segmenting music on the website (https://aidemos.meta.com/segment-anything/editor/segment-audio), if the prompted instrument isn't present in the inputted audio, the target is silent or quiet noise.

2. When the instrument exists in the crop, I also have issues where ~10-20% of the crop segmentations fail in one of the following ways
   a. fails to segment (fully misses the instrument)
   b. segments the wrong instrument
   c. switching between segmenting a single and sometimes multiple instruments
   No number of rerankings or crop durations has been able to fully fix these issues. Having rerank 2-3 makes it better than 1, but I don't see much improvement after that.


* Are there any parameters or different prompts I can use to mitigate these issues?

* Is there a specific recommended strategy for maintaining consistency for longer songs? (I'm now using multi-diffusion on the chunks as well, without much difference)

* How is the website demo version configured, does it have some specific thresholds or post-processing? (as it seems not to have the same issues)

* Is the model able to understand plural inputs like guitars? Or is it more sensitive to some instruments? (e.g. in my current tries, for the prompt "guitars" some crop segments only the acoustic guitar, and some both the acoustic and electric guitar.)

Model and settings I've tried:

* `sam-audio-large` and `sam-audio-large-tv`

* Text prompt only for audio files (visual model disabled)

* Music durations are split into crops between 10 and 80 seconds

* Batches of crops between 1 and 8

* `predict_spans=False`

* `reranking_candidates` between 1-12
  
  * reranking with only CLAP
  * reranking with both CLAP and SAJ
  * changed the CLAP vs SAJ ratios

Prompts I've tried:

* guitar

* guitars

* electric guitar

* piano

* synth

* synthesizer

* saxophone

* drums

Hardware & OS:

* NVIDIA RTX 6000 Ada (48GB)

* Ubuntu 24.04

Inference code:

inputs = list(torch.split(source, num_samples, dim=-1))
if inputs[-1].shape[-1] < num_samples:
inputs[-1] = torch.nn.functional.pad(inputs[-1], (0, num_samples - inputs[-1].shape[-1]))

batches = [inputs[i:i + BATCH_SIZE] for i in range(0, len(inputs), BATCH_SIZE)]

targets = []
residuals = []
with torch.inference_mode():
for batch_inputs in tqdm(batches, desc="Processing batches", unit="batch"):
batch = processor(
audios=batch_inputs,
descriptions=[DESCRIPTION] * len(batch_inputs),
).to(device)

    result = model.separate(batch, predict_spans=False, reranking_candidates=5)
    
    targets.extend(result.target)
    residuals.extend(result.residual)

full_length = source.shape[-1]
target = torch.cat(targets, dim=-1).clamp(-1, 1).cpu()[..., :full_length]
residual = torch.cat(residuals, dim=-1).clamp(-1, 1).cpu()[..., :full_length]

Hi, did you find the workarounds because I am facing similar issues. I am trying to seperate human speech from the videos of animals and if speech is not present it gives weird results. For example, for one video where it was only dog barking with some background voice of machine and I gave prompt "human speaking" and it sperated dog sound as human sound.

[Preet-Sojitra]

Hi @Cactooz , did you find the workarounds because I am facing similar issues. I am trying to seperate human speech from the videos of animals and if speech is not present it gives weird results. For example, for one video where it was only dog barking with some background voice of machine and I gave prompt "human speaking" and it sperated dog sound as human sound.

[Cactooz]

Hello @Preet-Sojitra sadly I did not find any major workarounds for using the model with many files at once, but for single files I could get good results after a lot of tinkering. However, I did implement a few things that made it more accurate:

• Iterative reranking, instead of always having X reranks, run reranking iteratively if they don't reach the threshold. This allows the model to sometimes do only 1 rerank, but for harder parts it can do 20 or any number you set as the max.
• Use multi-diffusion to avoid abrupt changes between the chunks.
• Set the spans or predict the spans if you know where in the song the specific thing to segment is. Ignore all chunks where it does not exist at all.
• Try to prompt it slightly different until it segments the correct thing.

It seems like others (#24) have gotten better results, so I do not know what the issue really is ¯\_(ツ)_/¯