Where is multi-diffusion implementation? Currently repo is really weak

[FurkanGozukara]

I am using vocals extraction and it is really weak working bad

According to paper multi-diffusion is working best but where is it?

[chevalierNoir]

Multi-diffusion is not implemented yet in this repo. An example vocal extraction can be found in in here. To help debug, you could try 30s chink first, which doesn't need multi-diffusion in principle.

[FurkanGozukara]

Multi-diffusion is not implemented yet in this repo. An example vocal extraction can be found in in here. To help debug, you could try 30s chink first, which doesn't need multi-diffusion in principle.

i already have 20 second chunk + 5 second overlap

but quality is not great imo

even with 4 candidates and ranker

here listen

01 - Michael Jackson - Billie Jean_sam (2).mp3

[chevalierNoir]

There are boundary artifacts, which should be reduced by multi-diffusion. For leaking of non-vocals in some timestamps, you can try improving iteratively. For example, use span (or span prediction) for "synth" on the output track and take the residual which should be just vocal.

[FurkanGozukara]

There are boundary artifacts, which should be reduced by multi-diffusion. For leaking of non-vocals in some timestamps, you can try improving iteratively. For example, use span (or span prediction) for "synth" on the output track and take the residual which should be just vocal.

you mean manually iteratively improve right? ye that is too much thing i need it automatic

CODEX implemented multi diffusion but i guess it wanst accurate since it didnt improve my results

[chevalierNoir]

There are boundary artifacts, which should be reduced by multi-diffusion. For leaking of non-vocals in some timestamps, you can try improving iteratively. For example, use span (or span prediction) for "synth" on the output track and take the residual which should be just vocal.

you mean manually iteratively improve right? ye that is too much thing i need it automatic

CODEX implemented multi diffusion but i guess it wanst accurate since it didnt improve my results

You can automate that with span prediction. Or apply reranking at segment-level and apply cross-fading to each audio segment. They are not proposed in the paper but could potentially improve performance.

[FurkanGozukara]

There are boundary artifacts, which should be reduced by multi-diffusion. For leaking of non-vocals in some timestamps, you can try improving iteratively. For example, use span (or span prediction) for "synth" on the output track and take the residual which should be just vocal.

you mean manually iteratively improve right? ye that is too much thing i need it automatic
CODEX implemented multi diffusion but i guess it wanst accurate since it didnt improve my results

You can automate that with span prediction. Or apply reranking at segment-level and apply cross-fading to each audio segment. They are not proposed in the paper but could potentially improve performance.

just implemented and tested with vocal extraction

it is literally almost same no difference

i used 4 candidates + SAM-Audio Judge

20 seconds split 5 seconds overlap

No Predict spans 01 - Michael Jackson - Billie Jean_sam_residual.mp3

Predict spans 01 - Michael Jackson - Billie Jean_sam.mp3

[chevalierNoir]

There are boundary artifacts, which should be reduced by multi-diffusion. For leaking of non-vocals in some timestamps, you can try improving iteratively. For example, use span (or span prediction) for "synth" on the output track and take the residual which should be just vocal.

you mean manually iteratively improve right? ye that is too much thing i need it automatic
CODEX implemented multi diffusion but i guess it wanst accurate since it didnt improve my results

You can automate that with span prediction. Or apply reranking at segment-level and apply cross-fading to each audio segment. They are not proposed in the paper but could potentially improve performance.

just implemented and tested with vocal extraction

it is literally almost same no difference

i used 4 candidates + SAM-Audio Judge

20 seconds split 5 seconds overlap

No Predict spans 01 - Michael Jackson - Billie Jean_sam_residual.mp3

Predict spans 01 - Michael Jackson - Billie Jean_sam.mp3

Do you apply span prediction directly when using as input? In the iterative method I described, you should apply it on the output audio: use instruments of the song as prompt (e.g., base, synth, etc) to run span prediction --> feed the detected span to the separation model if any is detected --> take the residual stream (i.e., vocals)

[FurkanGozukara]

@chevalierNoir implemented that approach and it fully failed

new iterative approach 01 - Michael Jackson - Billie Jean_sam.mp3

[chevalierNoir]

@FurkanGozukara there is prob a bug in your implementation and you need to check. The first few seconds in your current output is different from what it was in your previous audio, which should be identical since the latter is just silence.

[FurkanGozukara]

@FurkanGozukara there is prob a bug in your implementation and you need to check. The first few seconds in your current output is different from what it was in your previous audio, which should be identical since the latter is just silence.

do you have anywhere implementation code published?

[chevalierNoir]

@FurkanGozukara there is prob a bug in your implementation and you need to check. The first few seconds in your current output is different from what it was in your previous audio, which should be identical since the latter is just silence.

do you have anywhere implementation code published?

Please refer to my previous reply, this part: use instruments of the song as prompt (e.g., base, synth, etc) to run span prediction --> feed the detected span to the separation model if any is detected --> take the residual stream (i.e., vocals)

[FurkanGozukara]

@FurkanGozukara there is prob a bug in your implementation and you need to check. The first few seconds in your current output is different from what it was in your previous audio, which should be identical since the latter is just silence.

do you have anywhere implementation code published?

Please refer to my previous reply, this part: use instruments of the song as prompt (e.g., base, synth, etc) to run span prediction --> feed the detected span to the separation model if any is detected --> take the residual stream (i.e., vocals)

from your description alone, CODEX 5.5 XHigh is not able to add meaningful implementation