retrocirce / zero_shot_audio_source_separation

The official code repo for "Zero-shot Audio Source Separation through Query-based Learning from Weakly-labeled Data", in AAAI 2022

Hi I’m Knut(Ke) Chen 👋

zero_shot_audio_source_separation's Issues

How can train using my own dataset

Same as title : How can train using my own dataset ?

I'm having trouble inferring a 15 second song.

I was able to use it now with your settings, but I'm facing the following error when playing with a 15 second song, how to fix it? my gpu is RTX 2060 6GB, thanks once again.

Is it possible to have result sound kept for each individual sample ?

During my testing, I found out that 6 samples in one sample folder versus 6 individual separations give different results and I assume this is because of some kind of generalization (i don't know how this works :P). Also found this out one time when I put 500 instrument samples the output was basically NULL.

Is there any mode or a script in the code that could change it for example to include audio out of each sample extraction in one mixdown file or at least every sample having its output file it would make things so much easier for me instead of doing it manually because this method serves me better for things I'm testing.
(i tried gather mode, I don't know if that's the idea but output still doesn't seem to equal 6 individual sample inferences combined)
Thank you!

Quality not up to par with the demo maybe.

Hi!
First of all, I think this tool is amazing as it does help improve quality when used with other tools such as MDX
But, I noticed exports tend to have a bit of leak in them which doesn't seem to appear to be as bad as in the demo.
I have a few questions:

There are three checkpoint files and so far I know one of them only works for htsat config which is: htsat_audioset_2048d.ckpt
The other 2 I don't know what is the difference between them and where are they supposed to be used in the config file (zeroshot_asp_full.ckpt and zeroshot_asp_held_out.ckpt)

All predicted files (other, vocals, bass, drums) have the same outputs and I am not sure if this is because of the type of inference but changing "dataset_type" in htsat config does not seem to make any difference.

Are samples supposed to be only 10 seconds? is that possible to increase or if there is no point in doing so?

And last question what would you recommend setting up in configs for the best quality possible?

Thank you.

Can't use overlap + clicks issue

"raise "ffmpeg does not exist. Install ffmpeg or set config.overlap_rate to zero."
TypeError: exceptions must derive from BaseException"

Yes, I have ffmpeg installed

Basically that's it. I tried overlap 1.0 but it doesn't seem to work, 0.0 works though.

I wanted to try it because the outputs always have periodic clicks in them

Different length of input and output

Hello. After applying model the size of output is slightly different from input. For example 9265664 became 9216000. It gives problem for validation on MUSDB dataset as well as creating inverse of extracted stem. What is the best way to align output to input. Right now I do the following:

    if audio.shape[-1] > vocals.shape[-1]:
        audio = audio[..., :vocals.shape[-1]]
    invert = audio - vocals

ERROR: Unexpected bus error encountered in worker.

I get this error running the model on replicate for a 5 minute song (If I truncate the song to 1 minute it works.)

Any ideas?

ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).

problem with installation

Hi.
I encountered problems when installing, there might be issues with the requirements.txt.
I ran pip install -r requirements.txt on a conda environment and then got an error about user requesting numpy 1.22.0.
Numba 0.55.1, which is in the requirements, doesn't support 1.22.0.
So I installed numpy 1.21.5. When running main.py I got omegaconf error about unsupported value type.
Then I did pip install omegaconf -U and pip install hydra-core --upgrade and that fixed the issue.
I've been enjoying the program so far. The inferencing runs well on my cpu and I am able to do things that I wasn't able to with other source separation algorithms.

Poor results (and doubt)

Hello how are you?

Finally with your explanations I managed to use Zero Shot ASS,

However, when examining the results, using the test keys "vocals",

I got unattractive results, I didn't understand why, being that in the examples of the videos they sound very good,

So my question is as follows,

In the "Data/Query" folder, to get decent results, how many examples should I put there?

Because I did as follows,

In "mixture.wav" is the complete instrumental with the vocals,

And in the "Query" folder I put 15 examples of clean vocals (from the singer in question),

So am I doing it wrong?

NOTE: Each example is on average 7 to 15 seconds long in .wav format

I tested it on other examples and got the same pattern (unattractive results),

When I refer to unattractive results, I mean almost no final changes, the files in the "wavoutput" folder are practically unmodified.

I only tested the model (htsat_audioset_2048d.ckpt) because the other two models are incompatible with this code, I received checkpoint errors (because the script is not standardized to accept them).

So if you can let me know why I would be grateful,

My thanks in advance,

Lucas Rodrigues.

Code release

Hi, authors. When will the code be released?
Thanks.

Total lenght of 'dataloader' across ranks is zero. Please make sure that it returns at least 1 batch.

This is weird, it was working a few days ago when it suddenly spit out this error. In my understanding it's suggesting that the (reference) samples are not enough? Even though I'm using the same song and samples as prior to this error, I haven't changed anything

AssertionError: there should be a saved model when inferring

Hi how are you?

Fllowed the guidelines for using Zero Shot ASS but I am getting the following error when trying to use inference (same error occurs on google colab and on my local computer)

I already downloaded the checkpoint templates and already made the directory changes in the config.py file, but I still get this error, how can I resolve this issue? again thank you very much!

RETURN ERROR:

This is my already configured config.py file

Colab notebook?

Are there any plans to create a Colab notebook for this model?

Possible to run on CPU only ?

Doubt about increasing samples (Query)

Would it be possible to increase the examples in the Query folder? I noticed that when I put more than 74 .wav files I get this error: "ZeroDivisionError: integer division or module by zero".

So I would like to know if it is possible to increase the samples, thanks again.

How to use

Hello how are you? would you be able to tell us how to use this tool, thank you very much!

Reproducing paper results

Unfortunately I wasn't able to get close to paper results. My results on MUSDB18HQ dataset are following:

SDR bass: 1.0778
SDR drums: 2.5914
SDR other: 0.6353
SDR vocals: 2.9829

My method - I used as query 100 train tracks from MUSDB18HQ dataset. Then I created averaged 2048-vector for ['bass', 'drums', 'other' and 'vocals']. Then I split test tracks using this vector and calculated the SDR value. May be I missing something?

Export pred_.wav to stereo

Hello how are you? I'm really enjoying Zero Shot ASS,

I noticed that the files exported after the inference, are in WAV Mono format,

So my question is, would it be possible to somehow export the final result in stereo (2 channels)? If so, I imagine I should modify some lines in the config.py file, no?

Thanks again.