sigsep / open-unmix-js Goto Github PK

while waiting users should be able to know the current wait status (aka. current block out of maximum number of blocks)

Long tracks take forever currently and also might crash the browser. Can we trigger a warning when the track is longer than 30s to allow user to abort the separation?

The original open-unmix model is quite slow and currently only runs on CPU.

• can you test if it runs with the new WASM backend?

e.g. this affects the package.json files

Instead of providing an upload we should also provide a url option. This should especially be used to so that we can provide a demo track.

a callback function should be supported so that the UI would be able to get the current progress in percent.

e.g. remove

• spleeter folder and python code
• in the source code, add contributions when code is from (e.g. tfjs, magenta, koestra)

https://github.com/kaliatech/web-audio-recording-tests

currently the model is expected to be running stereo. However, I also uploaded a mono model for speech enhancement where nb_channels = 1.

This can be implemented in the following way:

• if the model only required as a single channel audio spectrogram but the audio input signal has two channels, the input spectrogram can just be averaged across the channel dimension and after model(input) the output mask would need to be duplicated again.
• if the model only works with single channel and the audio input is also single channel, the preprocessing code would need to adjusted. This can be useful when we use the user input (from microphone) to record a mono signal and process it

recent chrome versions are fine using webworkers + gpu. See this article: https://medium.com/@wl1508/webworker-in-tensorflowjs-49a306ed60aa

In this task you get familiar with the most relevant signal processing that we need to perform separation with open-unmix or spleeter.
Pre-processing should be entirely in javascript but tested against python code:

Working pre-processing prototype

• loads audio from a url into array
• playback the audio using an tag or by creating an audiobuffer
• Perform the STFT/ISTFT on audio signal
• write unit test to evaluate the reconstruction error (should be in the ballpark of <10^-3)
• extended unit tests to make the STFT/ISTFT compatible with numpy/scipy or tensorflow

Further References

• Music Processing Book

When #8 is addressed the next step is to split the processing code.
Currently we have lot of duplicated code in demo/src/lib/umx.js and src/open-unmix.js.

We also have two npm packages: umx_js_demo (demo app) and open-unmix-js (the one that should live on npm)

Idea:

• Core functionality/processing in src/open-unmix.js. This should basically export two functions. One that loads a model given a url (not there yet, since its hardcoded) and another one that actually process buffers (modelProcess).
• I/O as well as web audio stuff should not be part of src/open-unmix.js.

That means src/open-unmix.js should work in node as well as browser. We can maybe re-add specific node function to load audio from cli, but I don't think this is high priority right now.

@deltonvaz @ClaraJacintho Do you like this?

One thing I don't know how to do: how can we add the open-unmix-js dev dependencies from umx_js_demo until the package is not registered on npm?

Currently the model parameters set globally when loading the lib. This makes changing the parameters when loading a new model a little bit cumbersome.

Therefore we should allow to load a model and also pass its parameters. The following ones are necessary:

// Fourier Params
const FRAME_LENGTH = config.fourierParams.frameLength
const HOP_LENGTH = config.fourierParams.hopLength
const FFT_SIZE = config.fourierParams.fftSize

// MODEL input dimensions
const N_FREQUENCIES = config.modelInput.N_FREQUENCIES
const N_FRAMES = config.modelInput.N_FRAMES
const N_CHANNELS = config.modelInput.N_CHANNELS
const N_BATCHES = config.modelInput.N_BATCHES

• the demo uses and older version of the player, I think we can, instead, incorporate it into the share platform code.

lets make the root folder the main folder for the npm package. The code for the vue-demo should be put in a subfolder like demo. The final (servable) demo should make available only in the gh-pages branch

spleeter_js implements a fully working example of a similar project that includes STFT and tensorflowjs model inference to do source separation.

The source code can be inspected in the web page.

First steps include:

• Evaluate the website
• Check which parts of the code can be re-used/copied
• Check which parts of the code could be better done in a framework
• Find existing frameworks for STFT/ISTFT and general signal processing

References

• https://github.com/mikolalysenko/stft
• ...

Further References

Open-Unmix

• Website
• Paper

Spleeter (Convolutional Neural Network)

• Repository
• Paper

write a python script that lives in a subfolder called "models"

• download the 2-stem model that can be downloaded here
• convert this to a tfjs model
• apply compression if suitable

do not forget to add (conda) environment files.