hearbenchmark / hear-preprocess Goto Github PK
View Code? Open in Web Editor NEWDataset preprocessing code for the HEAR 2021 NeurIPS competition
License: Apache License 2.0
Dataset preprocessing code for the HEAR 2021 NeurIPS competition
License: Apache License 2.0
If the file exists, it should be an error
Python soundfile cannot read mp3s but this is one of the formats that is searched for by audio_dir_stats_wav: https://github.com/neuralaudio/hear-eval-kit/blob/531adcc778582300fcdf7f4a19507cb16b8d31aa/heareval/tasks/util/audio.py#L93
Update the formats that are searched for or use something like audioread in the case of an mp3. Perhaps include the soundfile info function in a try block and load in with audioread and get the stats that way if it fails.
We need to change the behavior of the preprocessing pipeline to accommodate for tasks that require multiple audio files for each example of training data.
Currently, the metadata prepared in the ExtractMetadata
is such that each row corresponds to one training example and can have only one audio file.
Changing from relpath
to relpaths
in the metadata to accommodate more than one audiofile with each example will help us achieve the multi-audio-label training example scenario.
Possible changes -
split key
has to be defined for each example. i.e. for each relpaths
str(relpaths)
because that is unique at an example level.relpaths
(subsampled on str(relpath)
in previous step) into the corresponding task directory.split.json
has the following format -{
"filename: str": "labels: (Union[List[Any], str]"
}
To have multiple audio files for each example, the format of this file has to be changed to
[
{
"relpaths": List[str],
"labels": List[Any]
}
]
Few open questions after this -
nxm
(assuming num(relpaths) == m) and label --> nx1
. This way we will have a correspondance between each set of embeddings(m audio embeddings for one example in this case) and the corresponding label.set of embeddings for the audio in
relpathsand the
label or labels(for multi-label)`, we can use the same downstream setup(like the data loader)Also add soxi -T total to each stats file?
And audio_samplerate_count
should be audio_file_count
.
Lastly we need audio_duration
which is total number of samples / SR
Lastly, we want label counts + percentages.
This can be broken into smaller issues
Metadata is the labels, or let's call them labels everywhere and drop the term "metadata".
Using metadata for both task config and labels is confusing.
There are some messages being logged within runner.py using the luigi interface logger that are not showing up. I believe this is b/c the logger can only be called from within the pipeline (i.e in a function that runs within the pipeline). Need to check on that. Either way we are trying to log a couple things that are not showing up.
Throughout the preprocessing pipeline we have the concept of metadata for each task -- this is really just the labels and we should refactor the code to reflect this. i.e., rename 'metadata' to 'labels' throughout the pipeline.
speech commands could be simpler and safer if we just downloaded the tensorflow dataset where they run the original generation, instead of our ported version GenerateTrainDataset.
nsynth too? Do we save anything there?
There are also probably good tf datasets we could cherry pick. A simple tf dataset task could do wonders.
When task subsampling is None, don't subsample that split.
Total number of hours in the dataset should be in the final tar name, unless None
secrettasks repo (NOT hear-eval which is public) should run pytest and travis on secret tasks using heareval
The command listed in the readme doesn't correctly add the secret task submodule. I had to run this to get it to work properly: git submodule add [email protected]:neuralaudio/hear2021-secret-tasks hearpreprocess/secrettasks
https://stackoverflow.com/questions/3336995/git-will-not-init-sync-update-new-submodules
"This is different thing but it would be great to update all the print statements to use the luigi logger. logger.info("")"
For event embeddings, it's fine.
For scene embeddings, it's potentially a problem if those relpaths don't even appear in the metadata, because we will believe that this file doesn't exist! Do we have at least an assertion to check if that's the case?
Luigi should create README and LICENSE, as described in the tasks/README.md
Is ffmpeg padding?
Maybe consider longer datasets for fast embeddings?
ffmpeg>=4.2
installed (possibly from conda-forge).soxr
support, which might require packageCan we check these programmatically?
See also the discussion here: hearbenchmark/hear-eval-kit#156
Ending up with a weird number of samples in the train/valid split:
This is caused by the background_noise subsampling in the tasks/sampler.py. In speech commands all the background noise samples (which are labelled as silence) are delivered as longer audio samples that are expected to be sliced up into smaller chunks. When we are subsampling this dataset only one background noise sample is being included (running_tap.wav), and that happens to be in the validation set. As a result we are ending up with a validation set that is almost exclusively silence samples.
@jorshi writes:
I think you can do logger = logging.getLogger('name')
that will either retrieve or create a new logger
Note that Loggers should NEVER be instantiated directly, but always through the module-level function logging.getLogger(name). Multiple calls to getLogger() with the same name will always return a reference to the same Logger object.
le on how to setup a logger
https://docs.python.org/3/howto/logging-cookbook.html
import logging
logger = logging.getLogger('hear-logger')
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler("hear-preprocess.log")
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info("message")
Now that we have assert on symlinks not existing, this causes errors when trying to rerun a pipeline that was stopped in the middle of a task that creates symlinks.
Let's delete partially complete task workdirs when restarting a task, or atleast remove existing symlinks at the beginning of tasks that create them?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.