dcase-repo / desed_task

Domestic environment sound event detection task

DESED_task

Domestic environment sound event detection task.

---

DCASE Task 4

Baseline DCASE Task 4 recipes:

• DCASE 2021 Task 4
• DCASE 2022 Task 4
• DCASE 2023 Task 4
• DCASE 2024 Task 4

Updates are on the website and join us in the dedicated slack channel.

Installation Notes

You want to run a recipe or DCASE Task 4 baseline system

Go to ./recipes/YOUR_DESIRED_RECIPE and follow the instructions there in the README.md

In each recipe we provide a conda script which creates a suitable conda environment with all dependencies, including pytorch with GPU support in order to run the recipe. There are also instructions for data download and preparation.

You need only desed_task package for other reasons

Run python setup.py install to install the desed_task package

Your own recipes ?

If you want to share your recipe in this repo, do not hesitate to create a pull request. To be able to contribute/modify the code install desed_task via python setup.py develop.

Note

By default a pre-commit is installed via requirements.txt. The pre-commit hook checks for Black formatting on the whole repository. Black ensures that code style is consistent through the whole repository and recipes for better readability.

desed_task's Issues

Unidentified Symbol error when running train_sed.py

I used to have issues running generate_dcase_task4_2021.py, but fixed it using by doing this:
import desed
desed.download.filter_sins("PATH/TO/DCASE2018-task5-dev", destination_folder=os.path.join("PATH/TO/background", "sins"),
classes_kept=["other"],
rm_original_sins=True)

however, after running generate_dcase_task4_2021.py, I now get this error when trying to run train_sed.py:
Traceback (most recent call last):
File "train_sed.py", line 17, in
from desed_task.utils.encoder import ManyHotEncoder
File "/media/roestlab/Data3/User/LeonX/github/DESED_task/desed_task/utils/init.py", line 2, in
from .schedulers import ExponentialWarmup
File "/media/roestlab/Data3/User/LeonX/github/DESED_task/desed_task/utils/schedulers.py", line 1, in
from asteroid.engine.schedulers import *
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/asteroid/init.py", line 3, in
from .models import ConvTasNet, DCCRNet, DCUNet, DPRNNTasNet, DPTNet, LSTMTasNet, DeMask
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/asteroid/models/init.py", line 2, in
from .base_models import BaseModel
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/asteroid/models/base_models.py", line 4, in
from .. import separate
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/asteroid/separate.py", line 14, in
from .dsp.overlap_add import LambdaOverlapAdd
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/asteroid/dsp/init.py", line 2, in
from .overlap_add import LambdaOverlapAdd, DualPathProcessing
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/asteroid/dsp/overlap_add.py", line 3, in
from ..losses.pit_wrapper import PITReorder
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/asteroid/losses/init.py", line 14, in
from .stoi import NegSTOILoss as SingleSrcNegSTOI
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/asteroid/losses/stoi.py", line 5, in
from torch_stoi import NegSTOILoss as _NegSTOILoss
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/torch_stoi/init.py", line 1, in
from .stoi import NegSTOILoss
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/torch_stoi/stoi.py", line 5, in
import torchaudio
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/torchaudio/init.py", line 1, in
from . import extension
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/torchaudio/extension/init.py", line 5, in
_init_extension()
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/torchaudio/extension/extension.py", line 12, in _init_extension
_init_script_module(ext)
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/torchaudio/extension/extension.py", line 19, in _init_script_module
torch.classes.load_library(path)
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/torch/_classes.py", line 46, in load_library
torch.ops.load_library(path)
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/torch/_ops.py", line 104, in load_library
ctypes.CDLL(path)
File "/home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/ctypes/init.py", line 381, in init
self._handle = _dlopen(self._name, mode)
OSError: /home/roestlab/anaconda3/envs/dcase2021/lib/python3.8/site-packages/torchaudio/_torchaudio.so: undefined symbol: _ZNK3c104Type14isSubtypeOfExtESt10shared_ptrIS0_EPSo

I tried to delete all the symlinks and re-run generate_dcase_task4_2021.py but still get this error when trying to run train_sed.py .

Baseline performance

Hi, I downloaded the checkpoint from the given site.

And when I ran train_sed.py, the evaluation results were far less than the provided value.

DATALOADER:0 TEST RESULTS
{'hp_metric': 0.4607104032219199,
 'test/student/event_f1_macro': 0.36400517676011424,
 'test/student/intersection_f1_macro': 0.6944254152896272,
 'test/student/loss_strong': 0.13684727251529694,
 'test/student/psds_score_scenario1': 0.28276984860907106,
 'test/student/psds_score_scenario2': 0.4607104032219199,
 'test/teacher/event_f1_macro': 0.3735340054530239,
 'test/teacher/intersection_f1_macro': 0.7144140221660716,
 'test/teacher/loss_strong': 0.12656442821025848,
 'test/teacher/psds_score_scenario1': 0.2917064844285735,
 'test/teacher/psds_score_scenario2': 0.47517726567916774}

I am wondering if my test set is correct. Could you please tell me which folder shall I check?

Many thanks

undefined symbol: _ZNK5torch8autograd4Node4nameEv

OSError: /home/karna/miniconda3/envs/sedb/lib/python3.8/site-packages/torchaudio/lib/libtorchaudio.so: undefined symbol: _ZNK5torch8autograd4Node4nameEv

Getting this error on running train_sed.py for DCASE 2022 baseline.

Earlier was getting No module named 'desed_task.dataio'
then included
import sys
sys.path.append('../../')
to include desed_task.dataio path.

but now getting above mentioned error, not able to figure out how to solve, kindly help

Problem with Pytorch Lightning version

Newer Pytorch Lightning versions, which are installed automatically, don't support:

self.hparams = hparams

So, it must be used something like the following line:

self.hparams.update(hparams)

As indicated here. It will be nice to set in the requirements.txt the exact version to get all working (e.g. pytorch-lightning==1.2.10)

requests.exceptions.ProxyError: HTTPSConnectionPool(host='zenodo.org', port=443)

I'm sorry to bother you, but when I run the generate_dcase_task4_2022.py, I always comes this problem:

requests.exceptions.ProxyError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /record/4639776/files/audioset_metadata.tar.gz? /record/4639776/files/audioset_metadata.tar.gz?download=1 (Caused by ProxyError('Your proxy appears to only use to be HTTP. See: https://urllib3.readthedocs.io HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanwrong version number (_ssl.c:1108)'))))ced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)'))))

mixup not applied to weak labels

Hi,

if I'm not mistaken, mixup is applied to weakly labeled data in the following line (in file sed_trainer_pretrained.py):

features, embeddings, labels = self.apply_mixup(features, embeddings, labels, indx_strong, indx_weak)

However, the variable labels_weak is already assigned before and not modified after mixup. Is it intended to not mix the labels for weakly labeled clips, or am I overlooking something?

'synth_set' used twice

Hi, I was looking through the code for the DCASE'24 Task 4 baseline system and noticed the following lines in the file train_pretrained.py:

strong_full_set = torch.utils.data.ConcatDataset([strong_set, synth_set])
tot_train_data = [maestro_real_train, synth_set, strong_full_set, weak_set, unlabeled_set]
train_dataset = torch.utils.data.ConcatDataset(tot_train_data)

According to this, 'synth_set' is used twice. Is there a specific reason for this?

Problems on pretrained model

Two problems were encountered in training and testing with pre-trained models (AST).

1. Error occurs when reading the devtest.hdf5
   [Error message]
   
   [Error occurred point]
   

2. The weight of the Base CRNN model and Pretrained_model are different (weights file name: epoch=156-step=18525.ckpt)
   [Error message]
   

Download the data is difficult

Hi, I come from china, in our country download the data is difficult. Can you upload these data on Zenodo or other websit?

AttributeError: 'ConcatDatasetBatchSampler' object has no attribute 'batch_size'

KeyError: Caught KeyError in DataLoader worker process 0.

I keep getting this error when I try to run train_pretrained.py

I thought I might have problems on the dataset, but the error still pops up even after I redownload the dataset.

KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/user/.conda/envs/dcase2024/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/home/user/.conda/envs/dcase2024/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/user/.conda/envs/dcase2024/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/user/.conda/envs/dcase2024/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 335, in getitem
return self.datasets[dataset_idx][sample_idx]
File "/mnt/hdd_8TB/SED/DESED_task/recipes/dcase2024_task4_baseline/desed_task/dataio/datasets.py", line 216, in getitem
index = self.ex2emb_idx[name]
KeyError: 'metro_station_07-007300-008300'

Label missing in validation meta file

I found some audio clips in the validation set that do not have labels.

Can I safely delete them from the tsv files?

List of these files:

                            filename  onset  offset event_label
4051    YU0Cg_t_3TdI_30.000_40.000.wav    NaN     NaN         NaN
4059     Y70CtvdsGQf4_2.000_12.000.wav    NaN     NaN         NaN
4074    Y-jLujkP1aaw_30.000_40.000.wav    NaN     NaN         NaN
4093  YY09tBEoLETY_440.000_450.000.wav    NaN     NaN         NaN
4114    Y-YwPSwhe2jo_30.000_40.000.wav    NaN     NaN         NaN
4115    Y-RLILmQmubM_30.000_40.000.wav    NaN     NaN         NaN
4156    YjF73O9J4KEI_30.000_40.000.wav    NaN     NaN         NaN
4161  YuXDi40QfH9E_320.000_330.000.wav    NaN     NaN         NaN
4171     Y0IxyUtHw6Ec_0.000_10.000.wav    NaN     NaN         NaN
4187    Y-4pmCrSdMhg_30.000_40.000.wav    NaN     NaN         NaN
4188  YQb6N3HvbmgQ_580.000_590.000.wav    NaN     NaN         NaN
4222  Y09kqMqn5rWs_260.000_270.000.wav    NaN     NaN         NaN
4223    YzDULhAxsJfY_60.000_70.000.wav    NaN     NaN         NaN
4228    YR3_Gqgl-NYE_50.000_60.000.wav    NaN     NaN         NaN
4238    Y86owBlJa8f0_24.000_34.000.wav    NaN     NaN         NaN

I realized that these samples could testify the FPs for the models. But it seems that they would lead to some bugs in my code after I update my pandas package (not sure). Did anyone have the same issue?

UPDATE:
My solution:
in desed_task\dataio\datasets.py

line 141:
put labels_df = process_labels(labels_df, onset_s, offset_s) into the following if-else:

        if not len(labels_df):
            max_len_targets = self.encoder.n_frames
            strong = torch.zeros(max_len_targets, len(self.encoder.labels)).float()
        else:
            labels_df = process_labels(labels_df, onset_s, offset_s)
            strong = self.encoder.encode_strong_df(labels_df)
            strong = torch.from_numpy(strong).float()

Dont konw yet if it is the right thing to do, cuz my test for the provided baseline.ckpt is higher than the given value (student: 33.431%, 53.177%; teacher: 35.020%, 54.955%).

Modifications:

1. Use torchmetric.F1Score to replace the F1 function in pytorch-lightning;
2. Replace all df.append by pd.concat([df1, df2], ignore_index=True/False)

difficult to download audioset_strong

Hi! It's diffcult to download audioset_strong data from youtube_dl. Could you please upload these data on Zenodo or other websit? Thanks!

The difference of baseline performances between ver. 2021 and 2022.

Hi, I have a question about the baseline performances reported on the GitHub repositories.

On the README at the directory "DCASE-REPO/DESED_task/tree/master/recipes/dcase2021_task4_baseline", it reports the PSDS 1, PSDS 2, intersection-F1 score, and event-F1 score are 0.353, 0.553, 79.5%, and 42.1%, respectively.
However, those of the directory "DCASE-REPO/DESED_task/tree/master/recipes/dcase2022_task4_baseline" are reported 0.351, 0.531, 64.1% and 41.4%.
Others have a few differences, but the inter-F1 scores have a big difference.
Indeed, my co-worker had a similar score with the intersection-F1 score of the dcase2021, but I got similar scores with the dcase2022 in several random seeds on the same dataset.

Which one should I reference, the report of the dcase2021 or that of the dcase2022?

Problems on "multiprocessing"

problem when I run the "dcase2022_task4_baseline/train_pretrained.py"
Training: 0it [00:00, ?it/s]CODECARBON : No CPU tracking mode found. Falling back on CPU constant mode.
CODECARBON : Failed to match CPU TDP constant. Falling back on a global constant.
Epoch 0: 0%| | 0/229 [00:00<?, ?it/s] Traceback (most recent call last):
File "E:/DESED_task/recipes/dcase2022_task4_baseline/train_pretrained.py", line 436, in
single_run(
File "E:/DESED_task/recipes/dcase2022_task4_baseline/train_pretrained.py", line 352, in single_run
trainer.fit(desed_training)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 735, in fit
self._call_and_handle_interrupt(
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 682, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 770, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1193, in _run
self._dispatch()
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1272, in _dispatch
self.training_type_plugin.start_training(self)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\plugins\training_type\training_type_plugin.py", line 202, in start_training
self._results = trainer.run_stage()
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1282, in run_stage
return self._run_train()
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1312, in _run_train
self.fit_loop.run()
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\loops\base.py", line 145, in run
self.advance(*args, **kwargs)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 234, in advance
self.epoch_loop.run(data_fetcher)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\loops\base.py", line 140, in run
self.on_run_start(*args, **kwargs)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\loops\epoch\training_epoch_loop.py", line 141, in on_run_start
self._dataloader_iter = _update_dataloader_iter(data_fetcher, self.batch_idx + 1)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\loops\utilities.py", line 121, in _update_dataloader_iter
dataloader_iter = enumerate(data_fetcher, batch_idx)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 198, in iter
self._apply_patch()
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 133, in _apply_patch
apply_to_collections(self.loaders, self.loader_iters, (Iterator, DataLoader), _apply_patch_fn)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\utilities\fetching.py", line 181, in loader_iters
loader_iters = self.dataloader_iter.loader_iters
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 523, in loader_iters
self._loader_iters = self.create_loader_iters(self.loaders)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\trainer\supporters.py", line 563, in create_loader_iters
return apply_to_collection(loaders, Iterable, iter, wrong_dtype=(Sequence, Mapping))
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\pytorch_lightning\utilities\apply_func.py", line 92, in apply_to_collection
return function(data, *args, **kwargs)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\torch\utils\data\dataloader.py", line 367, in iter
return self._get_iterator()
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\torch\utils\data\dataloader.py", line 313, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\site-packages\torch\utils\data\dataloader.py", line 926, in init
w.start()
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "C:\Users\Payne\anaconda3\envs\dcase2022_re\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'single_run..ASTFeatsExtraction'

baseline model checkpoints do not seem accurate on simple sounds.

Hello, I was trying to use the model checkpoints to do inference on a couple straight-forward files, but the model always says class 8, which I assume is "speech". My code is below. I tried with sounds of an alarm https://www.youtube.com/watch?v=0ng703aLhXs&t=19s and another youtube video where someone is frying. Is my code wrong somehow?

import librosa
from torchsummary import summary
from torchaudio.transforms import AmplitudeToDB, MelSpectrogram

from desed_task.nnet.CRNN import CRNN
from desed_task.utils.scaler import TorchScaler

model = CRNN(
    kernel_size = [3, 3, 3, 3, 3, 3, 3],
    padding = [1, 1, 1, 1, 1, 1, 1],
    stride = [1, 1, 1, 1, 1, 1, 1],
    nb_filters = [ 16, 32, 64, 128, 128, 128, 128 ],
    pooling = [ [ 2, 2 ], [ 2, 2 ], [ 1, 2 ], [ 1, 2 ], [ 1, 2 ], [ 1, 2 ], [ 1, 2 ] ]
)
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
checkpoint = torch.load('epoch=156-step=18525.ckpt', map_location=torch.device('cpu'))
model.load_state_dict(checkpoint['sed_student'])
model.training=False

X, sr = librosa.load('alarm.wav',sr=22050)
X = np.concatenate((np.zeros(13000),X))
mel_spec = MelSpectrogram(
            sample_rate=sr,
            n_fft=2048,
            win_length=2048,
            hop_length=365,
            f_min=0,
            f_max=8000,
            n_mels=128,
            window_fn=torch.hamming_window,
            wkwargs={"periodic": False},
            power=1,
        )
X = mel_spec(torch.tensor(np.array([X]).astype(np.float32)))
amp_to_db = AmplitudeToDB(stype="amplitude")
amp_to_db.amin = 1e-5  # amin= 1e-5 as in librosa
X = amp_to_db(X).clamp(min=-50, max=80)
scaler = TorchScaler(
    "instance",
    checkpoint['hyper_parameters']["scaler"]["normtype"],
    checkpoint['hyper_parameters']["scaler"]["dims"],
)
X = scaler.forward(X)

out = model(X)
out = out[1].cpu().detach().numpy()
print(np.argmax(out))

Test step takes long time

Hi,

the testing step in the 2022 recipe lasts much longer than the validation step, even though it uses the same amount of data and the on_test_epoch_end function has not been called yet. Is there something I don't see or what might be the reason for that?

Best regards

Errors when testing with psds_eval>=0.4.0

It worked well when training with psds_eval>=0.4.0,But an error occurs in the final test.

When I downgraded PSD_eval to 0.3.0, it worked fine. However, the published baseline cannot be reproduced, and psDS1 is 0.3153 and PSDS2 is 0.5101 using PSDS_eval ==0.3.0

Questions about Apple Silicon （M1 Pro） not being able to run the project

Hello, I am a beginner, but I want to participate in this project, I have a problem:
I don't know whether this project can be run under M1 Pro?
My Environment configuration；

• conda 4.14.0

• pip -> 22.1.2 from /opt/homebrew/Caskroom/miniforge/base/envs/dcase2022/lib/python3.8/site-packages/pip (python 3.8)

• Python 3.8.5

In order to run on M1 arm64 pytorch, I installed the adapter arm64 tensorflow
link: https://developer.apple.com/metal/tensorflow-plugin/

Test the following
To prove that PyTorch is correct I executed it from https://github.com/pytorch/examples/blob/master/mnist/main.py

python main.py

The output is Test set: Average loss: 0.0287, Accuracy: 9908/10000 (99%)

There is nothing wrong with my Pytorch, right ？

I executed recipes/dcase2022_task4_baseline/conda_create_environment.sh ，Dependencies are installed.

But I execute train_sed.py console output

  /opt/homebrew/Caskroom/miniforge/base/envs/dcase2022/bin/python /Users/myUser/Desktop/workspace/python/DESED_task/recipes/dcase2022_task4_baseline/train_sed.py
  
  Process finished with exit code 138 (interrupted by signal 10: SIGBUS)

Is there anyone who can help me, I want to know what the problem is that I can't run this project？

_pickle.UnpicklingError: invalid load key, '\xef'.

An error occurred while running recipes/dcase2024_task4_baseline/extract_embeddings. py

Skipping download as file in ./pretrained_models/BEATS_iter3_plus_AS2M.pt exists already
Traceback (most recent call last):
File "/home/dl/data/DESED_task-master/recipes/dcase2024_task4_baseline/extract_embeddings.py", line 189, in
pretrained = BEATsModel(cfg_path="./pretrained_models/BEATS_iter3_plus_AS2M.pt")
File "/home/dl/data/DESED_task-master/recipes/dcase2024_task4_baseline/local/beats/BEATs.py", line 194, in init
checkpoint = torch.load(cfg_path)
File "/home/dl/anaconda3/envs/dcase2024/lib/python3.8/site-packages/torch/serialization.py", line 1040, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/dl/anaconda3/envs/dcase2024/lib/python3.8/site-packages/torch/serialization.py", line 1262, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\xef'.

Intersection F1 calculation code

Since PSDS Eval package has been removed from github and so the support for it, is there a plan to have a separate standalone code for evaluation of this metric in the repo without having to import from a somewhat obscure psds_eval package that has been removed from github?

I was getting NaNs in my "per class F1 score" so i had to go through the psds_eval package only to discover that its due to the line:

        num_gts = per_class_tp / tp_ratios

where it calculates num_gts in this really bizarre way assuming tp_ratio never being zero(!) and hence yields the false negatives and F1 of all classes with zero TP, to become nan.

def compute_macro_f_score(self, detections, beta=1.):
        """Computes the macro F_score for the given detection table

        The DTC/GTC/CTTC criteria presented in the ICASSP paper (link above)
        are exploited to compute the confusion matrix. From the latter, class
        dependent F_score metrics are computed. These are further averaged to
        compute the macro F_score.

        It is important to notice that a cross-trigger is also counted as
        false positive.

        Args:
            detections (pandas.DataFrame): A table of system detections
                that has the following columns:
                "filename", "onset", "offset", "event_label".
            beta: coefficient used to put more (beta > 1) or less (beta < 1)
                emphasis on false negatives.

        Returns:
            A tuple with average F_score and dictionary with per-class F_score

        Raises:
            PSDSEvalError: if class instance doesn't have ground truth table
        """
        if self.ground_truth is None:
            raise PSDSEvalError("Ground Truth must be provided before "
                                "adding the first operating point")

        det_t = self._init_det_table(detections)
        counts, tp_ratios, _, _ = self._evaluate_detections(det_t)

        per_class_tp = np.diag(counts)[:-1]
        num_gts = per_class_tp / tp_ratios
        per_class_fp = counts[:-1, -1]
        per_class_fn = num_gts - per_class_tp
        f_per_class = self.compute_f_score(per_class_tp, per_class_fp,
                                           per_class_fn, beta)

        # remove the injected world label
        class_names_no_world = sorted(set(self.class_names
                                          ).difference([WORLD]))
        f_dict = {c: f for c, f in zip(class_names_no_world, f_per_class)}
        f_avg = np.nanmean(f_per_class)

        return f_avg, f_dict 

This behaviour by the way could easily lead to a significant overestimation of macro intersection F1 using this code, because if the model's output for a rare class yields zero TPs, the macro F1 in this package ignores it and calculates average across the rest of the classes.

So I think it would be helpful to have a more transparent and clean standalone code for intersection based F1 in the repo.

problem when downloading the 'audioset_data'

When I run 'generate_dcase_task4_2021.py --basedir='../../data'', it paused as shown as follows:

How can I make the downloading successfully?

What Librosa version is used in Baseline??

Hii,
I have tried with different Librosa versions:

• With Versions above 0.8.1 which include (0.10.1, 0.10.0, 0.9.0), I am getting following error

tmp = librosa.resample(tmp, orig_fs, target_fs)
TypeError: resample() takes 1 positional argument but 3 were given

• With version 0.8.1

AttributeError: module 'numpy' has no attribute 'complex'. 'np.complexwas a deprecated alias for the builtincomplex. To avoid this error in existing code, use complexby itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, usenp.complex128` here.

May I know which version should be used to avoid such type of issues

Errors when training with multiple gpus

It worked well when training with single gpu.
But an error about batchsampler occurs when using multiple gpus

Got error while running the training script for SED

Hello, I got the following error while trying to train the model by running 'train_sed.py'.

Epoch 0: 0%| | 0/229 [00:00<?, ?it/s]/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torch/functional.py:515: UserWarning: stft will require the return_complex parameter be explicitly specified in a future PyTorch release. Use return_complex=False to preserve the current behavior or return_complex=True to return a complex output. (Triggered internally at /opt/conda/conda-bld/pytorch_1603729009598/work/aten/src/ATen/native/SpectralOps.cpp:653.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore
/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torch/functional.py:515: UserWarning: The function torch.rfft is deprecated and will be removed in a future PyTorch release. Use the new torch.fft module functions, instead, by importing torch.fft and calling torch.fft.fft or torch.fft.rfft. (Triggered internally at /opt/conda/conda-bld/pytorch_1603729009598/work/aten/src/ATen/native/SpectralOps.cpp:590.)
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore
Epoch 0: 97%|▉| 222/229 [01:24<00:02, 2.62it/s, loss=1.21, v_num=3, train/step=117.0, train/student/tot_self_loss=3.47e-5, train/student/tot_supervised=0Traceback (most recent call last):██████████████████████████████████████████████████████████████████████████████████▍ | 104/111 [00:18<00:01, 5.92it/s]
File "train_sed.py", line 295, in
single_run(
File "train_sed.py", line 244, in single_run
trainer.fit(desed_training)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
self._run(model)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
self.dispatch()
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
self.accelerator.start_training(self)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
self.training_type_plugin.start_training(trainer)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
self._results = trainer.run_stage()
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
return self.run_train()
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 869, in run_train
self.train_loop.run_training_epoch()
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 576, in run_training_epoch
self.trainer.run_evaluation(on_epoch=True)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in run_evaluation
output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 174, in evaluation_step
output = self.trainer.accelerator.validation_step(args)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 226, in validation_step
return self.training_type_plugin.validation_step(*args)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in validation_step
return self.lightning_module.validation_step(*args, **kwargs)
File "/home/SharedData/Suhasinee/DESED_task-master/recipes/dcase2021_task4_baseline/local/sed_trainer.py", line 371, in validation_step
self.get_weak_student_f1_seg_macro(
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torchmetrics/metric.py", line 168, in forward
self.update(*args, **kwargs)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torchmetrics/metric.py", line 216, in wrapped_func
return update(*args, **kwargs)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torchmetrics/classification/stat_scores.py", line 207, in update
tp, fp, tn, fn = _stat_scores_update(
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torchmetrics/functional/classification/stat_scores.py", line 90, in _stat_scores_update
preds, target, _ = _input_format_classification(
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torchmetrics/utilities/checks.py", line 418, in _input_format_classification
case = _check_classification_inputs(
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torchmetrics/utilities/checks.py", line 268, in _check_classification_inputs
_basic_input_validation(preds, target, threshold, multiclass)
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/torchmetrics/utilities/checks.py", line 36, in _basic_input_validation
raise ValueError("The target has to be an integer tensor.")
ValueError: The target has to be an integer tensor.
Exception ignored in: <function tqdm.del at 0x7f85288df160>
Traceback (most recent call last):
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/tqdm/std.py", line 1145, in del
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/tqdm/std.py", line 1299, in close
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/tqdm/std.py", line 1492, in display
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/tqdm/std.py", line 1148, in str
File "/nfs4/suhasinee/miniconda3/envs/dcase2021/lib/python3.8/site-packages/tqdm/std.py", line 1450, in format_dict
TypeError: cannot unpack non-iterable NoneType object

Please suggest on how to fix this issue.

Is there any way to check if there's any file still missing in the dataset

Recently I've noticed that the number of files I downloaded did not match the number in the description of DESED dataset
most of them are weakly labeled training data, and maybe some label tsv files
(I've already downloaded the missing files provided by Ms. Ronchini)
I'm wondering if there's a list of the file names or anything else that can help me check if I'm still missing anything in my folder
Thanks

error in sed_scores_eval

import sed_scores_eval leads to a SyntaxError.......?

Using embeddings for eval mode

When using the evaluation mode, by default the baseline with embeddings does not have the "extracted embeddings"
Something like this need to be done I guess:

eval_embeddings = None if config["pretrained"]["e2e"] else os.path.join(config["pretrained"]["extracted_embeddings_dir"],
                                                                                config["pretrained"]["model"], "eval.hdf5")

devtest_dataset = UnlabeledSet(
    config["data"]["eval_folder"],
    encoder,
    pad_to=None,
    embeddings_hdf5_file=eval_embeddings,
    return_filename=True, feats_pipeline=feature_extraction
)

Tell me if you want a pull request (otherwise just assign it to a further pull request if you put it inside an update)

ImportError: cannot import name 'FastChildWatcher' from 'asyncio'

Hello,

I'm trying to execute the "generate_dcase_task4_2023.py".

1. I've created a Conda environment by running the script "conda_create_environment.sh" line by line.
2. I've also installed the desed task module ("pip install desed")
3. When I run the script with the --only_strong option, I receive the following error:

(dcase2023) C:\Users\adiel\Documents\Research\DESED_task\recipes\dcase2023_task4_baseline>python generate_dcase_task4_2023.py --only_strong Traceback (most recent call last): File "generate_dcase_task4_2023.py", line 12, in <module> import desed File "C:\ProgramFilesAdiel\Anaconda\envs\dcase2023\lib\site-packages\desed\__init__.py", line 1, in <module> from .download import ( File "C:\ProgramFilesAdiel\Anaconda\envs\dcase2023\lib\site-packages\desed\download.py", line 1, in <module> from asyncio import FastChildWatcher ImportError: cannot import name 'FastChildWatcher' from 'asyncio' (C:\ProgramFilesAdiel\Anaconda\envs\dcase2023\lib\asyncio\__init__.py)

I'm using Anaconda, Pycharm, and Windows 10 Home.

I'm attaching the environment installed packages (pip freeze)

Regards,

Adiel
pip_freeze.txt

question about the label mapping

Hi there, I see that we have a label mapping dict here.
But in the groundtruth of MAESTRO, I think we only have "people talking", not "people_talking".
It seems that there is an additional underscore for the event "people talking" in this mapping dict.

the lightning version should be fixed to 1.9.x

The lightning released the new 2.0 version last week. There are some APIs have been removed including the strategy 'dp' and so on. If someone updates to the new version of lightning may meet the error ValueError: You selected an invalid strategy name: strategy='dp'. There are also some NoImplement errors. In a word, we need to fix the version of lightning to 1.9.x when using the baseline. Maybe we could notice others in requirement.

Confused about the development dataset

Sorry to bother you, but there is something really confusing us. On the task description web page, we notice that the strong labelled part of the development set only contains 10k synthetic data. However, in this repository, we find that there is some other strong labeled data (namely strong_real) that can be used to train the baseline model. So, could participants use these strong_real data to train their own model?

Files don't get saved when downloading

Hello,

using the dcase2022_task4_baseline/generate_dcase_task4_2022.py script downloading the audio data using the desed package is showing a progress bar with reasonable speed, but the files just don't get saved in the filesystem. I tried both using Windows 10 (and adjusting the filepaths) or opening the repo directory within a docker unix container and running it there.

However, when I use the original DESED repo and apply the same functions, the files get saved. What could be the reason for this?

Regards

Run the "generate_dcase_task4_2021.py"

Hello,

We get some errors as below when running the generate_dcase_task4_2021.py.
I don't have any idea about this problem.
Would you tell me what to do? Thank you!

Traceback (most recent call last):
File "generate_dcase_task4_2021.py", line 794, in
generate_soundscapes(
File "generate_dcase_task4_2021.py", line 571, in generate_soundscapes
events = instantiate_soundscape(
File "generate_dcase_task4_2021.py", line 422, in instantiate_soundscape
sc = add_bg(sc)
File "generate_dcase_task4_2021.py", line 343, in add_bg
bg_instance = sc._instantiate()["annotations"][0]["data"][0][2]
File "/home/bee/anaconda3/envs/dcase21_task4/lib/python3.8/site-packages/scaper/core.py", line 1712, in _instantiate
value = self._instantiate_event(
File "/home/bee/anaconda3/envs/dcase21_task4/lib/python3.8/site-packages/scaper/core.py", line 1426, in _instantiate_event
label = _get_value_from_dist(label_tuple, self.random_state)
File "/home/bee/anaconda3/envs/dcase21_task4/lib/python3.8/site-packages/scaper/core.py", line 424, in _get_value_from_dist
return SUPPORTED_DIST[dist_tuple[0]](*dist_tuple[1:], random_state=random_state)
File "/home/bee/anaconda3/envs/dcase21_task4/lib/python3.8/site-packages/scaper/util.py", line 270, in _sample_choose
index = random_state.randint(len(new_list_of_options))
File "mtrand.pyx", line 746, in numpy.random.mtrand.RandomState.randint
File "_bounded_integers.pyx", line 1254, in numpy.random._bounded_integers._rand_int64
ValueError: low >= high

AttributeError: 'Energy' object has no attribute 'kwh'

DESED_task/recipes/dcase2023_task4_baseline/local/sed_trainer.py

Line 895 in a117c36

training_kwh = self.tracker_train._total_energy.kwh

Here is a little typo. It should be kWh not kwh.
It also exist in the func on_test_end.

How to modified my own model to fit a 3-class problem?

It gives me an error like:

ValueError: Using a target size (torch.Size([24, 10, 156])) that is different to the input size (torch.Size([24, 3, 156])) is deprecated. Please ensure they have the same size.

I already change the confs/default.yaml's nclass to 3.But the target size still seems to be 10, What else I should do?

Probem on psds_score_scenario2

Dear organizer:
When I using weak label to compute psds score scenario2, I got some problem, I don't know how to fix it. Thanks for your time and kindly help~
(1) Transfer learning from PANNs to train an audio tagging system，using weak label （torch.ones_like(ground_truth strong_labels) * weak_preds） as the predict result to compute psds_score_scenario2:
DATALOADER:0 TEST RESULTS
{'hp_metric': 0.7078940868377686,
'test/student/event_f1_macro': 0.21531902253627777,
'test/student/intersection_f1_macro': 0.8512769937515259,
'test/student/psds_score_scenario1': 0.04927980527281761,
'test/student/psds_score_scenario2': 0.7078940868377686,
'test/teacher/event_f1_macro': 0.21711485087871552,
'test/teacher/intersection_f1_macro': 0.8789225220680237,
'test/teacher/psds_score_scenario1': 0.04864690080285072,
'test/teacher/psds_score_scenario2': 0.7067073583602905}

(2) Using ground truth strong label in test set to get ground truth weak label, and compute the psds_score_scenario2 using ground truth weak label（torch.ones_like(ground_truth strong_labels) * ground_truth_weak_label）.
DATALOADER:0 TEST RESULTS
{'hp_metric': 0.7472647428512573,
'test/student/event_f1_macro': 0.24901337921619415,
'test/student/intersection_f1_macro': 0.980992317199707,
'test/student/psds_score_scenario1': 0.04510604217648506,
'test/student/psds_score_scenario2': 0.7472647428512573,
'test/teacher/event_f1_macro': 0.24901337921619415,
'test/teacher/intersection_f1_macro': 0.980992317199707,
'test/teacher/psds_score_scenario1': 0.04510604217648506,
'test/teacher/psds_score_scenario2': 0.7472647428512573}

Validation file missing

I already done downloading the datasets and the missing files
But when I tried to start the training process with python train_sed.py command
I got this error

And these are all my files in the directory now

I'm not sure if I should change anything to match the file name in the code or I need to download the missing file from somewhere, since I already ran the python generate_dcase_task4_2022.py --basedir="../../data" command multiple times and none of the items in the missing file list is a .tsv file

Got erros when getting dataset from generate_dcase_tast4_2021 and during training

Hi, I got two erros when running the file [generate_dcase_task4_2021.py] and [train_sed.py]

1. There's an error "shut.ReadError: ~/DESED_task-master/recipes/dcase2021_task4baseline/tmp_88czbqta/tmpct_t19xo..zip is not a zip file" when run the generate_dcase_task4_2021.py

2. There's an error like

Traceback (most recent call last):
File "generate_dcase_task4_2021.py", line 794, in
generate_soundscapes(
File "generate_dcase_task4_2021.py", line 571, in generate_soundscapes
events = instantiate_soundscape(
File "generate_dcase_task4_2021.py", line 422, in instantiate_soundscape
sc = add_bg(sc)
File "generate_dcase_task4_2021.py", line 343, in add_bg
bg_instance = sc._instantiate()["annotations"][0]["data"][0][2]
File "/home/gook/miniconda3/lib/python3.8/site-packages/scaper/core.py", line 1712, in _instantiate
value = self._instantiate_event(
File "/home/gook/miniconda3/lib/python3.8/site-packages/scaper/core.py", line 1426, in _instantiate_event
label = _get_value_from_dist(label_tuple, self.random_state)
File "/home/gook/miniconda3/lib/python3.8/site-packages/scaper/core.py", line 424, in _get_value_from_dist
return SUPPORTED_DIST[dist_tuple[0]](*dist_tuple[1:], random_state=random_state)
File "/home/gook/miniconda3/lib/python3.8/site-packages/scaper/util.py", line 270, in _sample_choose
index = random_state.randint(len(new_list_of_options))
File "mtrand.pyx", line 746, in numpy.random.mtrand.RandomState.randint
File "_bounded_integers.pyx", line 1254, in numpy.random._bounded_integers._rand_int64
ValueError: low >= high

when run the generate_dcase_task4_2021.py again (got error 1 and run again)

3. when run the train_sed.py in this state, there is an error message "No File or Directory /DESED_task-master/data/raw_datasets/desed_real/metadata/validation/duration.tsv"

Is there any way to fix those bugs?

Where is the downloaded dataset?

I have completely run the script: python generated_case_task4_2024. py, but in the directory DESED_task master/data/dcase/dataset/audio/train/, the strong_label_real, unlabel_in_domain, and weak are all empty.

When I run the script: Python train_pretrained.py, the result is as follows:
Resampling syntax folder to 16 kHz
Resampling synth_val_folder to 16 kHz
Resampling real_masestro_train_folder to 16 kHz
Resampling real_masestro_val_folder to 16 kHz
Resampling strong_folder to 16 kHz
0it [00:00,? It/s]
Resampling weak_folder to 16 kHz
0it [00:00,? It/s]
Resampling unlabeled_folder to 16 kHz
0it [00:00,? It/s]
Resampling test_folder to 16 kHz
0it [00:00,? It/s]
Global seed set to 42
Traceback (most recent call last):
File "/home/dl/data/DESED_task master/recipes/dcase2024_task4_baseline/train_pretrained. py", line 621, in
Single_run（
File "/home/dl/data/DESED_task master/recipes/dcase2024_task4_baseline/train_pretrained. py", line 413, in single_run
Samplers=[torch. utils. data. RandomSampler (x) for x in tottrain_data]
File "/home/dl/data/DESED_task master/recipes/dcase2024_task4_baseline/train_pretrained. py", line 413, in
Samplers=[torch. utils. data. RandomSampler (x) for x in tottrain_data]
File "/home/dl/anaconda3/envs/dcase2024/lib/python3.8/site packages/torch/utils/data/sampler. py", line 143, in __ init__
Raise Value Error (f "num samples should be a positive integer value, but got num samples={self. num samples}")
Value Error: num samples should be a positive integer value, but got num samples=0

I don't know what to do, do I still need to add: only strong only real only synth only maestro after Python generated_case_task4_2024. py and run it again? Or should I do something else to solve this problem?

error: Couldn't find a setup script in /tmp/easy_install-zeojqoxu/scipy-1.11.2.tar.gz

getting this error on running setup command
python3 setup.py install

pytorch_lightning version

Hi,

you have forgotten to list the lightning version in requirements.txt for the baseline. With the aweful instability of pytorch_lightning's the code wont run and It gives so many errors on the version I use on (sampler, dataset concat, ddp,...)

Running train_sed.py

When running this for the first time I have come across this error. All dependencies have been installed successfully via the shell file provided.

Error torchmetrics in recipes dcase_2022

When launching the baseline at test, I get this error:

loaded model: epoch=156-step=18525.ckpt?download=1
at epoch: 157
Global seed set to 42
Traceback (most recent call last):
  File "train_sed.py", line 377, in <module>
    single_run(
  File "train_sed.py", line 228, in single_run
    desed_training = SEDTask4(
  File "/projects/pul51/shared/calcul/users/nturpault/DESED_task/recipes/dcase2022_task4_baseline/local/sed_trainer.py", line 122, in __init__
    self.get_weak_teacher_f1_seg_macro = torchmetrics.classification.f_beta.F1(
AttributeError: module 'torchmetrics.classification.f_beta' has no attribute 'F1'

I have run the conda_create_environment from the recipe, am I the only one to get this ?

Softmax in attention block

Just realized something in the base model:
https://github.com/DCASE-REPO/DESED_task/blob/master/desed_task/nnet/CRNN.py

Could you please elaborate why in the attention block you are applying softmax across the classes dimension? I believe the softmax here should be applied across the time axis.

 self.softmax = nn.Softmax(dim=-1)

if self.attention:
            sof = self.dense_softmax(x)  # [bs, frames, nclass]
            if not pad_mask is None:
                sof = sof.masked_fill(pad_mask.transpose(1, 2), -1e30)  # mask attention
            sof = self.softmax(sof)
            sof = torch.clamp(sof, min=1e-7, max=1)
            weak = (strong * sof).sum(1) / sof.sum(1)  # [bs, nclass]