non-deterministic results when decoding with noises about av_hubert

[chevalierNoir]

Hi Zhengyang,

Merry Christmas. Yes, we also observed that there is some randomness in the result per run. The WER averaged across noise SNR/types is usually more stable. Also as we tune the decoding hyperparameters (e.g., beam size) on the valid set, the test WER is generally stable. Another way to reduce the randomness is to have multiple runs and report the mean WER.

[joyolee]

Hi Zhengyang,

Merry Christmas. Yes, we also observed that there is some randomness in the result per run. The WER averaged across noise SNR/types is usually more stable. Also as we tune the decoding hyperparameters (e.g., beam size) on the valid set, the test WER is generally stable. Another way to reduce the randomness is to have multiple runs and report the mean WER.

Hi, thanks a lot! Averaging the WER of multiple runs is also the method that we use. The average WER is relatively stable.

[Rongjiehuang]

Hi, could you please tell me what the noise directory looks like? After data preparation, we have all, babble, music and noise directories under musan/tsv/, while I could not find the speech directory mentioned in this work.

[chevalierNoir]

Hi, could you please tell me what the noise directory looks like? After data preparation, we have all, babble, music and noise directories under musan/tsv/, while I could not find the speech directory mentioned in this work.

Please find how to set up speech directory here.

[nastia-lado]

Hi, @joyolee!

If I understood correctly, you used AVSR model for decoding in noisy conditions.
Did you use colab by any chance?
I'm trying to modify the colab example to work with AVSR model (the example is only for VSR), but the colab crashes for unknown reasons. I'm trying to understand whether it's only my problem.