Difficulties pre-training with filter bank features about s3prl

[apoorv2904]

A few more details:
-- I had extracted features using Kaldi and set them up using the instructions provided.
-- I changed gelu to torch.nn.functional.gelu
-- using torch.nn.LayerNorm as TransformerLayerNorm

Since I could get the mel-160 features to work with the above changes, I don't think this should be the reason why filterbank features are not working.

--Apoorv

[andi611]

Hi,

The value of loss depends on the feature, hence loss computed from fbank is different then loss computed from mel-160.

The reason why you did bad on the Montreal-phone set is the following:
The fbank features are pre-processed with a stride of 10 (a.k.a a fbank vector for every 10 ms, a default of Kaldi), which does not match the phone alignment labels of the Montreal-phone set.

On the other hand, the mel-160 feature matches the duration of the Montreal-phone set, that's why you obtained reasonable results when pretrain using mel-160 features.

You should either evaluate with the cpc phone task (suggested approach), or change the parameters in preprocess_alignment.py to generate a phone label for every 10 ms (a more complicated approach).

I have verified that pre-training with fbank and evaluating with the cpc task can yield reasonable result (65.9% linear classification accuracy), pre-training with fbank feature is guaranteed to work. You only have to confirm your phone label.

I hope this helps!

[apoorv2904]

Hi,

Thank you for the clarification. After changing to cpc-phone set, I get the mentioned classification accuracy.

I will close the issue.

Thanks,
Apoorv