How can you use speech labels with downstream tasks? about s3prl

[rainmaker712]

Hi, thank you for appreciating our work.
To answer your questions:

1. In this work, we did not use any labels in the pre-training process. The reason is, as you mentioned, the model can learn good representations from reconstructing frames along. The purpose of this unsupervised learning framework is to allow models to take advantage of the easily acquired large amount of unlabeled data. However, you can always try to add labels in attempt to feed more information to the model.
2. Yes, when training downstream tasks, we train classifiers from scratch on top of the pre-trained models, where we use only 0.1% of labeled data (x:audio, y:phone) among the whole dataset (train-clean-360).

but in preprocess.py, why use text? Why not just use voice datasets for training?

Thanks for your reply and work. I will try to test with different language also.

[andi611]

Hi, thank you for appreciating our work.

To answer your questions:

1. In this work, we did not use any labels in the pre-training process. The reason is, as you mentioned, the model can learn good representations from reconstructing frames along. The purpose of this unsupervised learning framework is to allow models to take advantage of the easily acquired large amount of unlabeled data. However, you can always try to add labels in attempt to feed more information to the model.

2. Yes, when training downstream tasks, we train classifiers from scratch on top of the pre-trained models, where we use only 0.1% of labeled data (x:audio, y:phone) among the whole dataset (train-clean-360).

[Pydataman]

Hi, thank you for appreciating our work.

To answer your questions:

1. In this work, we did not use any labels in the pre-training process. The reason is, as you mentioned, the model can learn good representations from reconstructing frames along. The purpose of this unsupervised learning framework is to allow models to take advantage of the easily acquired large amount of unlabeled data. However, you can always try to add labels in attempt to feed more information to the model.
2. Yes, when training downstream tasks, we train classifiers from scratch on top of the pre-trained models, where we use only 0.1% of labeled data (x:audio, y:phone) among the whole dataset (train-clean-360).

but in preprocess.py, why use text? Why not just use voice datasets for training?

[andi611]

The reason that preprocess.py includes text preprocessing is that in our future work, we plan to expand this project with a downstream ASR system. Currently, the text is not used.
You can simply try to remove that part of the code if you do not wish to download and preprocess the text data.

[Pydataman]

The reason that preprocess.py includes text preprocessing is that in our future work, we plan to expand this project with a downstream ASR system. Currently, the text is not used.
You can simply try to remove that part of the code if you do not wish to download and preprocess the text data.

text label is required to train asr, this pre trian model is unsupervisied, why use text label in pre training with a downstream asr system, should not all use voice data? likely bert and so on

[andi611]

text label is required to train asr, this pre trian model is unsupervisied,

That is correct.

why use text label in pre training with a downstream asr system,

First of all, text labels are for supervised training of an ASR system. And, we do not use text labels in pre-training of the Mockingjay feature extraction model.

should not all use voice data? likely bert and so on

Yes, only voice data is used for pre-training. We did not use any text label in pre-training, hence it is like BERT.

[Pydataman]

text label is required to train asr, this pre trian model is unsupervisied,

That is correct.

why use text label in pre training with a downstream asr system,

First of all, text labels are for supervised training of an ASR system. And, we do not use text labels in pre-training of the Mockingjay feature extraction model.

should not all use voice data? likely bert and so on

Yes, only voice data is used for pre-training. We did not use any text label in pre-training, hence it is like BERT.

thanks!