Evaluating other upstream models about s3prl

[Hguimaraes]

Thank you very much @andi611

I will look into more details in the new paper, although I was using the mockingjay as a benchmark. TERA looks like a better alternative but I'm not sure if the comparisons are so straightforward for my use case.

Best regards,

[andi611]

To make a fair comparison I would like to use the "One layer RNN" that is described on your paper. The parameters used for speaker recognition and sentiment analysis are not clear to me. Could you confirm if you did use these ones that I found on your GDrive?

rnn:
mode: 'classification'
input_dim: 'None'
select_hidden: 'last'
sample_rate: 5
pre_linear_dims: []
hidden_size: 32
post_linear_dims: [32]
drop: 0.5

Hi,
Thanks for liking our work!

Yes, these are the parameters used for "one layer RNN" in the Mockingjay paper.

FYI, in our new TERA paper, we report results that are easier for comparison. The below downstream setting also aligns with the ones used in the CPC paper, hence easier to make more comparisons (Mockingjay, TERA, CPC). And I would like to recommend these settings to you.
Including:

• "linear phone classification" and "1-hidden phone classification" on the CPC reported phone set.
• "frame-wise speaker classification" with a linear model
• "average-over-time speaker classification" with a linear model, the CPC paper also reported on this setting.

In our new paper TERA, the reported results include the Mockingjay and the TERA model.
In this toolkit, scripts for running the above 4 downstream tasks are provided, you can run with the default arguments and evaluate simply:

First, prepare the phone label following instructions from here.
Next, see this section to plugin your model to our toolkit: Evaluate your upstream model
Finally, use the arguments provided here to run the above tasks: Downstream Evaluations

Kind Regards,