What is the actual input to the model? about s3prl

[andi611]

For the inner hidden state, can that be thought of as an autoencoded representation?

Yes, that is the representation we want to learn.

And can we somehow go back to raw audio? With MFCC, for example, I believe we cannot.

You can, but performance may vary (depending on the downstream task).

[andi611]

For reconstruction, I’m talking about no downstream task... simple get back raw audio.

Example: raw audio -> Mel spectrogram -> encoded representation -> Mel spectrogram -> raw audio

For example, with wav2vec, this would be difficult since it goes from 480-dimensions for 30ms at 16KHz down to a 39-dimensional vector. There’s too much loss of information to adequately reconstruct.

Then it depends on the reconstruction target, if the target is mel or linear I believe we can get back raw waveform.
MFCC for example, I also belive we cannot.

[andi611]

The input to pre-trained models can be any acoustic feature of choice (MFCC, FBANK, fMLLR, mel, linear, etc).
The input we used in Mockingjay is 80-dim Mel and its first derivative, hence a total of 160-dim Mel.
The input we used in TERA is 40-dim fMLLR extracted from Kaldi.

[shamoons]

Hmmm. Okay thank you for clarifying. For the inner hidden state, can that be thought of as an autoencoded representation?

And can we somehow go back to raw audio? With MFCC, for example, I believe we cannot.

[shamoons]

For reconstruction, I’m talking about no downstream task... simple get back raw audio.

Example: raw audio -> Mel spectrogram -> encoded representation -> Mel spectrogram -> raw audio

For example, with wav2vec, this would be difficult since it goes from 480-dimensions for 30ms at 16KHz down to a 39-dimensional vector. There’s too much loss of information to adequately reconstruct.