Questions about the masking about s3prl

[andi611]

Hi,

In the BERT paper, only the masked tokens are predicted.
Following BERT, we only reconstruct the masked frames.

I did not try to reconstruct all the frames, however, this can be easily done by setting this line to torch.ones_like.

Reconstructing only the masked part makes training more efficient,
I think reconstructing all will also work but with more update steps.

ANdy

[iezhanqingran]

Thanks for your reply, I will look at it!