Understanding runtimes of different models about mlm-scoring

[JulianSlzr]

Hi Gerard,

These numbers make sense.

• LM vs MLM is due to 1 copy vs. L parallel copies. Look at our maskless finetuning to mitigate this; or perhaps unmasked scores might suit your needs (e.g., this new paper: https://arxiv.org/abs/2010.09535)
• While ALBERT has fewer unique parameters, inference still passes sequentially through 12 layers (plus the factorized embedding, which breaks one computation into two). PyTorch models are also slower right now since we go via MXNet's dataloader. Did you try increasing the batch size? You should be able to load more as the model is smaller.
• DistilBERT's inference speed is in line with the 40% reduction from their paper (Table 3: https://arxiv.org/pdf/1910.01108.pdf). You could also increase the batch size here.

Hope this helps!