About loss functions about textbrewer

[airaria]

I'm glad you like this work!

For classification tasks, kd_ce_loss is preferable to kd_mse_loss in most cases.
But limited by the computational resources, we couldn't perform a comprehensive benchmark. Thus the above conclusion is based on our preliminary experiment results.

However, some authors are prefer to kd_mse_loss:
Natural Language Generation for Effective Knowledge Distillation
So we provide the both options.

If you are dealing with regression tasks, you should use kd_mse_loss instead of kd_ce_loss, as stated in
BAM! Born-Again Multi-Task Networks for Natural Language Understanding

[empty-id]

Thank you for you reply! I get it.