Question on different input format for teacher and student models about textbrewer

[meisyarahd]

I figured I don't have to pass the aforementioned dict key for the distillation to run. So I close this issue

[meisyarahd]

Hi, I reopen the issue because I'm still curious about the answer to my question. Although my code is working now, I just think the performance might have been influenced since the teacher model doesn't read the "token_type_ids".

In addition to my question above, let's say I want to distill a BERT model to a bi-RNN like in this paper, will Textbrewer be able to handle this case?

[airaria]

Q1:
If I understand correctly, your question is: how to give different input representations to the teacher and the student? Since your teacher model takes token_type_ids while your student model doesn't.

A solution I come up with is simply adding a **kwargs parameter to your student model's forward method, so that it can accept any additional keyword arguments, for example:

class StudentModel(nn.Module):
    def forward(self, input_ids, attention_masks, labels, **kwargs):
        ...

Now you can use token_type_ids key in your inputs for the both teacher and student.

Q2:
Yes. It's able to distill BERT to Bi-RNN. You could set kd_loss_type to mse and temperature to 1 to get the same loss function as (2) in that paper.

[meisyarahd]

Thanks! That seems to solve the issue