Are pseudo labels with high confidence retained ? about bond

[cliang1453]

We optimize the student model only based on the losses of the tokens which the student model predicts with high confidence. The corresponding line in the code is

BOND/model_utils.py

Line 88 in 32f2698

label_mask = (pred_labels.max(dim=-1)[0]>_threshold)

.

[tarunbhatiaind]

Thanks for the reply. But, in the above line 'pred_labels' are coming from teacher model and you are getting the mask of confident predictions of teacher model, right ? What I understood was, you check which token predictions of teacher are confident and then calculate loss of student for those tokens only . Please correct me if I'm wrong.

[cliang1453]

What you understand is correct. I was saying that "we optimize the student model only based on the losses of the tokens which the teacher model predicts with high confidence". Sorry for the typo.

[tarunbhatiaind]

Thanks @cliang1453 . Just one more query. My task is also token level classification. Would it make sense to just utilize mean teacher for training. Something like :
Learn a model in stage 1 with less data
Then use that model to initialize teacher and student for second stage.
Give teacher all unlabeled data and it will generate pseudo labels for student to train on(calculating cross entropy loss on confident predictions of teacher). Use consistency cost between soft labels of student and teacher, and then update teacher with exponential moving average of student's weights. Continue this for later epochs.
It would be really helpful if I can just get a comment on this.
Thanks in advance!