Distillation loss produces NaNs about lmops

[t1101675]

What is the base model you use? Can you provide some information about the training log?
The NaN issue is probably caused by dividing by 0 in -torch.sum(x * mask.view(-1), dim=0) / torch.sum(mask.view(-1), dim=0) . You can check whether there exists any full-zero row in mask.

[khalidsaifullaah]

That was indeed the issue, looks like my tokenizer was truncating the inputs early. Thanks!