RuntimeError: Found dtype Float but expected BFloat16 about embmarker

[yjw1029]

From the log information, I think you enabled deepspeed to run the experiments, which is different from the settings we provided. Do you follow the steps in README? (i.e. using the same docker image and the same lanching scripts)?

[yjw1029]

Meanwhile, the log information is strange 🤔️. The total_loss += loss.detach().float() should have no relationship to backward process. If you still have this issue after checking the environmental setup and launching script, you can try to remove total_loss += loss.detach().float() line since it is for logging and irrelevant to the main logic.