Is there any solusion to overcome underflow issues? about deepspeed

[xuanhua]

@zhoubay yes, It looks like there is nowhere to set raise_error_at_min_scale at False if you grep deepspeed's source code.
And this is not an 'underflow' issue but an 'overflow' one. Maybe you should just use fp32 to do your training of bf16 instead of fp16. And you can also double check your code if there is some bug that cause this overflow issue.

[zhoubay]

@zhoubay yes, It looks like there is nowhere to set raise_error_at_min_scale at False if you grep deepspeed's source code. And this is not an 'underflow' issue but an 'overflow' one. Maybe you should just use fp32 to do your training of bf16 instead of fp16. And you can also double check your code if there is some bug that cause this overflow issue.

Thank you for your reply! After changing training of fp16 to bf16, the error disappears!

I'm closing this issue!