Your method is just equivalent to SGD with a changable global learning rate. about adabelief-optimizer

[juntang-zhuang]

No, I don't think it's equivalent to SGD. SGD does not work for GAN while AdaBelief works. Depends on what default eps you are using, I'm using 1e-16, if you set as 1e-8 it would be the case as you mentioned.
PS: EAdam is inspired by AdaBelief

[Yonghongwei]

Thanks for your reply. For the classification and detection tasks, I find this problem incurs. I have not checked the problem on other tasks. On these high-level vision tasks, SGD usually performs well. I also find small eps on these tasks will drop the performance. Therefore, I think the main reason Adabelief works well on high-level vision tasks is it performs like SGD.

[juntang-zhuang]

Thanks for reply. It's possible but not always true, I have tried eps=1e-16 with ResNet18 on ImageNet with a batchsize 4096, it achieved above 72% top-1 accuracy, much better than with eps=1e-8 and even better than SGD. I'm not sure if it's a problem with framework since I used the Flax implementation to use TPU. I feel like even for classification tasks, with proper decoupled decay and lr schedule, adaptive optimizers can perform no worse than SGD, but it would require some tuning.