recommended experiments about adabelief-optimizer

[juntang-zhuang]

Hi, thanks for the comment, they are very good points.

For the first question, I think extra experiments with different learning rate might answers it. Note that m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, and m_t - g_t = \beta_1 ( m_{t-1} - g_t ). This implies that using a different learning rate (divided by \beta_1) is equivalent to using g_t - m_{t-1}. Some experiment with different learning rates are in the appendix, though only on CIFAR10 dataset, seems different learning rates do not generate a significantly different result. But I have not tested on more examples.

For the second question, it's quite likely that the default \beta values should be set as different from Adam. I have not tested it yet, so don't have a concrete result now. Will try that later or incorporate it into the next version of release.