No learning rate annealing during training. about psgld

[ChunyuanLI]

In theory, the learning rate decay guarantees the asymptotic convergence. In practice, its implementation varies case by case.

Depending on your purpose, the answer can be different.
(1) To validate the theory, or pursue better performance, the decay scheme may help.
(2) To see the performance difference between SGLD and pSGLD, I don't think the decay scheme matters much, to see pSGLD algorithm gives better samples by adjusting the learning rate. One experiment you can try is to compare the two algorithms with the same learning rate decay. That said, perhaps more rigorous comparison is to use the same learning rate decay. I wish I have done it at the first place.

[ylfzr]

I tried pSGLD on the first experiment in [Welling 2011], which has 2 modes in the posterior and the two variables are strongly negatively correlated. I get proper posterior samples with similar learning rate annealing scheme.
Thanks.