Why alpha and mu are global, not parameter-wise? about yellowfin_pytorch

[JianGoForIt]

Hi @glogowski-wojciech,

We have not extensively trying out per-layer lr/momentum. In YF, we want to demonstrate momentum is related the range of curvatures of all variables. That is the reason why we decide to use single lr/momentum. I am not sure in a general setting what would happen if there is multiple momentum. If the Hessian can be closely approximated by a diagonal block matrix, multiple momentum might also work I think. On the other hand, it might also be possible the Hessian can not be properly approximated by diagonal block matrix, then it might violate our analysis on analyzing the range of curvature. We do not have direct empirical evidence on this question :).