Did you increase the decoupled weight decay simultaneously when decreasing the learning rate? about lion-pytorch

[xiangning-chen]

Thanks for the update!
Yeah disabling weight decay for both optimizers is pretty meaningful and fair, thank you!

[lucidrains]

@xiangning-chen Hi Xiangning! Thank you for this interesting paper

So far I have been only testing with weight decay turned off. There are a lot of networks that are still trained with just plain Adam, and I wanted to see how Lion fares against Adam alone

[lucidrains]

@xiangning-chen but yes, I have noted the section in the paper where you said the weight decay needs to be higher

Let me add that to the readme to increase the chances people train it correctly

[lucidrains]

@xiangning-chen ok good luck! hope this technique holds up to scrutiny!