lessw2020 / best-deep-learning-optimizers Goto Github PK

Hi, there is yet another recent proposed optimizer: https://github.com/clovaai/AdamP. Do you plan to include this?

Thanks for making this nice recap of the hart of AI the optimizers!

I have a question regarding the SLS method can you implement something similar to Adam so without the Closure part ?

It would be nice to have a generic method compatible with classical optimizers in pytorch.

Another related question: is it so good like the paper claim ?

Hi,

I was wondering what are the benchmarks you used for testing these optimizers?
Thanks!

was training a model with cyclic learning rate and after 7th epoch i get nan validation loss..isn't it "exploding gradient problem"? will gradient accumulation be able to solve this issue? i don't get such errors when i try adam or adamW.
i am facing this problem whenever i try deepmemory or diffgrad optimizer,any help?

training statistics so far of the model i am trying :

Train Epoch: 0 LR: 0.0049400000 Loss: 1.878861
Dev loss: 2.1991
Train Epoch: 1 LR: 0.0086800000 Loss: 1.840539
Dev loss: 1.7849
Train Epoch: 2 LR: 0.0075800000 Loss: 1.847198
Dev loss: 1.9127
Train Epoch: 3 LR: 0.0038400000 Loss: 1.287447
Dev loss: 1.3331
Train Epoch: 4 LR: 0.0023000000 Loss: 1.416327
Dev loss: 1.2588
Train Epoch: 5 LR: 0.0060400000 Loss: 1.299999
Dev loss: 1.4838
Train Epoch: 6 LR: 0.0097800000 Loss: 1.540868
Dev loss: 1.5280
Train Epoch: 7 LR: 0.0064800000 Loss: 1.790969
Dev loss: 1.2738
Train Epoch: 8 LR: 0.0027400000 Loss: 1.092477
Dev loss: nan

hv * vi), dim=[2, 3], keepdim=True)) / vi[0, 1].numel() # Hessian diagonal block size is 9 here: torch.sum() reduces the dim 2/3.

IndexError: index 1 is out of bounds for dimension 1 with size 1

Hi, I came across this paper today, but they dont have a public implementation. Thought of sharing it here. Thanks for your work!!

https://arxiv.org/pdf/2203.01400.pdf

Hi,

I'm currently trying DeepMemory, but with latest pytorch (1.7.0), I get the following warning. Do you have any update for DeepMemory?

/content/utils/optimizers/deepmemory.py:129: UserWarning: This overload of add_ is deprecated:
	add_(Number alpha, Tensor other)
Consider using one of the following signatures instead:
	add_(Tensor other, *, Number alpha) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
  exp_avg.mul_(beta1).add_(1 - beta1, grad)

Thanks,
Clément

Hi

Please use diffGrad_v2.py from https://github.com/shivram1987/diffGrad.