My everyday learning Journal
-
Overview of all optimization algorithms http://ruder.io/optimizing-gradient-descent/
...Batch gradient descent
...Stochastic gradient descent
...Mini-batch gradient descent
...SGD + Momentum ...SGD + Nesterov accelerated gradient ...AdaGrad ...AdaDelta ...RMSProp ...ADAM ...AdaMax ...Nadam ...AMSGrad -
Stochastic Gradient Descent and its Variant https://wiseodd.github.io/techblog/2016/06/22/nn-optimization/
-
Adam Optimization Algorithm