icb-dcm / delos Goto Github PK

Using the idea of gradient flow together with an implicit ODE solver can give approximative profiles. This can in turn be used with mini-batching, maybe when combined with some kind of memory idea as implemented in adaptive SGD optimizers.
Something like this should be tested and implemented soon, as this might make approximative profiles possible for very, very large models with very, very much data...

Currently, only RMSProp with Nesterov momentum is available.
The other optimizers have to be transferred from the old version (RMSProp, Adam, Adadelta) or added (Nadam, AMSGrad, AdaMax)

Currently, the learning rate schedule (and as well the momentum and gradient history schedules) are computed by DeLOS itself based on tuning parameters.
It would be good, to have this learning etc. rate more customizable (as done with mini batches), so the user can define his own learning rat and passes it to DeLOS

Currently, the whole dataset is reassembled in mini-batching-mode.
It would be much more efficient, if the indices of the whole data set (which the user could theoretically already pass to DeLOS) would also be used in finishOptimization

Currently, DeLOS only only returns the total computation time.
It would be interesting to also have a time trace per step.