Existing implementations:

• https://gist.github.com/SnippyHolloW/8a0f820261926e2f41cc#file-dnn-py-L21
• https://github.com/stencilman/deep_nets_iclr04/blob/master/lib/layer_blocks.py#L174
• https://github.com/mdenil/dropout/blob/master/mlp.py#L116
• Almost all of https://github.com/benanne/kaggle-galaxies/blob/master/layers.py#L397

http://arxiv.org/abs/1502.03167

https://github.com/torch/nn/blob/master/doc/containers.md#nn.Containers

People (you know who you are =) really like them.

=)

Either by

• checking which nodes support it and inserting these, or
• doing 'full' and then cropping.

Also interesting discussion in soumith/convnet-benchmarks#35.

But be careful:

os.fork() is problematic for CUDA if you fork a process where a CUDA context has already been initialized, even if you do not use the context in the forked process.
If the fork happens before the initialization of CUDA, it should be OK.

(Source)

See also: https://github.com/Theano/Theano/wiki/Using-Multiple-GPUs

It was quite useful, esp. the saving and loading of the training.

Ideally by making a "MultiCost" cost layer which uses the weighted sum of multiple other cost layers as cost.

Currently, there's no way to make them deterministic!

i.e. as an weighted sums of feature maps.

Would speed-up augmented prediction tremendously if it fits into RAM.

Can be done either at the Augmenter-level or at the Predictor-level, not sure yet which is better.

This should actually be a straightforward but useful change, esp. for #1.

• Tanh
• 2DConv
• MaxPool

The following verbose outputs would be useful:

• containers let you know which layers they wire together for initialization.

www.cs.toronto.edu/~fritz/absps/imagenet.pdf

obviously.

It was very, very useful in loops of experiments for getting at variance.

• Momentum
  • Test it thoroughly
• Nesterov Momentum
  • Test it thoroughly
• AdaGrad
  • Test it thoroughly
• RmsProp
  • Test it thoroughly
• AdaDelta
  • Test it thoroughly

Extremely cool visualizations by Alec Radford.

Momentum

• https://github.com/benanne/kaggle-galaxies/blob/master/layers.py#L108
• https://github.com/benanne/Lasagne/blob/master/lasagne/updates.py#L21

Nesterov Momentum

• https://github.com/benanne/kaggle-galaxies/blob/master/layers.py#L122
• https://github.com/benanne/Lasagne/blob/master/lasagne/updates.py#L36

The relevant comment by Nicolas Boulanger actually has two ways to implement it.

AdaGrad

• Notes on AdaGrad
• https://github.com/benanne/kaggle-galaxies/blob/master/layers.py#L166
• https://github.com/benanne/Lasagne/blob/master/lasagne/updates.py#L50
• https://gist.github.com/SnippyHolloW/8a0f820261926e2f41cc#file-dnn-py-L258
• https://github.com/JonathanRaiman/theano_lstm/blob/master/theano_lstm/__init__.py#L550

RmsProp

• https://github.com/benanne/Lasagne/blob/master/lasagne/updates.py#L67 (And links in comment)
• https://github.com/benanne/kaggle-galaxies/blob/master/layers.py#L184
• http://deeplearning.net/tutorial/code/lstm.py

AdaDelta

• The paper (By M. Zeiler)
• https://github.com/benanne/kaggle-galaxies/blob/master/layers.py#L205
• https://github.com/benanne/Lasagne/blob/master/lasagne/updates.py#L87
• http://deeplearning.net/tutorial/code/lstm.py
• https://gist.github.com/SnippyHolloW/8a0f820261926e2f41cc#file-dnn-py-L291
• https://github.com/JonathanRaiman/theano_lstm/blob/master/theano_lstm/__init__.py#L550