nickledave / thrillington

Replication of "Recurrent models of visual attention", Mnih et al. 2014

Recurrent Models of Visual Attention

Replication in Tensorflow of the following paper:
Mnih, Volodymyr, Nicolas Heess, and Alex Graves.
"Recurrent models of visual attention."
Advances in neural information processing systems. 2014.
https://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention

Based in part on the following implementations:

• https://github.com/torch/rnn/blob/master/examples/recurrent-visual-attention.lua
  • (license: https://github.com/torch/rnn/blob/master/LICENSE.txt)
• https://github.com/seann999/tensorflow_mnist_ram
  • (MIT license: https://github.com/seann999/tensorflow_mnist_ram/blob/master/LICENSE)
• https://github.com/kevinzakka/recurrent-visual-attention

installation

$ pip install thrillington
(thrillington because there is already a ram on PyPI, and because https://en.wikipedia.org/wiki/Thrillington)

usage

The library can be run from the command line with a config file.

$ ram train ./RAM_config-2018-10-21.ini

...

  0%|          | 0/10000 [00:00<?, ?it/s]

config.train.resume is False,
will save new model and optimizer to checkpoint: /home/you/data/ram_output/results_20181021/checkpoints/ckpt

Epoch: 1/200 - learning rate: 0.001000

282.5s - hybrid loss: 1.690 - acc: 6.000: 100%|██████████| 10000/10000 [04:42<00:00, 35.65it/s]
  0%|          | 0/10000 [00:00<?, ?it/s]

mean accuracy: 9.97
mean losses: LossTuple(loss_reinforce=-1.1296023, loss_baseline=0.09972435, loss_action=2.3005059, loss_hybrid=1.2706277)

Epoch: 2/200 - learning rate: 0.001000

282.8s - hybrid loss: 1.223 - acc: 10.000: 100%|██████████| 10000/10000 [04:42<00:00, 35.50it/s]
  0%|          | 0/10000 [00:00<?, ?it/s]
...

For a detailed explanation of the config file format, please see here

CHANGELOG

To see past changes and work in progress, please check out the CHANGELOG.

Acknowledgments

• Research funded by the Lifelong Learning Machines program, DARPA/Microsystems Technology Office, DARPA cooperative agreement HR0011-18-2-0019

thrillington's Issues

add logging!

fix Tester to match fixes in Trainer, i.e. tf.nn.softmax on predictions

Fix momentum=constant in trainer

Here: https://github.com/NickleDave/RAM/blob/50d799c2c2887fc770ef402c96c087cb0d9bdf40/src/ram/trainer.py#L99
Should be momentum=config.train.momentum

have logger add 'interrupt' line upon ctrl-c during training

does zero padding images work correctly with normalization?

i.e. if mean after normalization is not zero then am I padding them with "grey" and it's mis-leading the model somehow?

just have one output dir instead of samples_dir, examples_dir, train_inds_dir, etc

include Element license from torch/rnn

can speed up data loading with num_parallel_calls and/or prefetch?

Try something like
https://www.tensorflow.org/guide/performance/datasets

Wish it was as easy as just adding an argument to dataloader like in PyTorch

add MC sampling to training? not just val / test

add ability to feed deterministic `l_0` to test how glimpses vary as function of intial loc

add [test] section to config

asi:

[test]
save_examples = True
num_examples_to_save = 15
test_examples_dir = test_examples

add Patience

add option to save shuffled indices for each epoch

fix how nan'd models are saved

add random seed option to train!!!

should be set at graph level at start of session

Need to be sure to set before any ops executed
https://github.com/tensorflow/tensorflow/issues/ 14134

fix running test across machines, weird path error

write non-eager version so that random seed actually works

add analysis Makefile that parses logs and spits out test acc as function of hyperparams

logger should log which checkpoint it uses!

fix / finish tests! e.g. for modules

add option for Trainer to shuffle train dataset before each epoch

add adam as option for optimizer

add 'replicates' option

Use Monte Carlo sampling for validation and test accuracy

Average across multiple runs, and/or add ability to make model deterministic (not mentioned in paper tho)

plot glimpse paths across training to see change -- becoming less random?

add `save_nan` option

in [TRAIN] that will move checkpoints etc from model whose loss functions take on nan values into a sub-folder for post-mortem

have cli add paths from paths_dict to [data] section of config

less fragile, more transparent then expecting a paths_config dict to be there, doesn't require a user to obey the pattern of making a function that saves a thing called paths_dict.json

Add other MNISTs e.g. MNIST cluttered

https://github.com/deepmind/mnist-cluttered