[Feature request] Full variational Bayes about vae-torch

[y0ast]

So the idea of algorithm 2 is to perform variational inference on the parameters of the decoder too.

To implement this, it is necessary to create a new module that reparametrizes theta and then sets the parameters of the decoding layer or perhaps add ζ as parameter to each module in the decoding layer. This would not be a trivial change to the code and I have my doubts if it will improve the negLL by much. Feel free to submit a PR and I will help you out where necessary.

[AjayTalati]

Hi Joost,

thanks alot for the helpfull reply!

I am working on this now, there's some nicely documented theano code which does what you describe. It's not so bad as I have some background in stochastics/Monte Carlo. I'll a submit a PR in the next few days.

Pylearn2 has'nt implemented the adam optimizer yet, so that Theano code only gets negLL of about 94. So it seems that stochastic sampling (full VB), gets an improvment of 7, and adam gets a further improvement of about 7.

There's a new paper DRAW, which uses a recurrent encoder network to selectively read patches of the input, and a recurrent decoder network to selectively write/deposit probability mass to regions of a canvas matrix. The canvas matrix can then be used to generate/reconstruct output. They state they get a negLL of 81 - which is the world best at the moment. Without selective read/write the DRAW model is basically the same as the Welling/Kingma model, except with LTSM encoder/decoder networks, and it gets 87 negLL.

With your experience with your Variational Recurrent Auto-Encoders work, you have much of the background needed to produce the DRAW result? So perhaps we could work on this together, if you are interested?

Coding a variational recurrent auto-encoder which uses importance sampling to perform VI on the parameters of both the encoder/read mechanism and decoder/write mechanism seems like the first step in building this?

Feel free to email me if you like?

All the best,

Aj