duvenaud / relax

Optimizing control variates for black-box gradient estimation

Backpropagation through the Void: Optimizing control variates for black-box gradient estimation

https://arxiv.org/abs/1711.00123

by Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, David Duvenaud

We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables, based on gradients of a learned function. These estimators can be jointly trained with model parameters or policies, and are applicable in both discrete and continuous settings. We give unbiased, adaptive analogs of state-of-the-art reinforcement learning methods such as advantage actor-critic. We also demonstrate this framework for training discrete latent-variable models.

Code for VAE Experiments lives here. The Discrete RL experiments can be found at: https://github.com/wgrathwohl/BackpropThroughTheVoidRL.

A simplified, pure-python implementation is in /relax-autograd/relax.py

If you have any questions about the code or paper please contact Will Grathwohl ([email protected]). The code is in "research-state" at the moment and I will be updating it periodically. If you have questions feel free to email me and I will do my best to respond. -Will

relax's Issues

Error

surrogate_cond, grad_surrogate_cond = value_and_grad(surrogate_cond)(params):

value_and_grad only applies to real scalar-output functions. Try jacobian, elementwise_grad or holomorphic_grad.

gradvars issue

Hi, can anyone help me? It got the following issue?

Traceback (most recent call last):
  File "mnist_vae.py", line 100, in <module>
    gradvars = inf_gradvars + gen_gradvars + rebar_optimizer.variance_gradvars
TypeError: unsupported operand type(s) for +: 'zip' and 'list'

about the d z_tilde / d logits

I found that in the Appendix B of paper, v' can still differentiate with respect to \theta but the code (https://github.com/duvenaud/relax/blob/master/pytorch_toy.py#L105) omits it. Why? Because the effect of it is small?

Computing v' in Appendix B

Dear authors,

Thanks for sharing your code. I noticed a difference between the code and paper in computing \tilde{z}. Specifically, in the code 1-\theta is performed, but in Appendix B of the paper when computing v', \theta is directly used. Is the latter a typo, or am I missing something?

Thanks