action distribution for estimating V about softqlearning

[haarnoja]

Thanks for your question. We use uniform sampling because there is no direct way to evaluate the log-probabilities of action of SVGD policies, which would be needed for the importance weights. Using some other tractable policy representation could fix this issue.

You're right that uniform samples do not necessarily scale well to higher dimensions. I haven't really studied how accurate the uniform value estimator is, but from my experience, using more samples to estimate the value improves the performance only marginally.

[immars]

ok, i see.
Thanks for the reply!

I could be totally misunderstanding, but doesn't appendix C.2 talk about how one can use the sampling network for q_a' and derive the corresponding densities so long as the jacobian of a'/epsilon' is non-singular?

[haarnoja]

I see, that's indeed confusing. You are right in that we could compute the log probs if the sampling network is invertible. My feeling is that, in our case, the network does not remain invertible, and that the log probs we would obtain that way are wrong. We initially experimented with this trick (and that's why we discuss it in the appendix), but in the end, uniform samples worked better. We'll fix this in the next version of the paper, thanks for pointing it out!

My pleasure! Glad I was sort of on the right track. That's very interesting, especially since non-singular weight matrices or choice of activation function are the only thing off the top of my head that might make a feedforward net non-invertible. I might play around with that.

[SJTUGuofei]

Also in "softqlearning/softqlearning/algorithms/sql.py"
ys = tf.stop_gradient(self._reward_scale * self._rewards_pl + (
1 - self._terminals_pl) * self._discount * next_value)
I just wonder is it sufficient that only one sample for computing the Expectation in $\hat Q$.
Thanks a lot!

[haarnoja]

Do you mean the expectation over states and actions in Eq. (11)? It is OK, since the corresponding gradient estimator is unbiased, though it can have high variance.

[SJTUGuofei]

I see.
Thank you so much!