Temperature parameter is not handled properly about softqlearning

[haarnoja]

Latest refactor removes the temperature coefficient (alpha). To adjust the temperature, you can change reward_scale instead.

[immars]

sorry to re-raise this issue,
I have implemented a version of soft q learning, and find it convenient to have a separate alpha, I think alpha acts effectively as entropy coefficient in policy gradient methods, which can be annealed without affecting value iteration of the critic.

[haarnoja]

In my experience, annealing alpha to zero was little problematic because of how it enters the value function (V = alpha * log sum exp (Q / alpha)) and naive way of implementing this and setting alpha -> 0 obviously fails. How did you fix this? If you'd like to share your code, I'd be happy to merge your PR :).

[immars]

that's right, in my experiment, if alpha is annealed below a threshold (near 0.08 or so), the training becomes numerically unstable. But it could take a much larger value at the beginning of the training to encourage exploration.

the code is now in a private branch of hobotrl. the code structure is very different from this repo i think, and is hard to merge. I created a gist with relevant code pieces.
alpha_exploration could be a object of a subclass of python float, with variable values when evaluated each time as float.

[immars]

I've pushed to hobotrl for your reference.