daisatojp / mpo Goto Github PK

Currently I think this code is not useful for MPO learner. Many part of the code can be misleading about the correspondences between the theory and the implementation. Fix this.

Hey again.

While evaluating MPO I got some strange raise ValueError("x0 violates bound constraints.").
They originate in this line. However I now implemented a "clamping" with np.max([self.η,1e-6]).

According to their code to check for the bound constraints, this should be totally fine.
But I keep getting this error from time to time and the training for the algorithm completely stops as it errors out.

Lines from the corresponding file scipy/optimize/_numdiff.py:

    if np.any((x0 < lb) | (x0 > ub)):
        ("`x0` violates bound constraints.")

Bounds are prepared like this:

def _prepare_bounds(bounds, x0):
    """
    Prepares new-style bounds from a two-tuple specifying the lower and upper
    limits for values in x0. If a value is not bound then the lower/upper bound
    will be expected to be -np.inf/np.inf.

    Examples
    --------
    >>> _prepare_bounds([(0, 1, 2), (1, 2, np.inf)], [0.5, 1.5, 2.5])
    (array([0., 1., 2.]), array([ 1.,  2., inf]))
    """
    lb, ub = [np.asarray(b, dtype=float) for b in bounds]
    if lb.ndim == 0:
        lb = np.resize(lb, x0.shape)

    if ub.ndim == 0:
        ub = np.resize(ub, x0.shape)

    return lb, ub

Any idea to this? Not really an algorithm related question but for me this seems strange.

Hey Dai,

On lines 291:298 of mpo.py u compute a loss_p.
Could you explain why the parameters in the construction of the distributions are switched?

In my understanding this should not be the case as we are interested in the real probabilites and not some sort of mixed stuff.

Could you further explain what loss exactly is computed there, since it is not present in the repo your implementation is taken from.

My background is a little not existent on the MPO mathematical background, and I sadly can't wrap my head around the equations in the original paper..

Thanks in advance!

Hi, first of all, thanks for your awesome implementation, it really helped me understanding MPO much better!

I tried to use the model for a custom environment, which sometimes works fine, but quite often i get an error when computing the KL divergence: for some reason, the determinants of Σ and Σi get negative for some samples, resulting in nans in line 47 of mpo.py because the argument of the log is negative:
inner_Σ = torch.log(Σ.det() / Σi.det()) - n + btr(Σ_inv @ Σi) # (B,)
The output of the actor looks alright to me, it is a diagonal matrix with positive entries on the diagonal axis, as described in the paper. As far as I understand, the covariance Σ should be positive definite when computed with the Cholesky output from the actor.
I'm not too sure about this because it's been some time since I had my last math lecture, but according to a quick Google search, the determinant of a positive definite matrix should be positive. So i don't understand how the determinant of Σ or Σi get negative.
Do you have any idea how this could happen? The state that is fed to the actor looks alright, no nans or anything similar, just normal floats.

enable model to learn Discrete Action Space

Hi, I developed my own custom environment (still based on gym), and was wondering whether your code can support that?

use Retrace Algorithm (paper) as Policy Evaluation

I know basic stats and calculus but I can't match anything from the paper to anything in this repo. I can't even pin point which part is the a posteriori in the paper.
It is above my head. The code looks great and I like the use of the symbols.
I would like to get to know it up close to use it as a multi objective MPO, (MOMPO) and RL algorithms beyond.
Thank you.

Hello Dai,

On lines 365 and 366 of file mpo.py the code is the following:

      self.α_μ = np.clip(0.0, self.α_μ, self.α_μ_max)
      self.α_Σ = np.clip(0.0, self.α_Σ, self.α_Σ_max)

shouldn't it be like that?

      self.α_μ = np.clip(self.α_μ, 0.0, self.α_μ_max)
      self.α_Σ = np.clip(self.α_Σ, 0.0,  self.α_Σ_max)

Regards!