jachiam / cpo

Constrained Policy Optimization

Constrained Policy Optimization for rllab

Constrained Policy Optimization (CPO) is an algorithm for learning policies that should satisfy behavioral constraints throughout training. [1]

This module was designed for rllab [2], and includes the implementations of

• Constrained Policy Optimization
• Primal-Dual Optimization
• Fixed Penalty Optimization

described in our paper [1].

To configure, run the following command in the root folder of rllab:

git submodule add -f https://github.com/jachiam/cpo sandbox/cpo

Run CPO in the Point-Gather environment with

python sandbox/cpo/experiments/CPO_point_gather.py 

---

1. Joshua Achiam, David Held, Aviv Tamar, Pieter Abbeel. "Constrained Policy Optimization". Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
2. Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel. "Benchmarking Deep Reinforcement Learning for Continuous Control". Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.

cpo's Issues

[Question] Bounds when f=V(π)

How does the term

gets converted to the advantage term of

when f = V(π)

Re-implementation of CPO

I have been struggling for re-implementing this thing using pytorch for a long time, yet still no break through. Does anybody have any suggestions?

Failure predictor networks

The paper mentions in Section 10.3.2 that failure predictor networks are used for experiments other than the Point Gather. Correct me if I am wrong, but I cannot find the implementation of the same in this repository. Can anyone point me to some reference on how they can be incorporated?

Thanks!

trying to run the code

anyone run experiment successfully?(e.g. CPO_point_gather.py)
i've been trying for weeks, and bugs constantly occur.
anyone know all the packages(with version) required for this code?

Thanks!

Unable to import MjModel from rllab.mujoco_py

Hi, i'm trying to Run CPO in the Point-Gather environment. However, the env can't import well.

To be specific, there is no “MjModel” or anything like it in rllab.mujoco_py.

Is anyone know where is “MjModel”?