befelix / safe_learning

Safe reinforcement learning with stability guarantees

Safe Reinforcement Learning with Stability Guarantees

This code accompanies the paper [1] and implements the code for estimating the region of attraction for a policy and optimizing the policy subject to stability constraints. For the old numpy-based code to estimate the region of attraction in [2] see the lyapunov-learning repository. The code for learning Lyapunov functions from [3] can be found in the examples folder.

[1]	F. Berkenkamp, M. Turchetta, A. P. Schoellig, A. Krause, Safe Model-based Reinforcement Learning with Stability Guarantees in Proc. of the Conference on Neural Information Processing Systems (NIPS), 2017.

[2]	F. Berkenkamp, R. Moriconi, A. P. Schoellig, A. Krause, Safe Learning of Regions of Attraction in Uncertain, Nonlinear Systems with Gaussian Processes in Proc. of the Conference on Decision and Control (CDC), 2016.

[3]	S. M. Richards, F. Berkenkamp, A. Krause, The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systems. Conference on Robot Learning (CoRL), 2018.

Getting started

This library is tested based on both python 2.7 and 3.5, together with the following dependencies, since pip>=19 does not support --process-dependency-links (see below)

pip install pip==18.1
pip install numpy==1.14.5

Based on this, you can install the library by cloning the repository and installing it with

pip install . --process-dependency-links

To run the tests with the bash script in scripts/test_code.sh, you need to install additional dependencies with

pip install ".[test]" --process-dependency-links

The --process-dependency-links flag is needed to install gpflow==0.4.0, which is not on pypi. You can skip it if that particular version of the library is already installed.

You can the find example jupyter notebooks and the experiments in the paper in the examples folder.

safe_learning's Issues

Installation with conda

I tried installing the package within a conda environment with the instructions given but failed.

EDIT:
I managed to find an alternative way that works

conda create -new "safe_learning" python=3.6
conda activate safe_learning
conda install tensorflow==1.12.0

Remove the following lines from requirements.txt

1. numpy>=1.0,<1.15
2. tensorflow>=1.6.0,<=1.12.0

Do:

pip install pip==18.1
pip install . --process-dependency-links

On MacOS:

export KMP_DUPLICATE_LIB_OK=TRUE

The above should be done before running jupyter notebooks.

Running errors and parameter initialization problems

Hi Felix,
I have these questions.
I run lyapunov_function_learning.ipynb and got an error like this.
ModuleNotFoundError: No module named 'safe_learning'
And I don't quite understand how the initialization parameters of the Lyapunov function are obtained.

1. In all the files, I did not find it.So where can I find this safe_learning module?
2. Could you please explain in detail how the parameter initialization is done?

pip does not support tensorflow installation requirements

Hi Felix,

it would appear that pip no longer supports the version of tensorflow (<=12.0,>=1.6) that you are requiring. I did set pip to 18.1 and numpy to 1.14.5.

I get the message

Could not find a version that satisfies the requirement tensorflow<=1.12.0,>=1.6.0 (from safe-learning==0.0.1) (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 1.15.0rc0, 1.15.0rc1, 1.15.0rc2, 1.15.0rc3, 1.15.0, 1.15.2, 2.0.0a0, 2.0.0b0, 2.0.0b1, 2.0.0rc0, 2.0.0rc1, 2.0.0rc2, 2.0.0, 2.0.1, 2.1.0rc0, 2.1.0rc1, 2.1.0rc2, 2.1.0)
No matching distribution found for tensorflow<=1.12.0,>=1.6.0 (from safe-learning==0.0.1)

I instead tried setting the requirement to tensorflow==1.13.0 just to see what happens. It installed fine.

I went to try and run 1d_example.ipynb and got an error from gpflow

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-4ef92e5e9104> in <module>
      3 from functools import partial
      4 
----> 5 import gpflow
      6 import tensorflow as tf
      7 import pkg_resources

/usr/local/lib/python3.7/site-packages/gpflow/__init__.py in <module>
     16 # flake8: noqa
     17 from __future__ import absolute_import
---> 18 from . import (likelihoods, kernels, ekernels, param,
     19                model, gpmc, sgpmc, priors, gpr, svgp,
     20                vgp, sgpr, gplvm, tf_wraps, tf_hacks)

/usr/local/lib/python3.7/site-packages/gpflow/model.py in <module>
     17 from .param import Parameterized, AutoFlow, DataHolder
     18 from .mean_functions import Zero
---> 19 from scipy.optimize import minimize, OptimizeResult
     20 import numpy as np
     21 import tensorflow as tf

/usr/local/lib/python3.7/site-packages/scipy/optimize/__init__.py in <module>
    388 from __future__ import division, print_function, absolute_import
    389 
--> 390 from .optimize import *
    391 from ._minimize import *
    392 from ._root import *

/usr/local/lib/python3.7/site-packages/scipy/optimize/optimize.py in <module>
     35                    asarray, sqrt, Inf, asfarray, isinf)
     36 import numpy as np
---> 37 from .linesearch import (line_search_wolfe1, line_search_wolfe2,
     38                          line_search_wolfe2 as line_search,
     39                          LineSearchWarning)

/usr/local/lib/python3.7/site-packages/scipy/optimize/linesearch.py in <module>
     16 from warnings import warn
     17 
---> 18 from scipy.optimize import minpack2
     19 import numpy as np
     20 from scipy._lib.six import xrange

ImportError: numpy.core.multiarray failed to import

I double checked that numpy 1.14.5 was being used and even forced it by adding the code

import pkg_resources
pkg_resources.require("numpy==`1.14.5")

Any thoughts?

using GPU

Hi, please let me know what are the modifications required to run these code on GPU server.

Problem while installing the requirments.txt

I am trying to install the package with pip version 19.0.2. It seems that gpflow==0.4.0 can not be installed by this pip. I have tried using pip 8.1.1 also where it says matplotlib is unable to install. Is there any pip version in between for which I can install both gpflow==0.4.0 as well as matplotlib.

installation instructions : Failed building wheel

Hi, with reference to your earlier comments (9fc7475) I have made the necessary changes , but still I am getting these errors ,

Failed building wheel for cvxpy

Failed building wheel for subprocess32

Failed building wheel for fastcache

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

How to resolve, thanks