taohuang13 / hindsight-experience-replay-with-demo

PyTorch implementation of the paper Overcoming Exploration in Reinforcement Learning with Demonstrations in surgical robot manipulation tasks.

Hindsight Experience Replay with Demonstrations

PyTorch implementation of the paper Overcoming Exploration in Reinforcement Learning with Demonstrations in surgical robot manipulation tasks.

Acknowledgement

• OpenAI Baselines for the tensorflow -based implementation.
• SurRoL for the training and testing simulation platform.
• DrQv2 for the coding structure and utils modules.

Setup

We use Python 3.8 and Anaconda3 for development. To create an environment and install dependencies, run the following steps:

# Clone and cd into herdemo
git clone https://github.com/TaoHuang13/hindsight-experience-replay-with-demo.git
cd hindsight-experience-replay-with-demo

# Create and activate environment
conda create -n herdemo python=3.8 -y
conda activate herdemo

# Install dependencies
pip install -e .

Then add one line of code in gym/gym/envs/__init__.py to register SurRoL tasks:

import surrol.gym

Run the following command to collect expert demonstration via the scripted policy in the individual task file:

python surrol/data/data_generation.py --env env_name

Here we have already provided demonstrations of several tasks.

Code Navigation

At a high-level, our code relies on the generic python script: train.py for training and evaluating RL agent. We use hydra for hyperparameterize this script with experiment-specific configuration. Specifically, all experiments should be configured in the directory configs/ or command lines.

The rest of code is organized as follows:

• configs/ config files for launching expriments.
• rl/ core implementation of HER+DEMO adopted from OpenAI Baselines.
• surrol/ simulation platform for surgical robotic manipulation based on PyBullet.
• scripts/ bash scripts to running a batch of experiments.
• train.py generic python script for training and evaluating RL agent.

To simply start a experiment, run the following command:

sh scripts/run_herdemo.sh

hindsight-experience-replay-with-demo's Issues

gym.error.UnregisteredEnv

Hello,

I have been running your code. I got an error that the environment is not registered properly within the openai Gym:

/home/yantian/Research/venv/bin/python /home/yantian/Research/hindsight-experience-replay-with-demo/train.py seed=1 task_name=NeedlePick-v0
workspace: /home/yantian/Research/hindsight-experience-replay-with-demo/exp_local/NeedlePick-v0/HerDemo/HerDemo_seed=1,task_name=NeedlePick-v0
Error executing job with overrides: ['seed=1', 'task_name=NeedlePick-v0']
Traceback (most recent call last):
File "/home/yantian/Research/venv/lib/python3.8/site-packages/gym/envs/registration.py", line 158, in spec
return self.env_specs[id]
KeyError: 'NeedlePick-v0'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/yantian/Research/hindsight-experience-replay-with-demo/train.py", line 174, in main
exp = Experiment(cfg)
File "/home/yantian/Research/hindsight-experience-replay-with-demo/train.py", line 22, in init
self.setup(cfg)
File "/home/yantian/Research/hindsight-experience-replay-with-demo/train.py", line 41, in setup
self.train_env = gym.make(self.cfg.task_name)
File "/home/yantian/Research/venv/lib/python3.8/site-packages/gym/envs/registration.py", line 235, in make
return registry.make(id, **kwargs)
File "/home/yantian/Research/venv/lib/python3.8/site-packages/gym/envs/registration.py", line 128, in make
spec = self.spec(path)
File "/home/yantian/Research/venv/lib/python3.8/site-packages/gym/envs/registration.py", line 203, in spec
raise error.UnregisteredEnv("No registered env with id: {}".format(id))
gym.error.UnregisteredEnv: No registered env with id: NeedlePick-v0

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Thanks a lot!
Yantian