avisingh599 / reward-learning-rl

[RSS 2019] End-to-End Robotic Reinforcement Learning without Reward Engineering

reward-learning-rl

This repository is the official implementation of the following paper:

End-to-End Robotic Reinforcement Learning without Reward Engineering
Avi Singh, Larry Yang, Kristian Hartikainen, Chelsea Finn, Sergey Levine
Robotics: Science and Systems 2019
Website | Video | Arxiv

Visual Draping	Visual Pushing	Visual Bookshelf

Visual Door Opening	Visual Pusher	Visual Picker

We propose a method for end-to-end learning of robotic skills in the real world using deep reinforcement learning. We learn these policies directly on pixel observations, and we do so without any hand-engineered or task-specific reward functions, and instead learn the rewards for such tasks from a small number of user-provided goal examples (around 80), followed by a modest number of active queries (around 25-75).

This implementation is based on softlearning.

Getting Started

Prerequisites

The environment can be run either locally using conda or inside a docker container. For conda installation, you need to have Conda installed. For docker installation you will need to have Docker and Docker Compose installed. Also, most of our environments currently require a MuJoCo license.

Conda Installation

1. Download and install MuJoCo 1.50 from the MuJoCo website. We assume that the MuJoCo files are extracted to the default location (~/.mujoco/mjpro150).

2. Copy your MuJoCo license key (mjkey.txt) to ~/.mujoco/mjkey.txt:

3. Clone reward-learning-rl

git clone https://github.com/avisingh599/reward-learning-rl.git ${REWARD_LEARNING_PATH}

4. Create and activate conda environment, install softlearning to enable command line interface.

cd ${REWARD_LEARNING_PATH}
conda env create -f environment.yml
conda activate softlearning
pip install -e ${REWARD_LEARNING_PATH}

The environment should be ready to run. See examples section for examples of how to train and simulate the agents.

Finally, to deactivate and remove the conda environment:

conda deactivate
conda remove --name softlearning --all

Docker Installation

docker-compose

To build the image and run the container:

export MJKEY="$(cat ~/.mujoco/mjkey.txt)" \
    && docker-compose \
        -f ./docker/docker-compose.dev.gpu.yml \
        up \
        -d \
        --force-recreate

You can access the container with the typical Docker exec-command, i.e.

docker exec -it softlearning bash

See examples section for examples of how to train and simulate the agents.

Finally, to clean up the docker setup:

docker-compose \
    -f ./docker/docker-compose.dev.gpu.yml \
    down \
    --rmi all \
    --volumes

Examples

Training an agent

softlearning run_example_local examples.classifier_rl \
--n_goal_examples 10 \
--task=Image48SawyerDoorPullHookEnv-v0 \
--algorithm VICERAQ \
--num-samples 5 \
--n_epochs 300 \
--active_query_frequency 10

The tasks used in the paper were Image48SawyerPushForwardEnv-v0, Image48SawyerDoorPullHookEnv-v0 and Image48SawyerPickAndPlace3DEnv-v0. For the algorithm, you can experiment with VICERAQ, VICE, RAQ, SACClassifier, and SAC. The --num-samples flag specifies the number of random seeds launched. All results in the paper were averaged across five random seeds. The hyperparameters are stored in examples/classifier_rl/variants.py.

examples.classifier_rl.main contains several different environments. For more information about the agents and configurations, run the scripts with --help flag: python ./examples/classifier_rl/main.py --help.

Version history

v0.1

• This version contains the code to reproduce the results in Singh et al, RSS 2019.

Citation

If this codebase helps you in your academic research, you are encouraged to cite our paper. Here is an example bibtex:

@article{singh2019,
  title={End-to-End Robotic Reinforcement Learning without Reward Engineering},
  author={Avi Singh and Larry Yang and Kristian Hartikainen and Chelsea Finn and Sergey Levine},
  journal={Robotics: Science and Systems},
  year={2019}
}

If you mainly use the VICE algorithm implemented here, you should also cite:

@article{fu2018,
  title={Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition},
  author={Justin Fu and Avi Singh and Dibya Ghosh and Larry Yang and Sergey Levine},
  journal={Neural Information Processing Systems},
  year={2018}
}

reward-learning-rl's Issues

ImportError: cannot import name 'register_goal_example_envs'

Hi, avisingh599 .

I have built docker image from README. But when I run "softlearning run_example_local examples.classifier_rl --n_goal_examples 10 --task=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 5 --n_epochs 300 --active_query_frequency 10" . There is a problem:

Traceback (most recent call last):
  File "/opt/conda/envs/softlearning/bin/softlearning", line 11, in <module>
    load_entry_point('softlearning', 'console_scripts', 'softlearning')()
  File "/softlearning_files/reward-learning-rl/softlearning/scripts/console_scripts.py", line 202, in main
    return cli()
  File "/opt/conda/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/softlearning_files/reward-learning-rl/softlearning/scripts/console_scripts.py", line 71, in run_example_local_cmd
    return run_example_local(example_module_name, example_argv)
  File "/softlearning_files/reward-learning-rl/examples/instrument.py", line 208, in run_example_local
    example_args = example_module.get_parser().parse_args(example_argv)
  File "/softlearning_files/reward-learning-rl/examples/classifier_rl/__init__.py", line 26, in get_parser
    from .utils import get_parser
  File "/softlearning_files/reward-learning-rl/examples/classifier_rl/utils.py", line 16, in <module>
    from multiworld.envs.mujoco import register_goal_example_envs
ImportError: cannot import name 'register_goal_example_envs'

The pip show that i have installed multiworld and I can import it in python cli:

>>> import multiworld.envs.mujoco as mu
>>> mu
<module 'multiworld.envs.mujoco' from '/opt/conda/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/__init__.py'>
>>> dir(mu)
['LOGGER', '_REGISTERED', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'create_image_48_sawyer_push_and_reach_arena_env_reset_free_v0', 'create_image_48_sawyer_push_and_reach_arena_env_v0', 'create_image_48_sawyer_reach_xy_env_v1', 'create_image_84_sawyer_reach_xy_env_v1', 'gym', 'logging', 'register', 'register_custom_envs']

It seem like the multiworld.envs.mujoco mudule do not has register_goal_example_envs.

How can I solove this problem?
Thanks!

VICE vs. SACClassifier

This is not a issue about the code implementation per se, but rather a question on the difference between two algorithms.

It seems that the VICE class implementation follows the equation from the original paper as well as the RSS paper when training the "logit" f(s) via the softmax discriminator D(s,a) with cross-entropy loss.

However, the SACClassifier class implementation does not use log_pi(a|s) and instead trains the "logit" via the sigmoid discriminator D(s) with cross-entropy loss. Since the SACClassifier utilizes negatives samples (by sampling from the replay buffer) when training the "logit" (or equivalently the event prob.) it doesn't seem to be the "Naive Classifier" case mentioned in the RSS paper.

What is the reasoning/theory behind SACClassifier? Any references (relevant paper, etc.) would be much appreciated :)

Multiworld Module

Hi,
I am using Anaconda to install your work but when I run "softlearning run_example_local examples.classifier_rl --n_goal_examples 10 --task=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 5 --n_epochs 300 --active_query_frequency 10
", I get error:
File "/home/nasim/reward-learning-rl/examples/classifier_rl/utils.py", line 16, in
from multiworld.envs.mujoco import register_goal_example_envs
ModuleNotFoundError: No module named 'multiworld'

I git clone https://github.com/vitchyr/multiworld.git but it does not work. Which Multiworld module have you used?

Can these tasks be ran with Mujoco_py version 0.5.7?

Hi all~,

When I tried to install Mujoco-py it keeps on getting the error: "Building wheel for mujoco-py (setup.py) ..." error and "You appear to be missing MuJoCo. We expected to find the file here: /Users/maggiefeng/.mujoco/mujoco200". However, when I install Mujoco-py 0.5.7 it can be installed successfully. Hence just wondering if the tasks described in the paper can be ran with version 0.5.7?

My environment is:

MacOS Mojave version 10.14.1
Python 3.5.2
Gym version I am not sure but I tried version 0.9 (but never succeeded overall)

Note: in the screenshot below, though I downgraded to python 3.5.2, the installed Mujoco-py 0.5.7 is still in the Python 3.7 library, not sure if this is an issue? Thank you very much for your help!

where is {SAC_CHECKPOINT_DIR} path

Hi~~
some things make me confused
I can run
softlearning run_example_local examples.development
after ten minutes, the new folder is
/home/ivan/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-05-09T11-56-23-2019-05-09T11-56-23
it have 1.log_syncadkyfm7w.log 2.params.pkl 3.result.json 4.params.json 5.progress.csv
in folder(/dc7a2587-algorithm=VICERAQ-seed=7463_2019-05-09_11-56-239zwaq6rv/)

and I run:
python examples/development/simulate_policy.py ~/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-05-09T11-56-23-2019-05-09T11-56-23/dc7a2587-algorithm\=VICERAQ-seed\=7463_2019-05-09_11-56-239zwaq6rv/ --max-path-length=1000 --num-rollouts=1 --render-mode=human
then output
FileNotFoundError: [Errno 2] No such file or directory: '/home/ivan/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-05-09T11-56-23-2019-05-09T11-56-23/params.json'

I cp the params.json to ..
but it output
FileNotFoundError: [Errno 2] No such file or directory: '/home/ivan/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-05-09T11-56-23-2019-05-09T11-56-23/dc7a2587-algorithm=VICERAQ-seed=7463_2019-05-09_11-56-239zwaq6rv/checkpoint.pkl'

I think that I missed something or the some steps. thinks!

Incompatibility issue with tensorflow version 2

Hello,

Thank you for sharing the code for the paper.
I would like to point that it seems that there is an incompatibility issue of the code
with the tensorflow version. More particularly I have a latest model of GPU, so
I have to use a latest version of tensorflow (version>=2).
However, running a simple example gives the following error:

softlearning run_example_local examples.classifier_rl --n_goal_examples 1 --task=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 1 --n_epochs 1 --active_query_frequency 1

Traceback (most recent call last):
  File "/home/argyrioschristodoul/anaconda3/envs/softlearning/bin/softlearning", line 33, in <module>
    sys.exit(load_entry_point('softlearning', 'console_scripts', 'softlearning')())
  File "/home/argyrioschristodoul/Projects/reward-learning-rl/softlearning/scripts/console_scripts.py", line 202, in main
    return cli()
  File "/home/argyrioschristodoul/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/argyrioschristodoul/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/argyrioschristodoul/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/argyrioschristodoul/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/argyrioschristodoul/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/argyrioschristodoul/Projects/reward-learning-rl/softlearning/scripts/console_scripts.py", line 71, in run_example_local_cmd
    return run_example_local(example_module_name, example_argv)
  File "/home/argyrioschristodoul/Projects/reward-learning-rl/examples/instrument.py", line 208, in run_example_local
    example_args = example_module.get_parser().parse_args(example_argv)
  File "/home/argyrioschristodoul/Projects/reward-learning-rl/examples/classifier_rl/__init__.py", line 26, in get_parser
    from .utils import get_parser
  File "/home/argyrioschristodoul/Projects/reward-learning-rl/examples/classifier_rl/utils.py", line 7, in <module>
    import softlearning.algorithms.utils as alg_utils
  File "/home/argyrioschristodoul/Projects/reward-learning-rl/softlearning/algorithms/__init__.py", line 1, in <module>
    from .sql import SQL
  File "/home/argyrioschristodoul/Projects/reward-learning-rl/softlearning/algorithms/sql.py", line 9, in <module>
    from .rl_algorithm import RLAlgorithm
  File "/home/argyrioschristodoul/Projects/reward-learning-rl/softlearning/algorithms/rl_algorithm.py", line 16, in <module>
    class RLAlgorithm(tf.contrib.checkpoint.Checkpointable):
AttributeError: module 'tensorflow' has no attribute 'contrib'

The code supports tensorflow version 1.13.0. I managed to run the previous example with my CPU, however when I use it with a tf-nightly-gpu (2.7.0-dev20210806) version then I get the previous runtime error.

Thank you

The conditions difference between training and evaluation in simulated tasks

Hi.
This is a question on the difference between training and evaluation.

In the normal RL simulations, there are training and evaluation. I know this two will have slight conditions difference to generalize the learning.

In the paper and code, I could not find the difference. So are there any conditions difference between training and evaluation in Visual Pusher task, Visual Door Opening task and Visual Picker task? If any, could you tell the difference and show me the part of code?

Regards,

FudiFudi

Re: running the code on local server + effectiveness of Mujoco simulation

Hi Avi,

While I am still tackling various issues with installation, just want to raise two questions that've been concerning me:

1. If I want to use GPU on a local server to run the code, do I just simply run the example codes on the server? any other installations that I need to do?

2. Regarding the effective of Mujoco simulation - as I am new in this field, just wondering:

1. What is the pros and cos of doing simulation on Mujoco+softlearning vs on real robotic arms? Is it that simulation can speed up the learning process hence is used before experimenting on real robots?

2. Are there big differences on the results obtained from simulations vs the result on real robots provided the same set of policies/strategies?

Many thanks for your reply!

failed call to cuInit

I have problem running the code on GPU. I'm quit sure I have GPU running in my system. The issue is the same for both docker and conda version. Do I miss something?
When I run debug the GPU works:
softlearning run_example_debug examples.classifier_rl --n_goal_examples 10 --task=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 1 --n_epochs 50 --active_query_frequency 10
When I run local only CPU works:
softlearning run_example_local examples.classifier_rl --n_goal_examples 10 --task=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 1 --n_epochs 50 --active_query_frequency 10

Error: (pid=3343) 2019-08-13 11:02:51.856630: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected

Handling reset when training on real robot

Hi,

how did you handle resetting the environment on the real robot? For example, in the pushing case, is there someone in attendance who gets prompted to reset the cup on environment resets, or does that work entirely without manual reset like in the arena environments (e.g. when it has reached the goal, or an irrecoverable position).

docker installation

I have some problems when i use the command:
(export MJKEY="$(cat ~/.mujoco/mjkey.txt)"
&& docker-compose
-f ./docker/docker-compose.dev.cpu.yml
up
-d
--force-recreate)
to install.
And the wrong information is as following:
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Ran pip subprocess with arguments:
['/opt/conda/envs/softlearning/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.3p1qs83v.requirements.txt']
Pip subprocess output:
Collecting git+https://github.com/deepmind/dm_control.git@0260f3effcfe2b0fdb25d9652dc27ba34b52d226 (from -r /tmp/./requirements.txt (line 23))
Cloning https://github.com/deepmind/dm_control.git (to revision 0260f3effcfe2b0fdb25d9652dc27ba34b52d226) to ./pip-req-build-10y0yui8
Collecting git+https://github.com/avisingh599/multiworld.git@19bf319422c0016260166bf64e194552bf2a9e68 (from -r /tmp/./requirements.txt (line 51))
Cloning https://github.com/avisingh599/multiworld.git (to revision 19bf319422c0016260166bf64e194552bf2a9e68) to ./pip-req-build-g6jw0ulu
Collecting git+https://github.com/hartikainen/mujoco-py.git@29fcd26290c9417aef0f82d0628d29fa0dbf0fab (from -r /tmp/./requirements.txt (line 52))
Cloning https://github.com/hartikainen/mujoco-py.git (to revision 29fcd26290c9417aef0f82d0628d29fa0dbf0fab) to ./pip-req-build-a7lmrnm6

Pip subprocess error:
Running command git clone -q https://github.com/deepmind/dm_control.git /tmp/pip-req-build-10y0yui8
Running command git checkout -q 0260f3effcfe2b0fdb25d9652dc27ba34b52d226
Running command git clone -q https://github.com/avisingh599/multiworld.git /tmp/pip-req-build-g6jw0ulu
Running command git checkout -q 19bf319422c0016260166bf64e194552bf2a9e68
Running command git clone -q https://github.com/hartikainen/mujoco-py.git /tmp/pip-req-build-a7lmrnm6
Running command git checkout -q 29fcd26290c9417aef0f82d0628d29fa0dbf0fab
fatal: reference is not a tree: 29fcd26290c9417aef0f82d0628d29fa0dbf0fab
ERROR: Command errored out with exit status 128: git checkout -q 29fcd26290c9417aef0f82d0628d29fa0dbf0fab Check the logs for full command output.

CondaEnvException: Pip failed

ERROR: Service 'softlearning-dev-cpu' failed to build: The command '/bin/bash -c echo "${MJKEY}" > ~/.mujoco/mjkey.txt && conda env update -f /tmp/environment.yml && conda clean --all -y && rm ~/.mujoco/mjkey.txt' returned a non-zero code: 1

Error in saving the videos

Hi @avisingh599 @hartikainen , when I am running "softlearning run_example_local examples.classifier_rl --n_goal_examples 10 --task=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 5 --n_epochs 300 --active_query_frequency 10 --video-save-frequency=1", I encountered the following error in saving the videos:
(I have used the tag v0.1 of the repository @avisingh599 - could this be the issue?)
Thanks!

ray.exceptions.RayActoError when running examples

I use anaconda version on Ubuntu 18.04.

I installed conda version with slightly changed requirements.txt:

git+https://github.com/hartikainen/mujoco-py.git

instead of :

git+https://github.com/hartikainen/mujoco-py.git@29fcd26290c9417aef0f82d0628d29fa0dbf0fab

The when running the example program from README I get the following errors for every algorithm:

(softlearning) ➜  reward-learning-rl git:(master) ✗ softlearning run_example_local examples.classifier_rl \
--n_goal_examples 10 \
--task=Image48SawyerDoorPullHookEnv-v0 \
--algorithm RAQ \    
--num-samples 5 \
--n_epochs 300 \
--active_query_frequency 10

/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.1) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)

WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
If you depend on functionality not listed there, please file an issue.

WARNING: Logging before flag parsing goes to stderr.
I1120 15:10:47.455576 140499140376000 __init__.py:34] MuJoCo library version is: 200
Warning: robosuite package not found. Run `pip install robosuite` to use robosuite environments.
I1120 15:10:47.474585 140499140376000 __init__.py:333] Registering multiworld mujoco gym environments
I1120 15:10:48.355936 140499140376000 __init__.py:14] Registering goal example multiworld mujoco gym environments
2019-11-20 15:10:48,428	INFO node.py:469 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-11-20_15-10-48_4328/logs.
2019-11-20 15:10:48,537	INFO services.py:407 -- Waiting for redis server at 127.0.0.1:49660 to respond...
2019-11-20 15:10:48,655	INFO services.py:407 -- Waiting for redis server at 127.0.0.1:39623 to respond...
2019-11-20 15:10:48,657	INFO services.py:804 -- Starting Redis shard with 3.35 GB max memory.

======================================================================
View the dashboard at http://10.152.10.45:8080/?token=dd68d1fa6ceee3a8e9315d6ec51c1c286dcc6c099b9ba066
======================================================================

2019-11-20 15:10:48,757	INFO node.py:483 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-11-20_15-10-48_4328/logs.
2019-11-20 15:10:48,758	INFO services.py:1427 -- Starting the Plasma object store with 5.03 GB memory using /dev/shm.
2019-11-20 15:10:48,923	INFO tune.py:64 -- Did not find checkpoint file in /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48.
2019-11-20 15:10:48,923	INFO tune.py:211 -- Starting a new experiment.
== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/1 GPUs
Memory usage on this node: 6.4/16.8 GB

== Status ==
Using FIFO scheduling algorithm.
Resources requested: 8/8 CPUs, 0/1 GPUs
Memory usage on this node: 6.4/16.8 GB
Result logdir: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48
Number of trials: 5 ({'RUNNING': 1, 'PENDING': 4})
PENDING trials:
 - 9fb14214-algorithm=RAQ-seed=7383:	PENDING
 - 8decac7c-algorithm=RAQ-seed=2376:	PENDING
 - 53f276a8-algorithm=RAQ-seed=6849:	PENDING
 - cd59f5d5-algorithm=RAQ-seed=867:	PENDING
RUNNING trials:
 - 384203e9-algorithm=RAQ-seed=3478:	RUNNING

(pid=4407) /home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.1) or chardet (3.0.4) doesn't match a supported version!
(pid=4407)   RequestsDependencyWarning)
(pid=4407) Warning: robosuite package not found. Run `pip install robosuite` to use robosuite environments.
(pid=4407) 
(pid=4407) WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
(pid=4407) For more information, please see:
(pid=4407)   * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
(pid=4407)   * https://github.com/tensorflow/addons
(pid=4407) If you depend on functionality not listed there, please file an issue.
(pid=4407) 
(pid=4407) 2019-11-20 15:10:51.510070: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
(pid=4407) 2019-11-20 15:10:51.516083: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
(pid=4407) 2019-11-20 15:10:51.516111: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:161] retrieving CUDA diagnostic information for host: desktop-in-the-corner
(pid=4407) 2019-11-20 15:10:51.516117: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:168] hostname: desktop-in-the-corner
(pid=4407) 2019-11-20 15:10:51.516152: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:192] libcuda reported version is: 410.48.0
(pid=4407) 2019-11-20 15:10:51.516171: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:196] kernel reported version is: 410.48.0
(pid=4407) 2019-11-20 15:10:51.516176: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version seems to match DSO: 410.48.0
(pid=4407) 2019-11-20 15:10:51.535151: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3998440000 Hz
(pid=4407) 2019-11-20 15:10:51.535673: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5593ac2cfe80 executing computations on platform Host. Devices:
(pid=4407) 2019-11-20 15:10:51.535695: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
(pid=4407) Using seed 3478
(pid=4407) WARNING: Logging before flag parsing goes to stderr.
(pid=4407) F1120 15:10:53.162240 140695482914240 core.py:90] GLEW initalization error: Missing GL version
(pid=4407) Fatal Python error: Aborted
(pid=4407) 
(pid=4407) Stack (most recent call first):
(pid=4407)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 841 in emit
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 863 in handle
(pid=4407)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 891 in handle
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1514 in callHandlers
(pid=4407)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1055 in handle
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1442 in _log
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1372 in log
(pid=4407)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1038 in log
(pid=4407)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 476 in log
(pid=4407)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 309 in fatal
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/dm_control/mujoco/wrapper/core.py", line 90 in _error_callback
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/mujoco_env.py", line 152 in initialize_camera
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/core/image_env.py", line 75 in __init__
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/__init__.py", line 324 in create_image_48_sawyer_door_pull_hook_v0
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 86 in make
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 125 in make
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 183 in make
(pid=4407)   File "/home/jedrzej/GitHub/reward-learning-rl/softlearning/environments/utils.py", line 48 in get_goal_example_environment_from_variant
(pid=4407)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/classifier_rl/main.py", line 30 in _build
(pid=4407)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/development/main.py", line 77 in _train
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trainable.py", line 151 in train
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/function_manager.py", line 783 in actor_method_executor
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 887 in _process_task
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 990 in _wait_for_and_process_task
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 1039 in main_loop
(pid=4407)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/workers/default_worker.py", line 98 in <module>
2019-11-20 15:10:53,287	ERROR trial_runner.py:494 -- Error processing event.
Traceback (most recent call last):
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 443, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 315, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 2193, in get
    raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2019-11-20 15:10:53,288	ERROR worker.py:1672 -- A worker died or was killed while executing task 00000000c1c94bc03ef5d18e0aaa637e52c77030.
2019-11-20 15:10:53,289	INFO ray_trial_executor.py:179 -- Destroying actor for trial 384203e9-algorithm=RAQ-seed=3478. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.
(pid=4500) /home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.1) or chardet (3.0.4) doesn't match a supported version!
(pid=4500)   RequestsDependencyWarning)
(pid=4500) Warning: robosuite package not found. Run `pip install robosuite` to use robosuite environments.
(pid=4500) 
(pid=4500) WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
(pid=4500) For more information, please see:
(pid=4500)   * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
(pid=4500)   * https://github.com/tensorflow/addons
(pid=4500) If you depend on functionality not listed there, please file an issue.
(pid=4500) 
(pid=4500) Using seed 7383
(pid=4500) 2019-11-20 15:10:55.430361: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
(pid=4500) 2019-11-20 15:10:55.436419: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
(pid=4500) 2019-11-20 15:10:55.436451: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:161] retrieving CUDA diagnostic information for host: desktop-in-the-corner
(pid=4500) 2019-11-20 15:10:55.436457: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:168] hostname: desktop-in-the-corner
(pid=4500) 2019-11-20 15:10:55.436492: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:192] libcuda reported version is: 410.48.0
(pid=4500) 2019-11-20 15:10:55.436512: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:196] kernel reported version is: 410.48.0
(pid=4500) 2019-11-20 15:10:55.436517: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version seems to match DSO: 410.48.0
(pid=4500) 2019-11-20 15:10:55.455115: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3998440000 Hz
(pid=4500) 2019-11-20 15:10:55.455574: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55eec40fce80 executing computations on platform Host. Devices:
(pid=4500) 2019-11-20 15:10:55.455595: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
(pid=4500) WARNING: Logging before flag parsing goes to stderr.
(pid=4500) F1120 15:10:57.092349 140252929529280 core.py:90] GLEW initalization error: Missing GL version
(pid=4500) Fatal Python error: Aborted
(pid=4500) 
(pid=4500) Stack (most recent call first):
(pid=4500)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 841 in emit
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 863 in handle
(pid=4500)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 891 in handle
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1514 in callHandlers
(pid=4500)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1055 in handle
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1442 in _log
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1372 in log
(pid=4500)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1038 in log
(pid=4500)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 476 in log
(pid=4500)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 309 in fatal
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/dm_control/mujoco/wrapper/core.py", line 90 in _error_callback
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/mujoco_env.py", line 152 in initialize_camera
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/core/image_env.py", line 75 in __init__
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/__init__.py", line 324 in create_image_48_sawyer_door_pull_hook_v0
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 86 in make
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 125 in make
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 183 in make
(pid=4500)   File "/home/jedrzej/GitHub/reward-learning-rl/softlearning/environments/utils.py", line 48 in get_goal_example_environment_from_variant
(pid=4500)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/classifier_rl/main.py", line 30 in _build
(pid=4500)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/development/main.py", line 77 in _train
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trainable.py", line 151 in train
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/function_manager.py", line 783 in actor_method_executor
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 887 in _process_task
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 990 in _wait_for_and_process_task
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 1039 in main_loop
(pid=4500)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/workers/default_worker.py", line 98 in <module>
2019-11-20 15:10:57,222	ERROR trial_runner.py:494 -- Error processing event.
Traceback (most recent call last):
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 443, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 315, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 2193, in get
    raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2019-11-20 15:10:57,223	INFO ray_trial_executor.py:179 -- Destroying actor for trial 9fb14214-algorithm=RAQ-seed=7383. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.
2019-11-20 15:10:57,226	ERROR worker.py:1672 -- A worker died or was killed while executing task 00000000ba4a3474bd990e2fd3ad6d1a4710733a.
== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/1 GPUs
Memory usage on this node: 6.5/16.8 GB
Result logdir: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48
Number of trials: 5 ({'ERROR': 2, 'PENDING': 3})
ERROR trials:
 - 384203e9-algorithm=RAQ-seed=3478:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/384203e9-algorithm=RAQ-seed=3478_2019-11-20_15-10-48haz3d1t1/error_2019-11-20_15-10-53.txt
 - 9fb14214-algorithm=RAQ-seed=7383:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/9fb14214-algorithm=RAQ-seed=7383_2019-11-20_15-10-53bov79p2k/error_2019-11-20_15-10-57.txt
PENDING trials:
 - 8decac7c-algorithm=RAQ-seed=2376:	PENDING
 - 53f276a8-algorithm=RAQ-seed=6849:	PENDING
 - cd59f5d5-algorithm=RAQ-seed=867:	PENDING

(pid=4403) /home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.1) or chardet (3.0.4) doesn't match a supported version!
(pid=4403)   RequestsDependencyWarning)
(pid=4403) Warning: robosuite package not found. Run `pip install robosuite` to use robosuite environments.
(pid=4403) 
(pid=4403) WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
(pid=4403) For more information, please see:
(pid=4403)   * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
(pid=4403)   * https://github.com/tensorflow/addons
(pid=4403) If you depend on functionality not listed there, please file an issue.
(pid=4403) 
(pid=4403) 2019-11-20 15:10:59.330464: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
(pid=4403) 2019-11-20 15:10:59.336562: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
(pid=4403) 2019-11-20 15:10:59.336589: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:161] retrieving CUDA diagnostic information for host: desktop-in-the-corner
(pid=4403) 2019-11-20 15:10:59.336595: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:168] hostname: desktop-in-the-corner
(pid=4403) 2019-11-20 15:10:59.336630: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:192] libcuda reported version is: 410.48.0
(pid=4403) 2019-11-20 15:10:59.336649: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:196] kernel reported version is: 410.48.0
(pid=4403) 2019-11-20 15:10:59.336654: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version seems to match DSO: 410.48.0
(pid=4403) Using seed 2376
(pid=4403) 2019-11-20 15:10:59.355170: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3998440000 Hz
(pid=4403) 2019-11-20 15:10:59.356824: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5571c899be80 executing computations on platform Host. Devices:
(pid=4403) 2019-11-20 15:10:59.356849: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
(pid=4403) WARNING: Logging before flag parsing goes to stderr.
(pid=4403) F1120 15:11:00.997669 140531732485568 core.py:90] GLEW initalization error: Missing GL version
(pid=4403) Fatal Python error: Aborted
(pid=4403) 
(pid=4403) Stack (most recent call first):
(pid=4403)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 841 in emit
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 863 in handle
(pid=4403)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 891 in handle
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1514 in callHandlers
(pid=4403)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1055 in handle
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1442 in _log
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1372 in log
(pid=4403)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1038 in log
(pid=4403)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 476 in log
(pid=4403)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 309 in fatal
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/dm_control/mujoco/wrapper/core.py", line 90 in _error_callback
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/mujoco_env.py", line 152 in initialize_camera
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/core/image_env.py", line 75 in __init__
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/__init__.py", line 324 in create_image_48_sawyer_door_pull_hook_v0
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 86 in make
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 125 in make
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 183 in make
(pid=4403)   File "/home/jedrzej/GitHub/reward-learning-rl/softlearning/environments/utils.py", line 48 in get_goal_example_environment_from_variant
(pid=4403)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/classifier_rl/main.py", line 30 in _build
(pid=4403)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/development/main.py", line 77 in _train
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trainable.py", line 151 in train
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/function_manager.py", line 783 in actor_method_executor
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 887 in _process_task
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 990 in _wait_for_and_process_task
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 1039 in main_loop
(pid=4403)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/workers/default_worker.py", line 98 in <module>
2019-11-20 15:11:01,121	ERROR trial_runner.py:494 -- Error processing event.
Traceback (most recent call last):
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 443, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 315, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 2193, in get
    raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2019-11-20 15:11:01,122	INFO ray_trial_executor.py:179 -- Destroying actor for trial 8decac7c-algorithm=RAQ-seed=2376. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.
2019-11-20 15:11:01,127	ERROR worker.py:1672 -- A worker died or was killed while executing task 00000000f21a19a109d06d8ceacb3983186c4952.
(pid=4409) /home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.1) or chardet (3.0.4) doesn't match a supported version!
(pid=4409)   RequestsDependencyWarning)
(pid=4409) Warning: robosuite package not found. Run `pip install robosuite` to use robosuite environments.
(pid=4409) 
(pid=4409) WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
(pid=4409) For more information, please see:
(pid=4409)   * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
(pid=4409)   * https://github.com/tensorflow/addons
(pid=4409) If you depend on functionality not listed there, please file an issue.
(pid=4409) 
(pid=4409) 2019-11-20 15:11:03.240185: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
(pid=4409) 2019-11-20 15:11:03.246380: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
(pid=4409) 2019-11-20 15:11:03.246413: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:161] retrieving CUDA diagnostic information for host: desktop-in-the-corner
(pid=4409) 2019-11-20 15:11:03.246419: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:168] hostname: desktop-in-the-corner
(pid=4409) 2019-11-20 15:11:03.246455: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:192] libcuda reported version is: 410.48.0
(pid=4409) 2019-11-20 15:11:03.246475: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:196] kernel reported version is: 410.48.0
(pid=4409) 2019-11-20 15:11:03.246480: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version seems to match DSO: 410.48.0
(pid=4409) 2019-11-20 15:11:03.271211: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3998440000 Hz
(pid=4409) 2019-11-20 15:11:03.271839: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55cb31502e80 executing computations on platform Host. Devices:
(pid=4409) 2019-11-20 15:11:03.271873: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
(pid=4409) Using seed 6849
(pid=4409) WARNING: Logging before flag parsing goes to stderr.
(pid=4409) F1120 15:11:04.917667 140652231005632 core.py:90] GLEW initalization error: Missing GL version
(pid=4409) Fatal Python error: Aborted
(pid=4409) 
(pid=4409) Stack (most recent call first):
(pid=4409)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 841 in emit
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 863 in handle
(pid=4409)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 891 in handle
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1514 in callHandlers
(pid=4409)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1055 in handle
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1442 in _log
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1372 in log
(pid=4409)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1038 in log
(pid=4409)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 476 in log
(pid=4409)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 309 in fatal
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/dm_control/mujoco/wrapper/core.py", line 90 in _error_callback
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/mujoco_env.py", line 152 in initialize_camera
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/core/image_env.py", line 75 in __init__
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/__init__.py", line 324 in create_image_48_sawyer_door_pull_hook_v0
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 86 in make
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 125 in make
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 183 in make
(pid=4409)   File "/home/jedrzej/GitHub/reward-learning-rl/softlearning/environments/utils.py", line 48 in get_goal_example_environment_from_variant
(pid=4409)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/classifier_rl/main.py", line 30 in _build
(pid=4409)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/development/main.py", line 77 in _train
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trainable.py", line 151 in train
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/function_manager.py", line 783 in actor_method_executor
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 887 in _process_task
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 990 in _wait_for_and_process_task
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 1039 in main_loop
(pid=4409)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/workers/default_worker.py", line 98 in <module>
2019-11-20 15:11:05,043	ERROR trial_runner.py:494 -- Error processing event.
Traceback (most recent call last):
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 443, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 315, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 2193, in get
    raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2019-11-20 15:11:05,044	INFO ray_trial_executor.py:179 -- Destroying actor for trial 53f276a8-algorithm=RAQ-seed=6849. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.
2019-11-20 15:11:05,047	ERROR worker.py:1672 -- A worker died or was killed while executing task 0000000011f7ea519b074068a3ee381ec07a9c5e.
== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/1 GPUs
Memory usage on this node: 6.5/16.8 GB
Result logdir: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48
Number of trials: 5 ({'ERROR': 4, 'PENDING': 1})
ERROR trials:
 - 384203e9-algorithm=RAQ-seed=3478:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/384203e9-algorithm=RAQ-seed=3478_2019-11-20_15-10-48haz3d1t1/error_2019-11-20_15-10-53.txt
 - 9fb14214-algorithm=RAQ-seed=7383:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/9fb14214-algorithm=RAQ-seed=7383_2019-11-20_15-10-53bov79p2k/error_2019-11-20_15-10-57.txt
 - 8decac7c-algorithm=RAQ-seed=2376:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/8decac7c-algorithm=RAQ-seed=2376_2019-11-20_15-10-57fbcmzv1m/error_2019-11-20_15-11-01.txt
 - 53f276a8-algorithm=RAQ-seed=6849:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/53f276a8-algorithm=RAQ-seed=6849_2019-11-20_15-11-013svgatwh/error_2019-11-20_15-11-05.txt
PENDING trials:
 - cd59f5d5-algorithm=RAQ-seed=867:	PENDING

(pid=4411) /home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.1) or chardet (3.0.4) doesn't match a supported version!
(pid=4411)   RequestsDependencyWarning)
(pid=4411) Warning: robosuite package not found. Run `pip install robosuite` to use robosuite environments.
(pid=4411) 
(pid=4411) WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
(pid=4411) For more information, please see:
(pid=4411)   * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
(pid=4411)   * https://github.com/tensorflow/addons
(pid=4411) If you depend on functionality not listed there, please file an issue.
(pid=4411) 
(pid=4411) 2019-11-20 15:11:07.190590: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
(pid=4411) 2019-11-20 15:11:07.197197: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
(pid=4411) 2019-11-20 15:11:07.197251: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:161] retrieving CUDA diagnostic information for host: desktop-in-the-corner
(pid=4411) 2019-11-20 15:11:07.197259: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:168] hostname: desktop-in-the-corner
(pid=4411) 2019-11-20 15:11:07.197314: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:192] libcuda reported version is: 410.48.0
(pid=4411) 2019-11-20 15:11:07.197344: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:196] kernel reported version is: 410.48.0
(pid=4411) 2019-11-20 15:11:07.197351: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version seems to match DSO: 410.48.0
(pid=4411) Using seed 867
(pid=4411) 2019-11-20 15:11:07.219099: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3998440000 Hz
(pid=4411) 2019-11-20 15:11:07.219622: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5558ada2ee80 executing computations on platform Host. Devices:
(pid=4411) 2019-11-20 15:11:07.219647: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
(pid=4411) WARNING: Logging before flag parsing goes to stderr.
(pid=4411) F1120 15:11:08.866851 140064546551232 core.py:90] GLEW initalization error: Missing GL version
(pid=4411) Fatal Python error: Aborted
(pid=4411) 
(pid=4411) Stack (most recent call first):
(pid=4411)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 841 in emit
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 863 in handle
(pid=4411)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 891 in handle
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1514 in callHandlers
(pid=4411)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1055 in handle
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1442 in _log
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/logging/__init__.py", line 1372 in log
(pid=4411)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 1038 in log
(pid=4411)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 476 in log
(pid=4411)   File "/home/jedrzej/.local/lib/python3.6/site-packages/absl/logging/__init__.py", line 309 in fatal
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/dm_control/mujoco/wrapper/core.py", line 90 in _error_callback
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/mujoco_env.py", line 152 in initialize_camera
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/core/image_env.py", line 75 in __init__
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/__init__.py", line 324 in create_image_48_sawyer_door_pull_hook_v0
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 86 in make
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 125 in make
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 183 in make
(pid=4411)   File "/home/jedrzej/GitHub/reward-learning-rl/softlearning/environments/utils.py", line 48 in get_goal_example_environment_from_variant
(pid=4411)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/classifier_rl/main.py", line 30 in _build
(pid=4411)   File "/home/jedrzej/GitHub/reward-learning-rl/examples/development/main.py", line 77 in _train
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trainable.py", line 151 in train
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/function_manager.py", line 783 in actor_method_executor
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 887 in _process_task
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 990 in _wait_for_and_process_task
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 1039 in main_loop
(pid=4411)   File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/workers/default_worker.py", line 98 in <module>
2019-11-20 15:11:08,994	ERROR trial_runner.py:494 -- Error processing event.
Traceback (most recent call last):
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 443, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 315, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 2193, in get
    raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
2019-11-20 15:11:08,995	ERROR worker.py:1672 -- A worker died or was killed while executing task 000000007bf1b17a5387bcd84e9b0cb85e2fd64e.
2019-11-20 15:11:08,996	INFO ray_trial_executor.py:179 -- Destroying actor for trial cd59f5d5-algorithm=RAQ-seed=867. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.
== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/8 CPUs, 0/1 GPUs
Memory usage on this node: 6.5/16.8 GB
Result logdir: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48
Number of trials: 5 ({'ERROR': 5})
ERROR trials:
 - 384203e9-algorithm=RAQ-seed=3478:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/384203e9-algorithm=RAQ-seed=3478_2019-11-20_15-10-48haz3d1t1/error_2019-11-20_15-10-53.txt
 - 9fb14214-algorithm=RAQ-seed=7383:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/9fb14214-algorithm=RAQ-seed=7383_2019-11-20_15-10-53bov79p2k/error_2019-11-20_15-10-57.txt
 - 8decac7c-algorithm=RAQ-seed=2376:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/8decac7c-algorithm=RAQ-seed=2376_2019-11-20_15-10-57fbcmzv1m/error_2019-11-20_15-11-01.txt
 - 53f276a8-algorithm=RAQ-seed=6849:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/53f276a8-algorithm=RAQ-seed=6849_2019-11-20_15-11-013svgatwh/error_2019-11-20_15-11-05.txt
 - cd59f5d5-algorithm=RAQ-seed=867:	ERROR, 1 failures: /home/jedrzej/ray_results/multiworld/mujoco/Image48SawyerDoorPullHookEnv-v0/2019-11-20T15-10-48-2019-11-20T15-10-48/cd59f5d5-algorithm=RAQ-seed=867_2019-11-20_15-11-05k6so04ax/error_2019-11-20_15-11-08.txt

Traceback (most recent call last):
  File "/home/jedrzej/anaconda3/envs/softlearning/bin/softlearning", line 11, in <module>
    load_entry_point('softlearning', 'console_scripts', 'softlearning')()
  File "/home/jedrzej/GitHub/reward-learning-rl/softlearning/scripts/console_scripts.py", line 202, in main
    return cli()
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/jedrzej/GitHub/reward-learning-rl/softlearning/scripts/console_scripts.py", line 71, in run_example_local_cmd
    return run_example_local(example_module_name, example_argv)
  File "/home/jedrzej/GitHub/reward-learning-rl/examples/instrument.py", line 228, in run_example_local
    reuse_actors=True)
  File "/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/tune.py", line 253, in run
    raise TuneError("Trials did not complete", errored_trials)
ray.tune.error.TuneError: ('Trials did not complete', [384203e9-algorithm=RAQ-seed=3478, 9fb14214-algorithm=RAQ-seed=7383, 8decac7c-algorithm=RAQ-seed=2376, 53f276a8-algorithm=RAQ-seed=6849, cd59f5d5-algorithm=RAQ-seed=867])

Any thoughts?

Package inconsistencies when installing according to requirements.txt

Hi Avi, @avisingh599

When I am installing the softlearning environment's packages according to the requirements.txt file (this is from the masters branch), I found the following warning about package inconsistency between dm_control and Python 3.6.5 and this stopped the Envs building process (I've also raised this on the softlearning GitHub and while waiting for reply just wondering if you've got any idea):

VICEClassifier training labels confusion

Hi,
I am confused with labels in vice classifer training.

Observations consist with

np.concatenate([negatives, positives]

and the labels are assigned with

        labels_batch = np.zeros((2*self._classifier_batch_size,2), dtype=np.int32)
        labels_batch[:self._classifier_batch_size, 0] = 1
        labels_batch[self._classifier_batch_size:, 1] = 1

and arguments of 'softmax_cross_entropy' are

        log_pi_log_p_concat = tf.concat([log_pi, log_p], axis=1)
        self._classifier_loss_t = tf.reduce_mean(
            tf.losses.softmax_cross_entropy(
                self._label_ph,
                log_pi_log_p_concat,
            )
        )

so,
the negative sample has output 'log_pi' with label 1,
the positive sample has output 'log_pi' with label 0;
the negative sample has classifer 'log_p' with label 0,
the positive sample has classifer 'log_p' with label 1;

Why 'log_pi' for negative sample is labelled 1?

ImportError: cannot import name 'constants'

Excuse me, I am confused about the above problem when I run the code:

softlearning run_example_local examples.classifier_rl --n_goal_examples 10 --t
ask=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 5 --n_epochs 300 --active_query_frequency 10

The whole error log is as follows:

/home/lab/anaconda3/envs/softlearning/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.1) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
Traceback (most recent call last):
  File "main.py", line 11, in <module>
    from softlearning.environments.utils import get_goal_example_environment_from_variant
  File "/home/lab/Github/reward-learning-rl/softlearning/environments/utils.py", line 8, in <module>
    from .adapters.dm_control_adapter import DmControlAdapter
  File "/home/lab/Github/reward-learning-rl/softlearning/environments/adapters/dm_control_adapter.py", line 6, in <module>
    from dm_control import suite
  File "/home/lab/Github/dm_control-0260f3effcfe2b0fdb25d9652dc27ba34b52d226/dm_control/suite/__init__.py", line 28, in <module>
    from dm_control.suite import acrobot
  File "/home/lab/Github/dm_control-0260f3effcfe2b0fdb25d9652dc27ba34b52d226/dm_control/suite/acrobot.py", line 24, in <module>
    from dm_control import mujoco
  File "/home/lab/Github/dm_control-0260f3effcfe2b0fdb25d9652dc27ba34b52d226/dm_control/mujoco/__init__.py", line 18, in <module>
    from dm_control.mujoco.engine import action_spec
  File "/home/lab/Github/dm_control-0260f3effcfe2b0fdb25d9652dc27ba34b52d226/dm_control/mujoco/engine.py", line 44, in <module>
    from dm_control.mujoco import index
  File "/home/lab/Github/dm_control-0260f3effcfe2b0fdb25d9652dc27ba34b52d226/dm_control/mujoco/index.py", line 92, in <module>
    from dm_control.mujoco.wrapper import util
  File "/home/lab/Github/dm_control-0260f3effcfe2b0fdb25d9652dc27ba34b52d226/dm_control/mujoco/wrapper/__init__.py", line 23, in <module>
    from dm_control.mujoco.wrapper import mjbindings
  File "/home/lab/Github/dm_control-0260f3effcfe2b0fdb25d9652dc27ba34b52d226/dm_control/mujoco/wrapper/mjbindings/__init__.py", line 24, in <module>
    from dm_control.mujoco.wrapper.mjbindings import constants
ImportError: cannot import name 'constants'

Actually, I have used all the requirements you mentioned and the version of dm_control is correct.
But there is one thing I need to mention is that
https://github.com/deepmind/dm_control.git@0260f3effcfe2b0fdb25d9652dc27ba34b52d226
need mujoco200 in its setup.py. So I wonder how you use this version and mujoco150 at the same time.

Thanks a lot!

dm_control+robosuite package not found, 'FeedforwardGaussianPolicy' object has no attribute '_Serializable__initialize'

Hi All, when running

softlearning run_example_local examples.classifier_rl
--n_goal_examples 10
--task=Image48SawyerDoorPullHookEnv-v0
--algorithm VICERAQ
--num-samples 5
--n_epochs 300
--active_query_frequency 10

I've experienced the following error:

1. dm_control package not found. (The thing is - I've already ran pip install git+https://github.com/deepmind/dm_control.git and it is shown on terminal that I'd successfully install it:

2. robosuite package not found (same as above, I've already ran pip3 install robosuite:

3. AttributeError: 'FeedforwardGaussianPolicy' object has no attribute '_Serializable__initialize':

FYI, my environment is:
MacOS Mojave version 10.14.1
Python 3.6.5
Gym 0.9.3
Mujoco: 200
mujoco-py 1.50.1.68

Let me know what have I missed/done wrong? Thank you very much!

Cannot Run the Classifier_rl Example

- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04 and 18.04
- Python version: 3.6
- Requirement.txt file from link
I'm trying to get one of the example module working but i keep getting this error.

(softlearning) seekaddo@ubuntu:~/DeepLearning/reward-learning-rl$ conda activate softlearning
(softlearning) seekaddo@ubuntu:~/DeepLearning/reward-learning-rl$ softlearning run_example_local examples.classifier_rl --n_goal_examples 10 --task=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 5 --n_epochs 300 --active_query_frequency 10
2020-12-11 06:19:09.321309: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/home/seekaddo/.mujoco/mujoco200/bin
2020-12-11 06:19:09.321377: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "/home/seekaddo/miniconda3/envs/softlearning/bin/softlearning", line 11, in <module>
    load_entry_point('softlearning', 'console_scripts', 'softlearning')()
  File "/home/seekaddo/DeepLearning/softlearning/softlearning/scripts/console_scripts.py", line 207, in main
    return cli()
  File "/home/seekaddo/miniconda3/envs/softlearning/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/seekaddo/miniconda3/envs/softlearning/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/seekaddo/miniconda3/envs/softlearning/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/seekaddo/miniconda3/envs/softlearning/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/seekaddo/miniconda3/envs/softlearning/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/seekaddo/DeepLearning/softlearning/softlearning/scripts/console_scripts.py", line 73, in run_example_local_cmd
    return run_example_local(example_module_name, example_argv)
  File "/home/seekaddo/DeepLearning/softlearning/examples/instrument.py", line 222, in run_example_local
    example_module = importlib.import_module(example_module_name)
  File "/home/seekaddo/miniconda3/envs/softlearning/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'examples.classifier_rl'

Conda installation issue

I run into following problem when installing conda environment:

Running command git clone -q https://github.com/hartikainen/mujoco-py.git /tmp/pip-req-build-pv0fx_b7
  Running command git checkout -q 29fcd26290c9417aef0f82d0628d29fa0dbf0fab
  fatal: reference is not a tree: 29fcd26290c9417aef0f82d0628d29fa0dbf0fab

But replacing git+https://github.com/hartikainen/mujoco-py.git@29fcd26290c9417aef0f82d0628d29fa0dbf0fab with git+https://github.com/hartikainen/mujoco-py.git in requirements.txt solves the issue.

EDIT: But afterwards when running the example code I get:

/home/jedrzej/anaconda3/envs/softlearning/lib/python3.6/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.1) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)

so it may be worth adjusting these dependencies.

Failed to initialize OpenGL

I find this but didn't help for me...
https://github.com/openai/mujoco-py/issues/187
My devire is ubuntu16, and has run FetchPickAndPlace-v1 from (penai/baselines) then it can run with env(softlearning)

Using seed 8179
2019-05-02 18:19:51.057481: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Registering multiworld mujoco gym environments
Choosing the latest nvidia driver: /usr/lib/nvidia-396, among ['/usr/lib/nvidia-390', '/usr/lib/nvidia-396']
Choosing the latest nvidia driver: /usr/lib/nvidia-396, among ['/usr/lib/nvidia-390', '/usr/lib/nvidia-396']
/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/logger.py:30: UserWarning: WARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.
warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
Registering goal example multiworld mujoco gym environments
Found 2 GPUs for rendering. Using device 0.
2019-05-02 18:19:53,330 ERROR trial_runner.py:412 -- Error processing event.
Traceback (most recent call last):
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 378, in _process_events
result = self.trial_executor.fetch_result(trial)
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 228, in fetch_result
result = ray.get(trial_future[0])
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/worker.py", line 2211, in get
raise value
ray.worker.RayTaskError: ray_ExperimentRunnerClassifierRL:train() (pid=6795, host=ivan-All-Series-Invalid-entry-length-16-Fixed-up-to-11)
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/ray/tune/trainable.py", line 151, in train
result = self._train()
File "/home/ivan/RL/reward-learning-rl/examples/development/main.py", line 66, in _train
self._build()
File "/home/ivan/RL/reward-learning-rl/examples/classifier_rl/main.py", line 29, in _build
env = self.env = get_goal_example_environment_from_variant(variant)
File "/home/ivan/RL/reward-learning-rl/softlearning/environments/utils.py", line 34, in get_goal_example_environment_from_variant
return GymAdapter(env=gym.make(variant['task']))
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 171, in make
return registry.make(id, **kwargs)
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 123, in make
env = spec.make(kwargs)
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/gym/envs/registration.py", line 84, in make
env = self._entry_point(_kwargs)
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/init.py", line 324, in create_image_48_sawyer_door_pull_hook_v0
non_presampled_goal_img_is_garbage=True,
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/core/image_env.py", line 75, in init
sim = self._wrapped_env.initialize_camera(init_camera)
File "/home/ivan/anaconda3/envs/softlearning/lib/python3.6/site-packages/multiworld/envs/mujoco/mujoco_env.py", line 152, in initialize_camera
viewer = mujoco_py.MjRenderContextOffscreen(sim, device_id=self.device_id)
File "mujoco_py/mjrendercontext.pyx", line 43, in mujoco_py.cymj.MjRenderContext.init
File "mujoco_py/mjrendercontext.pyx", line 108, in mujoco_py.cymj.MjRenderContext._setup_opengl_context
File "mujoco_py/opengl_context.pyx", line 128, in mujoco_py.cymj.OffscreenOpenGLContext.init
RuntimeError: Failed to initialize OpenGL

Docker installation not detect Mujoco key

Hi,
Thank you for provide source code, I'm trying to install your framework with docker image, I ran command:
export MJKEY="$(cat ~/.mujoco/mjkey.txt)" && docker-compose -f ./docker/docker-compose.dev.cpu.yml up -d --force-recreate

But when I go into container, and run command:
softlearning run_example_local examples.classifier_rl --n_goal_examples 10 --task=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 1 --n_epochs 300 --active_query_frequency 10

I face the error:

`WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons
  If you depend on functionality not listed there, please file an issue.

WARNING: Logging before flag parsing goes to stderr.
I0603 12:03:40.040023 140002190292736 acceleratesupport.py:13] OpenGL_accelerate module loaded
I0603 12:03:40.055296 140002190292736 arraydatatype.py:270] Using accelerated ArrayDatatype
I0603 12:03:40.331384 140002190292736 init.py:34] MuJoCo library version is: 200
Warning: robosuite package not found. Run pip install robosuite to use robosuite environments.
I0603 12:03:40.355057 140002190292736 init.py:333] Registering multiworld mujoco gym environments
F0603 12:03:41.536112 140002190292736 core.py:90] Invalid activation key
Aborted (core dumped)
`
I have tried to import another gym environment (Hopper-v2, FetchReach-v1, etc.) but it notifies "Invalid activation key". I think the docker image not recognize key. Is this problem caused by docker?

Error when setting up container

Hi,

when running

export MJKEY="$(cat ~/.mujoco/mjkey.txt)" \
    && docker-compose \
        -f ./docker/docker-compose.dev.cpu.yml \
        up \
        -d \
        --force-recreate

I'm getting the error:

  File "/usr/lib/python2.7/dist-packages/compose/service.py", line 13, in <module>
    from docker.utils import LogConfig
ImportError: cannot import name LogConfig

How can this be fixed?

Thanks!

Where is the sawyer environment defined?

Hi, you have used a customized environment in your simulation task. I quickly browsed the code and found nothing related to it. Did you open sourced this part of code?
By the way, parallel implementation broken ties and coherence of the code, which makes it harder to reproduce the experiment and understand the algorithm. Personally, I prefer a simpler version which can be run in a local PC.

Docker Installation

Hi,

The docker installation is not complete or the docker is not working. I would appreciate you put the complete installation here.

Thanks,

Run issue

When I run "softlearning run_example_local examples.classifier_rl --n_goal_examples 10 --task=Image48SawyerDoorPullHookEnv-v0 --algorithm VICERAQ --num-samples 5 --n_epochs 30 --active_query_frequency 10" . There is a problem:

**Traceback (most recent call last):

File "/home/wrf/anaconda3/bin/softlearning", line 6, in
from pkg_resources import load_entry_point
File "/home/wrf/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 3253, in
@_call_aside
File "/home/wrf/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 3237, in _call_aside
f(*args, kwargs)
File "/home/wrf/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 3266, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/home/wrf/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 584, in _build_master
ws.require(requires)
File "/home/wrf/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 901, in require
needed = self.resolve(parse_requirements(requirements))
File "/home/wrf/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 787, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'softlearning' distribution was not found and is required by the application

Python Segmentation fault in visual environments

Hello, I've tried to run Image48SawyerDoorPullHookEnv-v0 using the script:

softlearning run_example_local examples.classifier_rl  --n_goal_examples 10  --task Image48SawyerDoorPullHookEnv-v0  --algorithm VICERAQ  --num-samples 5   --n_epochs 300  --active_query_frequency 10

But got Fatal Python error: Segmentation fault , as the following :

(pid=9987) 2019-05-17 09:04:38.109309: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
(pid=9987) Using seed 5069
(pid=9987) 2019-05-17 09:04:38.152079: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
(pid=9987) 2019-05-17 09:04:38.152436: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55ac9957ce20 executing computations on platform Host. Devices:
(pid=9987) 2019-05-17 09:04:38.152461: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
(pid=9987) Fatal Python error: Segmentation fault
(pid=9987) 
(pid=9987) Stack (most recent call first):
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/multiworld/envs/mujoco/mujoco_env.py", line 154 in initialize_camera
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/multiworld/core/image_env.py", line 76 in __init__
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/multiworld/envs/mujoco/__init__.py", line 467 in create_image_48_sawyer_door_pull_hook_v0
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/gym/envs/registration.py", line 86 in make
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/gym/envs/registration.py", line 125 in make
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/gym/envs/registration.py", line 183 in make
(pid=9987)   File "/home/drl/PycharmProjects/ManipulationRL/reward-learning-rl/softlearning/environments/utils.py", line 49 in get_goal_example_environment_from_variant
(pid=9987)   File "/home/drl/PycharmProjects/ManipulationRL/reward-learning-rl/examples/classifier_rl/main.py", line 30 in _build
(pid=9987)   File "/home/drl/PycharmProjects/ManipulationRL/reward-learning-rl/examples/development/main.py", line 80 in _train
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/ray/tune/trainable.py", line 151 in train
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/ray/function_manager.py", line 783 in actor_method_executor
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/ray/worker.py", line 887 in _process_task
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/ray/worker.py", line 990 in _wait_for_and_process_task
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/ray/worker.py", line 1039 in main_loop
(pid=9987)   File "/home/drl/anaconda3/envs/softlearning_rf/lib/python3.6/site-packages/ray/workers/default_worker.py", line 98 in <module>
2019-05-17 09:04:40,670 ERROR trial_runner.py:494 -- Error processing event.

One thing I've noticed is that the envs without using camera in mujoco work well, like StateSawyerDoorPullHookEnv-v0. And it seems like that the error results from the following codes:
https://github.com/vitchyr/multiworld/blob/efb1b9e1bf42a2369f7f0a2434e78730e2dc60b9/multiworld/core/image_env.py#L74-L75

https://github.com/avisingh599/multiworld/blob/55573b45b878c38bc607164347122fc6d7a93d6b/multiworld/envs/mujoco/mujoco_env.py#L152

I guess that 'mujoco_py.MjRenderContextOffscreen' results in the multiprocessing error in the ray. So did you encounter that issue before? Could you give me some advice? Thanks.

VICEClassifier loss definition confusion

Hi,

In vice.py, the classifier loss defined as

     def _init_classifier_update(self):
        log_p = self._classifier([self._observations_ph])
        sampled_actions = self._policy.actions([self._observations_ph])
        log_pi = self._policy.log_pis([self._observations_ph], sampled_actions)
        log_pi_log_p_concat = tf.concat([log_pi, log_p], axis=1)

        self._classifier_loss_t = tf.reduce_mean(
            tf.losses.softmax_cross_entropy(
                self._label_ph,
                log_pi_log_p_concat,
            )
        )
        self._classifier_training_op = self._get_classifier_training_op()

When do classifier training, it seems the gradient will be applied to policy network? Is this the case?

In paper, it is said policy network trained with classifier's output as reward, but not updated when do classifier training. What am I missing ?

Thanks

Workflow recommendation for end to end learning

Hi,
Could you share with us some insights into how you approach the end to end learning development workflow?

Specifically, how do you deal with long iteration times for changes (few hours)?
Is there some deployment infrastructure (other than Ray, Docker, and git) in your group?
For example, do you try out one by one or do you submit a batch of changes and somehow a software reports it for you later on.

Cheers,
Shahram