improbable-ai / dribblebot

Code release accompanying DribbleBot: Dynamic Legged Manipulation in the Wild

Go1 Soccer (Dribblebot) Starter Kit

Table of contents

1. Overview
2. System Requirements
3. Training a Model
   1. Installation
   2. Environment and Model Configuration
   3. Training, Logging and Evaluation
4. Deploying a Model (Coming Soon)

Overview

This repository provides an implementation of the paper:

DribbleBot: Dynamic Legged Manipulation in the Wild
Yandong Ji*, Gabriel B. Margolis* and Pulkit Agrawal
International Conference on Robotics and Automation (ICRA), 2023
paper / bibtex / project page

This training code, environment and documentation build on Walk these Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior by Gabriel Margolis and Pulkit Agrawal, Improbable AI Lab, MIT (Paper: https://arxiv.org/pdf/2212.03238.pdf) and the Isaac Gym simulator from NVIDIA (Paper: https://arxiv.org/abs/2108.10470). All redistributed code retains its original license.

Our initial release provides the following features:

• Train reinforcement learning policies for the Go1 robot using PPO, IsaacGym, Domain Randomization to dribble a soccer ball in simulation following a random ball velocity command in global frame.
• Evaluate a pre-trained soccer policy in simulation.

System Requirements

Simulated Training and Evaluation: Isaac Gym requires an NVIDIA GPU. To train in the default configuration, we recommend a GPU with at least 10GB of VRAM. The code can run on a smaller GPU if you decrease the number of parallel environments (Cfg.env.num_envs). However, training will be slower with fewer environments.

Training a Model

Installation using Conda

Create a new conda environment with Python (3.8 suggested)

conda create -n dribblebot python==3.8
conda activate dribblebot

Install pytorch 1.10 with cuda-11.3:

pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio==0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

Install Isaac Gym

1. Download and install Isaac Gym Preview 4 from https://developer.nvidia.com/isaac-gym

2. unzip the file via:

   tar -xf IsaacGym_Preview_4_Package.tar.gz

3. now install the python package

   cd isaacgym/python && pip install -e .

4. Verify the installation by try running an example

   python examples/1080_balls_of_solitude.py

5. For troubleshooting check docs isaacgym/docs/index.html

Install the dribblebot package

In this repository, run pip install -e .

Evaluate the pre-trained policy

If everything is installed correctly, you should be able to run the evaluation script with:

python scripts/play_dribbling_pretrained.py

You should see a robot manipulate a yellow soccer following random global velocity commands.

Environment and Model Configuration

CODE STRUCTURE The main environment for simulating a legged robot is in legged_robot.py. The default configuration parameters including reward weightings are defined in legged_robot_config.py::Cfg.

There are three scripts in the scripts directory:

scripts
├── __init__.py
├── play_dribbling_custom.py
├── play_dribbling_pretrained.py
└── train_dribbling.py

Training, Logging and evaluation

To train the Go1 controller from Dribblebot, run:

python scripts/train_dribbling.py

After initializing the simulator, the script will print out a list of metrics every ten training iterations.

Training with the default configuration requires about 12GB of GPU memory. If you have less memory available, you can still train by reducing the number of parallel environments used in simulation (the default is Cfg.env.num_envs = 1000).

To visualize training progress, first set up weights and bias (wandb):

Set Up Weights and Bias (wandb):

Weights and Biases is the service that will provide you a dashboard where you can see the progress log of your training runs, including statistics and videos.

First, follow the instructions here to create you wandb account: https://docs.wandb.ai/quickstart

Make sure to perform the wandb.login() step from your local computer.

Finally, use a web browser to go to the wandb IP (defaults to localhost:3001)

To evaluate a pretrained trained policy, run play_dribbling_pretrained.py. We provie a pretrained agent checkpoint in the ./runs/dribbling directory.

Deploying a Model (Coming Soon)

We are working a modular version of the vision processing code so DribbleBot can be easily deployed on Go1. It will be added in a future release.

dribblebot's Issues

Stability issues

Hi,

Thank you for the amazing work!

While experimenting with your code, despite running the training multiple times, we're observing stability issues. Here is an example of one of the rew_total graphs:

Is this behavior expected or indicative of an underlying problem? Is the maximum total reward achieved here (around 350) the same as you got? Additionally, if you could share the graphs from one of your runs it might help us to track down the issue and understand the expected behavior.

Thanks!

A third-party sim-to-real deployment code

Hello, thanks for your impressive and outstanding work and open sourcing the training code.

I would like to share my sim-to-real deployment code here since the official deployment toolkit is still in preparation.

https://github.com/YouJiacheng/dribble-deploy

My code only uses 1 camera but can be extended to 2 cameras easily.

Reducing nem_envs doesn't reduce required resources

Hi,

I have a Nvidia Quadro p2000 GPU with 5GBs of memory with Ubuntu 18.04, CUDA 11.6 and driver version: 510.39.01.
I tried the training with num_envs = [16, 100, 500, 4000] and non of them is working with the following error:

internal error : PhysX Internal CUDA error. Simulation can not continue!
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 1459
Segmentation fault (core dumped)

It seems the resources are not enough and while reducing num_envs didn't solve the problem. Any help?

Note, I tried CUDA 11.3 as recommended by the README and a compatability error said that at least 11.4 is required. After trying 11.4 with pytorch 11.3, same error happened. So I decided to upgrade CUDA to 11.6 and the same for the pytorch (pip install torch==1.12.0+cu116 torchvision==0.13.0+cu116 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu116
).
Log.txt

_apply_drag_force() seems unused, and unstable reward curve during trainning

Question about _apply_drag_force() method and reward stability:

Hi, thanks for sharing the source code of such an interesting and impressive work!

I've been exploring the code, and I noticed something that raised a question. In the LeggedRobot class, there is an _apply_drag_force() method. However, it seems this method is actually unused during simulation, which is different from what's described in the original paper. Is the open-sourced code is a modified version or it is just because the force is forgotten to be applied in the code?

Additionally, when attempting to reproduce the results using the provided training script, I encountered some issues with the stability of the reward curve. Is this expected behavior, or is there something I might be missing?

Thank you for your attention to this issue.