zhrmichael-logm / holo-dex

Official Implementation of Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

Authors: Sridhar Pandian Arunachalam, Irmak Guzey, Soumith Chintala, Lerrel Pinto

This repository contains the official implementation of Holo-Dex including the unity scripts for the VR application, robot demonstration collection pipeline, training scripts and deployment modules. The VR application's APK can be found here.

Robot Runs

Method

Holo-Dex consists of two phases: demonstration colleciton, which is performed in real-time with visual feedback to VR Headset, and demonstration-based policy learning, which can learn to solve dexterous tasks from a limited number of demonstrations.

Pipeline Installation and Setup

The pipeline requires ROS for Server-Robot communication. This Package uses the Allegro Hand and Kinova Arm controllers from DIME-Controllers. This implementation uses Realsense cameras and which require the librealsense API. Also, OpenCV is required for image compression and other purposes.

After installing all the prerequisites, you can install this pipeline as a package with pip:

pip install -e .

You can test if it has installed correctly by running import holodex from a python shell.

Install the VR Application in your Oculus Headset using the APK provided here. To setup the VR Application in your Oculus Headset and enter the robot server's IP address (should be in the same network). The following stream border color codes indicate the following:

• Green - the right hand keypoints are being streamed
• Blue - the left hand keypoints are being stream.
• Red - the stream is paused and gives access to the menu.

Running the teleop

To use the Holo-Dex teleop module, open the VR Application in your Oculus Headset. On the robot server side, start the controllers first followed by the following command to start the teleop:

python teleop.py

The Holo-Dex teleop configurations can be adjusted in configs/tracker/oculus.yaml. The robot camera configurations can be adjusted in configs/robot_camera.yaml.

The package also contains an 30 Hz teleop implementation of DIME and you can run it using the following command:

python teleop.py tracker=mediapipe tracker/cam_serial_num=<realsense_camera_serial_number>

Data

All our data can be found in this URL: https://drive.google.com/drive/folders/1PiuqYkG7O1sIxE7YewVkni6ohLuNY7vF?usp=sharing

To collect demonstrations using this framework, run the following command:

python record_data.py -n <demonstration_number>

To filter and process data from the raw demonstrations run the following command:

python extract_data.py

You can change the data extraction configurations in configs/demo_extract.yaml.

Training Neural Networks

You can train encoders using Self-Supervised methods such as BYOL, VICReg, SimCLR and MoCo. Use the following command to train a resnet encoder using the above mentioned SSL methods:

python train_ssl.py ssl_method=<byol|vicreg|simclr|mocov3>

The training configurations can be changed in configs/train_ssl.yaml.

You can also train:

• Behavior Cloning:

python train_bc.py encoder_gradient=true

• Behavior Cloning-Rep:

python train_bc.py encoder_gradient=false

The training configurations can be changed in configs/train_bc.yaml.

Deploying Models

This implementation can deploy BC, BC-Rep and INN (all visual) on the robot. To deploy BC or BC-Rep, use the following command:

python deploy.py model=BC task/bc.model_weights=<bc_model-weights>

To deploy INN use the following command:

python deploy.py model=VINN task/vinn.encoder_weights_path=<ssl_encoder_weights_path>

You can set a control loop instead of pressing the Enter key to get actions using the following command:

python deploy.py model=<BC|VINN> run_loop=true loop_frequency=<action_loop_frequency>

Customizing the VR Application

To use Holo-Dex's VR side source code vr/Holo-Dex, you need to install Unity along with other dependencies which can be downloaded through Nuget:

• Oculus SDK
• NetMQ
• TextMeshPro

Citation

If you use this repo in your research, please consider citing the paper as follows:

@article{arunachalam2022holodex,
  title={Holo-Dex: Teaching Dexterity with Immersive Mixed Reality},
  author={Sridhar Pandian Arunachalam and Irmak Guzey and Soumith Chintala and Lerrel Pinto},
  journal={arXiv preprint arXiv:2210.06463},
  year={2022}
}