chri-lab / ros2_vision

AI-RedBack-Vision

Overview

The aim of this repo is to develop the visual components of the AI-RedBack project. The code should allow the robot to perceive and interpret its surroundings through cameras, enabling it to make informed decisions, through the identification of objects and human gestures.

ROS2 Nodes

There are three ROS nodes in this repository:

Feature	ROS2 Node
2D Object Detection (YOLO Object Detection)	yolo_detector
3D Object Detection (Distance Estimation)	realsense_depth_estimator
Human Posture Recognition Face Tracking	openpose_node

Installation

Step 1: Install Ubuntu 22.04 LTS (Jammy Jellyfish)

• Head to Ubuntu release page and download the “Desktop Image”
• Can either install via
  • Bootable USB flash drive
  • Virtual Machine
  • WSL

Step 2: Configure ROS2 environment

• Open the official documentation of ROS Humble for installation instructions

Step 3: Install required packages for object detection

• Please use pip to install the following packages

ultralytics >= 8.0

Step 4: Install required packages for distance estimation

• Please use pip to install the following packages

pyrealsense >= 2.54

• Please note that pyrealsense doesn't support Linux with ARM architecture

Step 5: Install required packages for human posture detection

• Please install the following packages

CUDA: https://developer.nvidia.com/cuda-11-7-1-download-archive
cuDNN: https://developer.nvidia.com/cudnn
Caffe prerequisites: Openpose repo -> ./scripts/ubuntu/install_deps.sh

• Please use apt-get to install the following packages

cmake-qt-gui
libopencv-dev
python3-dev

• Please use apt to install the following packages

protobuf-compiler
libgoogle-glog-dev
libboost-all-dev
libhdf5-dev
libatlas-base-dev

• Please use pip to install the following packages

numpy
opencv-python

Usage

Build

$ cd franka_ws
$ colcon build --symlink-install
$ source install/setup.sh

$ git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose
$ cd openpose
$ git submodule update --init --recursive --remote
$ mkdir build
$ cd build
$ cmake-gui ..
# Click configure, tick BUILD_PYTHON flag, and click configure again. Click generate.
$ make -j`nproc`

Run yolo_detector node

After successfully built, yolo_detector ROS2 node can be launched via the following command:

ros2 run object_detector yolo_detector

Details about yolo_detector can be found in docs

Run realsense_depth_estimator node

Similarly, realsense_depth_estimator ROS2 node can be launched via the following command:

ros2 run depth_estimator realsense_depth_estimator

Details about realsense_depth_estimator can be found in docs

Run openpose

After being built, openpose can be run with the following command:

./build/examples/openpose/openpose.bin 

Details about openpose can be found in docs

Camera Mounting

As mentioned on the page Depth Camera Mounting Analysis, a custom 3D-printed model needs to be designed and built to mount the depth camera on the end-effector of the robotic arm. We found there was another team working with computer vision solutions using both Franka Emika robotic arm and Intel Realsense depth camera released the STL file of the CAD model of a 3D printed part "that allows to attach an Intel RealSense D435 camera to the Panda flange".

The original model file was downloaded and rescaled to match the actual dimensions of the depth camera. The modified model file has been uploaded here. The process of analysing and building the 3D printed mounting component has been documented in detail in this PDF. This camera mounting model can be built using a 3D printer in the lab with the configurations specified in the documentation.

File Structure

.
├── docs
│   ├── camera_mounting
│   └── images
├── franka_ws
│   ├── build
│   ├── install
│   ├── log
│   └── src
│       ├── depth_estimator
│       └── object_detector
└── openpose_ws
    ├── build
    ├── install
    ├── log
    └── src
        └── openpose_controller

Contributors

Thanks goes to these wonderful people (emoji key):

jackson-hu1279 💻 📖

Tempest371 💻 📖

This project follows the all-contributors specification. Contributions of any kind welcome!