The aim of this repo is to develop the visual components of the AI-RedBack project. The code should allow the robot to perceive and interpret its surroundings through cameras, enabling it to make informed decisions, through the identification of objects and human gestures.
There are three ROS nodes in this repository:
Feature | ROS2 Node |
---|---|
2D Object Detection (YOLO Object Detection) |
yolo_detector |
3D Object Detection (Distance Estimation) |
realsense_depth_estimator |
Human Posture Recognition Face Tracking |
openpose_node |
Step 1: Install Ubuntu 22.04 LTS (Jammy Jellyfish)
- Head to Ubuntu release page and download the “Desktop Image”
- Can either install via
- Bootable USB flash drive
- Virtual Machine
- WSL
Step 2: Configure ROS2 environment
- Open the official documentation of ROS Humble for installation instructions
Step 3: Install required packages for object detection
- Please use
pip
to install the following packages
ultralytics >= 8.0
Step 4: Install required packages for distance estimation
- Please use
pip
to install the following packages
pyrealsense >= 2.54
- Please note that
pyrealsense
doesn't support Linux with ARM architecture
Step 5: Install required packages for human posture detection
- Please install the following packages
CUDA: https://developer.nvidia.com/cuda-11-7-1-download-archive
cuDNN: https://developer.nvidia.com/cudnn
Caffe prerequisites: Openpose repo -> ./scripts/ubuntu/install_deps.sh
- Please use
apt-get
to install the following packages
cmake-qt-gui
libopencv-dev
python3-dev
- Please use
apt
to install the following packages
protobuf-compiler
libgoogle-glog-dev
libboost-all-dev
libhdf5-dev
libatlas-base-dev
- Please use
pip
to install the following packages
numpy
opencv-python
$ cd franka_ws
$ colcon build --symlink-install
$ source install/setup.sh
$ git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose
$ cd openpose
$ git submodule update --init --recursive --remote
$ mkdir build
$ cd build
$ cmake-gui ..
# Click configure, tick BUILD_PYTHON flag, and click configure again. Click generate.
$ make -j`nproc`
After successfully built, yolo_detector
ROS2 node can be launched via the following command:
ros2 run object_detector yolo_detector
Details about yolo_detector
can be found in docs
Similarly, realsense_depth_estimator
ROS2 node can be launched via the following command:
ros2 run depth_estimator realsense_depth_estimator
Details about realsense_depth_estimator
can be found in docs
After being built, openpose can be run with the following command:
./build/examples/openpose/openpose.bin
Details about openpose
can be found in docs
As mentioned on the page Depth Camera Mounting Analysis, a custom 3D-printed model needs to be designed and built to mount the depth camera on the end-effector of the robotic arm. We found there was another team working with computer vision solutions using both Franka Emika robotic arm and Intel Realsense depth camera released the STL file of the CAD model of a 3D printed part "that allows to attach an Intel RealSense D435 camera to the Panda flange".
The original model file was downloaded and rescaled to match the actual dimensions of the depth camera. The modified model file has been uploaded here. The process of analysing and building the 3D printed mounting component has been documented in detail in this PDF. This camera mounting model can be built using a 3D printer in the lab with the configurations specified in the documentation.
.
├── docs
│ ├── camera_mounting
│ └── images
├── franka_ws
│ ├── build
│ ├── install
│ ├── log
│ └── src
│ ├── depth_estimator
│ └── object_detector
└── openpose_ws
├── build
├── install
├── log
└── src
└── openpose_controller
Thanks goes to these wonderful people (emoji key):
jackson-hu1279 💻 📖 |
Tempest371 💻 📖 |
This project follows the all-contributors specification. Contributions of any kind welcome!