alpaca-zip / ultralytics_ros

ROS/ROS 2 package for Ultralytics YOLOv8 real-time object detection and segmentation. https://github.com/ultralytics/ultralytics

ultralytics_ros's Issues

ERROR: Cannot load message class for [ultralytics_ros/YoloResult]. Are your messages built?

when i ehco the /yolo_result ERROR: Cannot load message class for [ultralytics_ros/YoloResult]. Are your messages built?

Any journal references available for this implementation?

Hello,

This really a great work.
I would like to understand the background of it, especially the projection of the pointcloud to the 2d image space and conversion of the 2d detection pixels to 3d coordinates. Is there any journal or article that you referred to?

How to detect camera information in real time

作者你好，我使用gazebo搭建了一个环境，并且在我的车体模型上安装了深度相机插件，我想实现识别结果以视频的形式展示出来，随着摄像头检测到画面的变化而不断变化，但是当我运行了readme里面指定的launch文件后，我只得到了一张识别结果图片，并且rviz显示无响应，当我关闭后控制台报错如图所示

我想知道我应该怎么配置或者编写launch文件才可以实现我想要的效果？还有我该怎么训练自己的模型？

No colcon_ws in alpacazip/ultralytics_ros:humble

Since the desktop was changed to Ubuntu Mate with the following PR, colcon_ws is not built correctly. Also, ultralytics_ros is not present in ros2 pkg list.
Tiryoh/docker-ros2-desktop-vnc#92

Temporary solution: use tiryoh/ros2-desktop-vnc:humble before June 17, 2023.

PinholeCameraModel not initialized

Branch

noetic-devel

Bug

To my understanding, the issue occurs because in the tracker_with_cloud_node.cpp the cloud2TransformedCloud function asks for the cam_model_ frame, while it has not been initialized yet.

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

Can you give me a rosbag download site?

syncCallback not receiving topic information with sensor_msgs::PointCloud2ConstPtr

Branch

noetic-devel

Question

there is isue something with message filter , syncCallback does not receive data, when I remove the PointCloud2 from the the system syncCallback working, there is isue with pcl data , I try everything , I increas que size and I try differen sync policies nothing changing, I also want to say my PointCloud2 data came form RGBD camera not lidar , but message type same. when I create seperate calback everything working, is it possioble that related rgbd data resulation high that create problem I don't know. can you give some advice .
I also want to say to you thank you for doing this great package and share with community , you are doing awsome job.
I also ask you when using this package . lidar and camera should located closed or when we add with tf some 1.5 or 2 meter offest it is will be okey ? in code I see algorithm take account ros tf.

Additional

No response

coordinate system

Branch

melodic-devel

Question

This work is significant, how did you get the coordinate system between the camera and the radar, and how do I proceed if I want to modify the coordinate relationship based on your work? Thank you for your busy reply!

Additional

No response

Ensure BGR Encoding Compliance for Images Passed to track Function

Quoted from: #15

The current implementation I've written does not explicitly handle the encoding of the subscribed topics; it simply passes the images to the track function as a numpy array. However, as per the official Ultralytics documentation, images passed to this function are expected to be in BGR encoding.

Tasks

• melodic-devel
• noetic-devel
• humble-devel

Dockerfile for Humble version

Branch

humble-devel

Question

I have build a sample Dockerfile for Humble version, but it has some errors .
Do you have any Dockerfile for Humble version ?

this is mine :

FROM --platform=linux/amd64 ros:humble-ros-base

ENV repo=/root/ros2_ws/src/Ros2_Yolov8
ENV wd=/root/ros2_ws
ENV ROS_DISTRO=humble

RUN mkdir -p $repo
WORKDIR $wd

COPY . $repo

# Set Environment Variables
ENV DEBIAN_FRONTEND noninteractive


# Add ROS 2 package repositories
RUN apt-get update && apt-get install -y software-properties-common && \
    add-apt-repository universe && \
    apt-add-repository restricted && \
    apt-add-repository multiverse && \
    apt-get update

# Install required packages
RUN apt-get update && apt-get upgrade -y && \
    apt-get install --no-install-recommends -y \
        python3-pip \
        ros-$ROS_DISTRO-ament-cmake \
        ros-$ROS_DISTRO-rclpy \
        ros-$ROS_DISTRO-sensor-msgs \
        ros-$ROS_DISTRO-vision-msgs \
        ros-$ROS_DISTRO-cv-bridge \
        ros-$ROS_DISTRO-image-geometry \
        ros-$ROS_DISTRO-image-transport \
        ros-$ROS_DISTRO-pcl-ros \
        ros-$ROS_DISTRO-image-view \
        python3-rosdep \
        python-is-python3 \
        ffmpeg libsm6 libxext6 && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN rosdep fix-permissions && \
    rosdep update --include-eol-distros && \
    rosdep install -r -y -i --from-paths . && \
    python3 -m pip install -r $repo/requirements.txt && \
    /bin/bash -c "source /opt/ros/$ROS_DISTRO/setup.bash && colcon build"


# Make the entrypoint script executable
RUN chmod +x $repo/docker/entrypoint.sh

# Set the entrypoint to the script
ENTRYPOINT ["/root/ros2_ws/src/Ros2_Yolov8/docker/entrypoint.sh"]
#CMD ["ros2", "launch", "ultralytics_ros", "tracker.launch.xml", "debug:=true"]

Additional

No response

Improvement of processing speed

Branch

noetic-devel

Description

The two functions that take the most time are projectCloud() and downsampleCloudMsg(), both of which seem to take similar amounts of time. While I can't think of any way to improve the speed of downsampleCloudMsg(), it seems that projectCloud() could benefit from parallel processing with OpenMP.

Additional

Within projectCloud(), the most time consuming processes are processPointsWithBbox() and processPointsWithMask(), with euclideanClusterExtraction() taking about a fifth of the time, which is less than I originally thought.

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

Unable to install rosbag/ros2 bag files

Branch

melodic-devel, noetic-devel, humble-devel

Bug

It seems that bag files are not being downloaded correctly from google drive as shown below. The same issue appears in noetic-devel and melodic-devel.

#10 [6/6] RUN cd ~/ &&     wget --load-cookies /tmp/cookies.txt "https://drive.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://drive.google.com/uc?export=download&id=1kN8boJxfMIl7hjytxdgmzANZBhWV7YXU' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1kN8boJxfMIl7hjytxdgmzANZBhWV7YXU" -O kitti_2011_09_26_drive_0106_synced.zip &&     rm -rf /tmp/cookies.txt &&     unzip kitti_2011_09_26_drive_0106_synced.zip &&     rm kitti_2011_09_26_drive_0106_synced.zip
#10 0.449 --2024-01-18 14:26:57--  https://drive.google.com/uc?export=download&confirm=&id=1kN8boJxfMIl7hjytxdgmzANZBhWV7YXU
#10 0.451 Resolving drive.google.com (drive.google.com)... 142.250.189.14, 2607:f8b0:4007:815::200e
#10 0.453 Connecting to drive.google.com (drive.google.com)|142.250.189.14|:443... connected.
#10 0.481 HTTP request sent, awaiting response... 303 See Other
#10 0.556 Location: https://drive.usercontent.google.com/download?id=1kN8boJxfMIl7hjytxdgmzANZBhWV7YXU&export=download [following]
#10 0.556 --2024-01-18 14:26:57--  https://drive.usercontent.google.com/download?id=1kN8boJxfMIl7hjytxdgmzANZBhWV7YXU&export=download
#10 0.556 Resolving drive.usercontent.google.com (drive.usercontent.google.com)... 142.250.72.161, 2607:f8b0:4007:803::2001
#10 0.589 Connecting to drive.usercontent.google.com (drive.usercontent.google.com)|142.250.72.161|:443... connected.
#10 0.617 HTTP request sent, awaiting response... 200 OK
#10 0.815 Length: 2450 (2.4K) [text/html]
#10 0.815 Saving to: 'kitti_2011_09_26_drive_0106_synced.zip'
#10 0.815 
#10 0.815      0K ..                                                    100% 14.9M=0s
#10 0.816 
#10 0.816 2024-01-18 14:26:57 (14.9 MB/s) - 'kitti_2011_09_26_drive_0106_synced.zip' saved [2450/2450]
#10 0.816 
#10 0.820 Archive:  kitti_2011_09_26_drive_0106_synced.zip
#10 0.820   End-of-central-directory signature not found.  Either this file is not
#10 0.820   a zipfile, or it constitutes one disk of a multi-part archive.  In the
#10 0.820   latter case the central directory and zipfile comment will be found on
#10 0.820   the last disk(s) of this archive.
#10 0.820 unzip:  cannot find zipfile directory in one of kitti_2011_09_26_drive_0106_synced.zip or
#10 0.820         kitti_2011_09_26_drive_0106_synced.zip.zip, and cannot find kitti_2011_09_26_drive_0106_synced.zip.ZIP, period.
#10 ERROR: process "/bin/bash -c cd ~/ &&     wget --load-cookies /tmp/cookies.txt \"https://drive.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://drive.google.com/uc?export=download&id=1kN8boJxfMIl7hjytxdgmzANZBhWV7YXU' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1kN8boJxfMIl7hjytxdgmzANZBhWV7YXU\" -O kitti_2011_09_26_drive_0106_synced.zip &&     rm -rf /tmp/cookies.txt &&     unzip kitti_2011_09_26_drive_0106_synced.zip &&     rm kitti_2011_09_26_drive_0106_synced.zip" did not complete successfully: exit code: 9

Additional

I think it would be easier to maintain the bag files directly using Git LFS.

---

Tasks

• melodic-devel
• noetic-devel
• humble-devel

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

Issue with Displaying Object Names in YOLO Detection Results

Branch

noetic-devel

Question

Both /camera/image_raw and /yolo_image nodes are displaying the correct images. However, when I check with the /yolo_results node, it shows the results in integer format. I was actually expecting image information along with the detected object names. Below, I've copied my output results for reference:

Additional

Class ID: 0.0, Class Name: insert, Score: 0.9874143004417419, BBox Center: (1854.5810546875, 962.247802734375)
Class ID: 3.0, Class Name: screw, Score: 0.8346330523490906, BBox Center: (1853.497314453125, 959.9049072265625)
Class ID: 1.0, Class Name: no_insert, Score: 0.7629039287567139, BBox Center: (761.804443359375, 1758.652099609375)
Class ID: 1.0, Class Name: no_insert, Score: 0.6941928267478943, BBox Center: (1356.8814697265625, 323.90386962890625)
Class ID: 3.0, Class Name: screw, Score: 0.6525463461875916, BBox Center: (1717.115966796875, 1096.2509765625)
Class ID: 0.0, Class Name: insert, Score: 0.9875774383544922, BBox Center: (1855.050537109375, 962.9561767578125)
Class ID: 3.0, Class Name: screw, Score: 0.8218352794647217, BBox Center: (1853.400634765625, 959.7418823242188)

Yolo_result echo output : 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ^Z

3D bounding box displaying problem

Branch

noetic-devel

Question

Hello,
Thank you for open sourcing your great work. I attempted to reproduce the object detection results using a point cloud (3D object detection). I downloaded the same kitti dataset (2011_09_26_drive_0106_sync) and run the "kitti_tracker_with_cloud.launch". 2D object detection with and without segmentation is fine, and there is no error. However, I could not see any 3D bounding boxes on the point cloud. This is illustrated in the attached screenshots. Could you please let me know what could be a possible solution to this problem?
Note: I tried all–yolov8 models (yolov8n-seg. pt, yolov8m-seg.pt, etc.) and also increases voxel_leaf_size, as suggested, but nothing works.

Here's my working configuration:

• Operating System: Ubuntu 20.04
• Hardware: Nvidia Agx_orin
• ROS Version: ROS Noetic

No Image Output

Hello @Alpaca-zip and thank you for sharing your project.

Could you provide more direction on how someone should get started with your project?
I am not sure if I am doing something wrong on my side, but I cannot get an output image produced.
I am using a webcam.

I started:

1. roslaunch ultralytics_ros tracker.launch debug:=true
2. I started RVIZ and subscribed to image topic.

I do not receive an output image from any of the subscribed topics. RVIZ says "no image received."

Thank you for your help.

Running on GPU

Is it possible to make the model run on GPU instead of CPU?

fatal error: ultralytics_ros/YoloResult.h: No such file or directory

When I catkin_make. I was prompted with
fatal error: ultralytics_ros/YoloResult.h: No such file or directory. and I didn't find the YoloResult.h file, where is it please?

AttributeError: module 'numpy' has no attribute 'float'.

hello Alpaca-zip,
Thank you for your program.
I encountered the following error when running tracker_node, it seems that ros_numpy is not compatible.
Looking forward to your response.

Error:
Traceback (most recent call last):
File "/home/gh/v8_0.2/devel/lib/v8/track.py", line 15, in
exec(compile(fh.read(), python_script, 'exec'), context)
File "/home/gh/v8_0.2/src/v8/src/track.py", line 3, in
import ros_numpy
File "/opt/ros/noetic/lib/python3/dist-packages/ros_numpy/init.py", line 7, in
from . import point_cloud2
File "/opt/ros/noetic/lib/python3/dist-packages/ros_numpy/point_cloud2.py", line 224, in
def get_xyz_points(cloud_array, remove_nans=True, dtype=np.float):
File "/home/gh/.local/lib/python3.8/site-packages/numpy/init.py", line 305, in getattr
raise AttributeError(former_attrs[attr])
AttributeError: module 'numpy' has no attribute 'float'.
np.float was a deprecated alias for the builtin float. To avoid this error in existing code, use float by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.float64 here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

Pose estimation

cannot see data when subscribing to yolo_result (ROS Melodic)

Branch

melodic-devel

Question

I am using Ubuntu 18,04 with ROS Melodic.
I am launching "roslaunch ultralytics_ros predict_with_cloud_node.launch".
The object are correctly detected and I can extract 2D/3D bounding boxes.
The problem arise when I launch "rostopic echo /yolo_result".
When I do that I can see all the data inside the message, but if I do a subscription to the topic "/yolo_result" through a ROS node it doesn't show anything.
I need those exact data because I need the bounding boxes in int format, while in the topic "/yolo_3d_result" they are almost close to 0.
Thanks in advance for your help.

Additional

No response

Can this ROS package be run under ROS1 (ROS melodic)?

Can this ROS package be run under ROS1 (ROS melodic)? Since ROS melodic depends on python 2.7, but yolov8 has python >= 3.8, I use $ roslaunch ultralytics_ros tracker.launch debug:=true and get ModuleNotFoundError: No module named 'ultralytics'. Is it possible to run this project under ROS melodic and if so what should I do?

[melodic-devel] Unable to install lap

In Ubuntu-18.04, there is a problem that lap cannot be installed as follows.

$ python3 -m pip install lap
ERROR: Could not build wheels for lap, which is required to install pyproject.toml-based projects

At least the pip, numpy, and setuptools version changes did not solve these problems. Therefore, modify [melodic-devel] to disable the use of lap.

If anyone is familiar with this issue, please reopen it and offer a solution.

The relevant isuue is:
https://github.com/ultralytics/ultralytics/issues?q=lap

Github Action Error

Branch

noetic-devel

Bug

https://github.com/Alpaca-zip/ultralytics_ros/actions/runs/7881844137?pr=58

ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device: '/tmp/pip-unpacked-wheel-38i3wtow/torch/lib/libtorch_python.so' -> '/usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so'

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

Can I obtain track id of an object?

Branch

noetic-devel

Question

I want to know track id for use this information but i wasn't take it. Could you give me a trick about that?

Additional

No response

foxy compatible

Branch

humble-devel

Question

Hello .
does this repository works with Foxy version ?

Additional

No response

Adding excluded points publisher in `track_with_cloud_node`.

Branch

noetic-devel

Description

Thanks to sharing great project.

I've noticed that the tracker_with_cloud node currently only publishes /detection_points. However, it would be highly beneficial to also have it publish excluded_points, which would be the original Point Cloud excluding the /detection_points.

For example, when doing Mapping or Localization, we may need a point cloud excluding people and cars. I think excluded_points would be useful in such cases.

Additional

I've implemented the publisher in my fork of the project. If you're interested, I'd be happy to send over a PR. Just let me know!

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

tracker_with_cloud_node not returnn 3D bounding box on topic /detection3d_result

Dear Alpaca,
Thank for you project, I am using your results. However, when I run for tracker_clound ... I can not get the value 3D bounding box of topic /detection3d_topic. Please check about your project this issue. Thank you so much.
"Detected objects(3D bounding box) to detection3d_topic parameter. (vision_msgs/Detection3DArray)"