Low performance ROS2 Humble Beta about ouster-ros

[Samahu]

@CollinNHays thanks for evaluating the ros2 driver, I mainly tested on the rolling distro and I would get 10 Hz for the 1024x10 lidar_mode but around ~17 Hz for the 1024x20.

Based on my own experience while testing the code, the ros2 topic hz did not seem to be very reliable. Isn't it a bit surprising that you have the /ouster/image_* publish at a rate of 10 Hz while the /ouster/points publishes at a rate of 3-5 Hz? After all the os_image node depends on os_cloud to receive the point cloud before it can publish the destaggered images.

I do suspect that the issue comes down to the underlying DDS implementation (we are currently using defaults). I am currently working on evaluating the performance of the different DDS vendors and we will have an update that provides some recommendation about which works best with our driver.

One thing you could currently try is to connect to the sensor through sensor.indepdenent.launch.xml instead of sensor.launch.xml the latter is a symbolic link to sensor.composite.launch.xml. The sensor.indepdenent.launch.xml launch the nodes as into separate processes. I have noticed different performance between the two launch files. The current implementation does not utilize message loaning so we may not be gaining much when running the nodes as a single process.

Again, thanks for evaluating the driver and for the feedback, and I will keep this ticket open until we have tested and addresses all performance related issues with the ros2 driver.

[headlee]

Have you adjusted your UDP kernel send/receive buffer sizes? I haven't had a chance to test this driver yet (and I'm on Windows 10), but I've had to increase the buffer size in the past for other high bandwidth applications. See this link for how to do it in Linux.

Looking forward to trying this out on Windows!

[CollinNHays]

@headlee I was using the default UDP buffer sizes. I followed that guide for modifying the UDP buffer sizes and did some testing switching between default and the increased buffer size, but did not find any differences in performance using this driver.

I'm looking forward to hearing about the results of @Samahu testing different DDS vendors.

[CollinNHays]

Thanks for quick response. Agreed, looking in RTQ Node Graph the os_image node subscribes to the /ouster/points topic, and since the image topics are publishing at 10Hz the data must be getting published on the topic correctly.

However, as you said the sensor.launch.xml uses message loaning between the os_sensor, os_cloud, and os_image nodes. This may be why the os_image node is able to receive all of the data on the /ouster/points topic successfully. When I use sensor.indepdenent.launch.xml the image topics begin publishing at the same ~4.5Hz rate as the /ouster/points topic.

This naturally leads to the ROS2 QOS options where I tried changing the reliability from BEST_EFFORT to RELIABLE.
Here are my findings.
I modified os_cloud_node.cpp to add qos.reliable(); between line 92 and 93.

/points publisher QOS	/points subscriber QOS	Hz Visible in RViz2
BEST_EFFORT	BEST_EFFORT	~4.5
RELIABLE	BEST_EFFORT	~4.5
RELIABLE	RELAIBLE	~10
BEST_EFFORT	RELIABLE	0 N/A

With the modified os_cloud_node.cpp running ros2 topic hz /ouster/points topic still shows the ~4.5Hz result, despite clearly updating at 10Hz in RViz2. This is likely because running ros2 topic info -v /ouster/points shows that ros2 topic hz uses a best effort subscriber.

While changing the QOS reliability to RELIABLE does solve the issue it doesn't take into account any issues that may arise from timing issues caused by the RELIABLE QOS. Since SensorDataQos uses BEST_EFFORT reliability by default I would expect it to be working better as is in this driver. As you said, this likely points towards the DDS implementation.

And it's not just this official driver, the unofficial Ouster ROS2 driver has performance issues (including incomplete scans) as well which are improved by setting the use_system_default_qos to true.

[Samahu]

@CollinNHays we have recently merged #146 which improves the performance and stability of the driver for ROS2. Would you able to update your clone and check out if this helps resolve this issue for you?

[Samahu]

@CollinNHays I believe the performance should be been substantially improved by the last update (mainly included in #146) based on my own assessment. - just make sure not to rely on sensor.independent.launch.xml. - As such I am going to consider the the issue as done but feel free to re-open the ticket is you think otherwise.