Cyclone DDS hangs in Galactic within a VM with all defaults using ros2cli about rmw_cyclonedds

[eboasson]

Looks to me like you have a firewall blocking all multicast traffic. This causes two problems:

• Discovery no longer works (several other DDS implementations always cater for unicast discovery, but Cyclone by default doesn't if the interface supports multicast);
• Some threads are dedicated to receiving data from a single socket, and on shutdown those need to unblocked. The best mechanism I have been able to find for that is sending a packet to the socket, but that does require the IP stack to cooperate.

I suspect the hanging is caused by ROS 2's signal handler being "too nice" if the impossible happens and stopping cleanly turns out to be impossible.

I'd suggest allowing multicast, but you can also disable multicast altogether (it is just that you lose out on a lot of niceties and add a significant amount of overhead):

<General>
  <AllowMulticast>false</AllowMulticast>
</General>
<Discovery>
  <ParticipantIndex>auto</ParticipantIndex>
  <Peers>
    <Peer address="localhost"/>
  </Peers>
</Discovery>

The hanging you could fix by not having threads dedicated to receiving data on a specific socket (Internal/MultipleReceiveThreads = false), but that alone won't get make discovery work. And both options above that should make discovery work should also solve the hanging.

[eboasson]

I am afraid I don't (won't) have an Apertis 2022 SDK at hand. It certainly sounds like that is exhibiting some behaviour that interacts badly with Cyclone. Unusual, but not impossible.

What I usually suggest is to gather some tracing information from Cyclone DDS because that usually gives some insights in what is going on when everything works but there is no communication. Enabling it is as simple as putting

CYCLONEDDS_URI="<Tr><V>finest</><Out>cdds.log.\${CYCLONEDDS_PID}</>"

in the environment. That log starts with the configuration options, then you get some information on network selection and so on.

It is not so likely to give a clue as to why it hangs. For that, generally the best thing to do is to attach gdb and get stack traces for all threads (thread apply all bt) and look for something suspicious.

[AlexisTM]

For the following logs, no other node was running.

cdds.log.1189.txt

cdds.log.1190.txt

[AlexisTM]

That seems to be the problem indeed. Thank you!