ros2 / rmw Goto Github PK
View Code? Open in Web Editor NEWThe ROS Middleware (rmw) Interface.
License: Apache License 2.0
The ROS Middleware (rmw) Interface.
License: Apache License 2.0
Seeing a heap over-read reported by ASAN when running cartographer or navigation on MacOS (compiled with ASAN enabled)
Required Info:
enable ASAN by adding below to cartographer and navigation CMakeLists.txt files
set (CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fno-omit-frame-pointer -fsanitize=address")
set (CMAKE_LINKER_FLAGS_DEBUG "${CMAKE_LINKER_FLAGS_DEBUG} -fno-omit-frame-pointer -fsanitize=address")
recompile cartographer, navigation (didn't try recompiling all of ROS with ASAN, not sure if that's a mistake)
colcon build --cmake-args -DBUILD_TESTING=OFF -DCMAKE_BUILD_TYPE=Debug -DFORCE_DEBUG_BUILD=True --symlink-install
run cartographer, navigation as normal
observe ASAN flagging 8-byte heap over-read
No traps from ASAN
ASAN flags over-read and dies
I've reviewed a few different nodes, and they all have some form of this stack trace, where a node (multiple of them are killed by ASAN) is doing something that invokes rmw::impl::cpp:parse_key_value()
[occupancy_grid_node-2] =================================================================
[occupancy_grid_node-2] ==37032==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020000042d5 at pc 0x0001034d46df bp 0x70000b8f0c30 sp 0x70000b8f03e0
[occupancy_grid_node-2] READ of size 8 at 0x6020000042d5 thread T3
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] 0 0x1034d46de in __asan_memcpy (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x596de)
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] 1 0x100906a8c in std::__1::enable_if<((std::__1::integral_constant<bool, true>::value) || (!(__has_construct<std::__1::allocator, bool*, bool>::value))) && (is_trivially_move_constructible::value), void>::type std::__1::allocator_traits<std::__1::allocator >::__construct_backward(std::__1::allocator&, bool*, bool*, bool*&) memory:1699
[occupancy_grid_node-2] 2 0x10090550a in std::__1::vector<unsigned char, std::__1::allocator >::__swap_out_circular_buffer(std::__1::__split_buffer<unsigned char, std::__1::allocator&>&) vector:931
[occupancy_grid_node-2] 3 0x101aee75a in void std::__1::vector<unsigned char, std::__1::allocator >::__push_back_slow_path<unsigned char const&>(unsigned char const&&&) (librclcpp.dylib:x86_64+0xdc75a)
[occupancy_grid_node-2] 4 0x111460d93 in std::__1::vector<unsigned char, std::__1::allocator >::push_back(unsigned char const&) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x21d93)
[occupancy_grid_node-2] 5 0x11146005a in rmw::impl::cpp::parse_key_value(std::__1::vector<unsigned char, std::__1::allocator >) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x2105a)
[occupancy_grid_node-2] 6 0x11145d855 in ParticipantListener::onParticipantDiscovery(eprosima::fastrtps::Participant*, eprosima::fastrtps::rtps::ParticipantDiscoveryInfo&&) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x1e855)
[occupancy_grid_node-2] 7 0x103057aa6 in eprosima::fastrtps::rtps::PDPSimpleListener::onNewCacheChangeAdded(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::rtps::CacheChange_t const*) (libfastrtps.1.dylib:x86_64+0x1c8aa6)
[occupancy_grid_node-2] 8 0x102ee419b in eprosima::fastrtps::rtps::StatelessReader::change_received(eprosima::fastrtps::rtps::CacheChange_t*) (libfastrtps.1.dylib:x86_64+0x5519b)
[occupancy_grid_node-2] 9 0x102ee4372 in eprosima::fastrtps::rtps::StatelessReader::processDataMsg(eprosima::fastrtps::rtps::CacheChange_t*) (libfastrtps.1.dylib:x86_64+0x55372)
[occupancy_grid_node-2] 10 0x102ef3a8c in eprosima::fastrtps::rtps::MessageReceiver::proc_Submsg_Data(eprosima::fastrtps::rtps::CDRMessage_t*, eprosima::fastrtps::rtps::SubmessageHeader_t*) (libfastrtps.1.dylib:x86_64+0x64a8c)
[occupancy_grid_node-2] 11 0x102ef1869 in eprosima::fastrtps::rtps::MessageReceiver::processCDRMsg(eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::CDRMessage_t*) (libfastrtps.1.dylib:x86_64+0x62869)
[occupancy_grid_node-2] 12 0x102efaaac in eprosima::fastrtps::rtps::ReceiverResource::OnDataReceived(unsigned char const*, unsigned int, eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::Locator_t const&) (libfastrtps.1.dylib:x86_64+0x6baac)
[occupancy_grid_node-2] 13 0x102f57a91 in eprosima::fastrtps::rtps::UDPTransportInterface::perform_listen_operation(eprosima::fastrtps::rtps::UDPChannelResource*, eprosima::fastrtps::rtps::Locator_t) (libfastrtps.1.dylib:x86_64+0xc8a91)
[occupancy_grid_node-2] 14 0x102f5b402 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, void (eprosima::fastrtps::rtps::UDPTransportInterface::)(eprosima::fastrtps::rtps::UDPChannelResource, eprosima::fastrtps::rtps::Locator_t), eprosima::fastrtps::rtps::UDPTransportInterface*, eprosima::fastrtps::rtps::UDPChannelResource*, eprosima::fastrtps::rtps::Locator_t> >(void*) (libfastrtps.1.dylib:x86_64+0xcc402)
[occupancy_grid_node-2] 15 0x7fff5c05a2ea in _pthread_body (libsystem_pthread.dylib:x86_64+0x32ea)
[occupancy_grid_node-2] 16 0x7fff5c05d248 in _pthread_start (libsystem_pthread.dylib:x86_64+0x6248)
[occupancy_grid_node-2] 17 0x7fff5c05940c in thread_start (libsystem_pthread.dylib:x86_64+0x240c)
[occupancy_grid_node-2]
[occupancy_grid_node-2] 0x6020000042d8 is located 0 bytes to the right of 8-byte region [0x6020000042d0,0x6020000042d8)
[occupancy_grid_node-2] allocated by thread T3 here:
[occupancy_grid_node-2] 0 0x1034e3b92 in wrap__Znwm (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x68b92)
[occupancy_grid_node-2] 1 0x101ac4e88 in std::__1::__libcpp_allocate(unsigned long, unsigned long) (librclcpp.dylib:x86_64+0xb2e88)
[occupancy_grid_node-2] 2 0x101aca084 in std::__1::allocator::allocate(unsigned long, void const*) (librclcpp.dylib:x86_64+0xb8084)
[occupancy_grid_node-2] 3 0x101ac9ec0 in std::__1::allocator_traits<std::__1::allocator >::allocate(std::__1::allocator&, unsigned long) (librclcpp.dylib:x86_64+0xb7ec0)
[occupancy_grid_node-2] 4 0x101aedf89 in std::__1::__split_buffer<unsigned char, std::__1::allocator&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator&) (librclcpp.dylib:x86_64+0xdbf89)
[occupancy_grid_node-2] 5 0x101aede0c in std::__1::__split_buffer<unsigned char, std::__1::allocator&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator&) (librclcpp.dylib:x86_64+0xdbe0c)
[occupancy_grid_node-2] 6 0x101aee70c in void std::__1::vector<unsigned char, std::__1::allocator >::__push_back_slow_path<unsigned char const&>(unsigned char const&&&) (librclcpp.dylib:x86_64+0xdc70c)
[occupancy_grid_node-2] 7 0x111460d93 in std::__1::vector<unsigned char, std::__1::allocator >::push_back(unsigned char const&) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x21d93)
[occupancy_grid_node-2] 8 0x11146005a in rmw::impl::cpp::parse_key_value(std::__1::vector<unsigned char, std::__1::allocator >) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x2105a)
[occupancy_grid_node-2] 9 0x11145d855 in ParticipantListener::onParticipantDiscovery(eprosima::fastrtps::Participant*, eprosima::fastrtps::rtps::ParticipantDiscoveryInfo&&) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x1e855)
[occupancy_grid_node-2] [WARN] [occupancy_grid_node]: submap_slices and last_frame_id is empty
[occupancy_grid_node-2] 10 0x103057aa6 in eprosima::fastrtps::rtps::PDPSimpleListener::onNewCacheChangeAdded(eprosima::fastrtps::rtps::RTPSReader*, eprosima::fastrtps::rtps::CacheChange_t const*) (libfastrtps.1.dylib:x86_64+0x1c8aa6)
[occupancy_grid_node-2] 11 0x102ee419b in eprosima::fastrtps::rtps::StatelessReader::change_received(eprosima::fastrtps::rtps::CacheChange_t*) (libfastrtps.1.dylib:x86_64+0x5519b)
[occupancy_grid_node-2] 12 0x102ee4372 in eprosima::fastrtps::rtps::StatelessReader::processDataMsg(eprosima::fastrtps::rtps::CacheChange_t*) (libfastrtps.1.dylib:x86_64+0x55372)
[occupancy_grid_node-2] 13 0x102ef3a8c in eprosima::fastrtps::rtps::MessageReceiver::proc_Submsg_Data(eprosima::fastrtps::rtps::CDRMessage_t*, eprosima::fastrtps::rtps::SubmessageHeader_t*) (libfastrtps.1.dylib:x86_64+0x64a8c)
[occupancy_grid_node-2] 14 0x102ef1869 in eprosima::fastrtps::rtps::MessageReceiver::processCDRMsg(eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::CDRMessage_t*) (libfastrtps.1.dylib:x86_64+0x62869)
[occupancy_grid_node-2] 15 0x102efaaac in eprosima::fastrtps::rtps::ReceiverResource::OnDataReceived(unsigned char const*, unsigned int, eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::Locator_t const&) (libfastrtps.1.dylib:x86_64+0x6baac)
[occupancy_grid_node-2] 16 0x102f57a91 in eprosima::fastrtps::rtps::UDPTransportInterface::perform_listen_operation(eprosima::fastrtps::rtps::UDPChannelResource*, eprosima::fastrtps::rtps::Locator_t) (libfastrtps.1.dylib:x86_64+0xc8a91)
[occupancy_grid_node-2] 17 0x102f5b402 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct >, void (eprosima::fastrtps::rtps::UDPTransportInterface::)(eprosima::fastrtps::rtps::UDPChannelResource, eprosima::fastrtps::rtps::Locator_t), eprosima::fastrtps::rtps::UDPTransportInterface*, eprosima::fastrtps::rtps::UDPChannelResource*, eprosima::fastrtps::rtps::Locator_t> >(void*) (libfastrtps.1.dylib:x86_64+0xcc402)
[occupancy_grid_node-2] 18 0x7fff5c05a2ea in _pthread_body (libsystem_pthread.dylib:x86_64+0x32ea)
[occupancy_grid_node-2] 19 0x7fff5c05d248 in _pthread_start (libsystem_pthread.dylib:x86_64+0x6248)
[occupancy_grid_node-2] 20 0x7fff5c05940c in thread_start (libsystem_pthread.dylib:x86_64+0x240c)
[occupancy_grid_node-2]
[occupancy_grid_node-2] Thread T3 created by T0 here:
[occupancy_grid_node-2] 0 0x1034cf02d in wrap_pthread_create (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x5402d)
[occupancy_grid_node-2] 1 0x102f5b30d in std::__1::thread::thread<void (eprosima::fastrtps::rtps::UDPTransportInterface::)(eprosima::fastrtps::rtps::UDPChannelResource, eprosima::fastrtps::rtps::Locator_t), eprosima::fastrtps::rtps::UDPTransportInterface*, eprosima::fastrtps::rtps::UDPChannelResource*&, eprosima::fastrtps::rtps::Locator_t const&, void>(void (eprosima::fastrtps::rtps::UDPTransportInterface::&&)(eprosima::fastrtps::rtps::UDPChannelResource, eprosima::fastrtps::rtps::Locator_t), eprosima::fastrtps::rtps::UDPTransportInterface*&&, eprosima::fastrtps::rtps::UDPChannelResource*&&&, eprosima::fastrtps::rtps::Locator_t const&&&) (libfastrtps.1.dylib:x86_64+0xcc30d)
[occupancy_grid_node-2] 2 0x102f578e6 in eprosima::fastrtps::rtps::UDPTransportInterface::CreateInputChannelResource(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, eprosima::fastrtps::rtps::Locator_t const&, bool, unsigned int, eprosima::fastrtps::rtps::TransportReceiverInterface*) (libfastrtps.1.dylib:x86_64+0xc88e6)
[occupancy_grid_node-2] 3 0x102f5748a in eprosima::fastrtps::rtps::UDPTransportInterface::OpenAndBindInputSockets(eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::TransportReceiverInterface*, bool, unsigned int) (libfastrtps.1.dylib:x86_64+0xc848a)
[occupancy_grid_node-2] 4 0x102f40671 in eprosima::fastrtps::rtps::UDPv4Transport::OpenInputChannel(eprosima::fastrtps::rtps::Locator_t const&, eprosima::fastrtps::rtps::TransportReceiverInterface*, unsigned int) (libfastrtps.1.dylib:x86_64+0xb1671)
[occupancy_grid_node-2] 5 0x102efa214 in eprosima::fastrtps::rtps::ReceiverResource::ReceiverResource(eprosima::fastrtps::rtps::TransportInterface&, eprosima::fastrtps::rtps::Locator_t const&, unsigned int) (libfastrtps.1.dylib:x86_64+0x6b214)
[occupancy_grid_node-2] 6 0x102ef85f9 in eprosima::fastrtps::rtps::NetworkFactory::BuildReceiverResources(eprosima::fastrtps::rtps::Locator_t&, unsigned int, std::__1::vector<std::__1::shared_ptreprosima::fastrtps::rtps::ReceiverResource, std::__1::allocator<std::__1::shared_ptreprosima::fastrtps::rtps::ReceiverResource > >&) (libfastrtps.1.dylib:x86_64+0x695f9)
[occupancy_grid_node-2] 7 0x102efde05 in eprosima::fastrtps::rtps::RTPSParticipantImpl::createReceiverResources(eprosima::fastrtps::rtps::LocatorList_t&, bool) (libfastrtps.1.dylib:x86_64+0x6ee05)
[occupancy_grid_node-2] 8 0x102efd0e7 in eprosima::fastrtps::rtps::RTPSParticipantImpl::RTPSParticipantImpl(eprosima::fastrtps::rtps::RTPSParticipantAttributes const&, eprosima::fastrtps::rtps::GuidPrefix_t const&, eprosima::fastrtps::rtps::RTPSParticipant*, eprosima::fastrtps::rtps::RTPSParticipantListener*) (libfastrtps.1.dylib:x86_64+0x6e0e7)
[occupancy_grid_node-2] 9 0x102f05a2e in eprosima::fastrtps::rtps::RTPSDomain::createParticipant(eprosima::fastrtps::rtps::RTPSParticipantAttributes const&, eprosima::fastrtps::rtps::RTPSParticipantListener*) (libfastrtps.1.dylib:x86_64+0x76a2e)
[occupancy_grid_node-2] 10 0x102f08882 in eprosima::fastrtps::Domain::createParticipant(eprosima::fastrtps::ParticipantAttributes const&, eprosima::fastrtps::ParticipantListener*) (libfastrtps.1.dylib:x86_64+0x79882)
[occupancy_grid_node-2] 11 0x11145c087 in rmw_fastrtps_shared_cpp::create_node(char const*, char const*, char const*, eprosima::fastrtps::ParticipantAttributes) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x1d087)
[occupancy_grid_node-2] 12 0x11145cf07 in rmw_fastrtps_shared_cpp::__rmw_create_node(char const*, char const*, char const*, unsigned long, rmw_node_security_options_t const*) (librmw_fastrtps_shared_cpp.dylib:x86_64+0x1df07)
[occupancy_grid_node-2] 13 0x1113f2de3 in rmw_create_node (librmw_fastrtps_cpp.dylib:x86_64+0x1cde3)
[occupancy_grid_node-2] 14 0x1021f37fc in rmw_create_node (librmw_implementation.dylib:x86_64+0x47fc)
[occupancy_grid_node-2] 15 0x102077874 in rcl_node_init (librcl.dylib:x86_64+0x19874)
[occupancy_grid_node-2] 16 0x101aa2c75 in rclcpp::node_interfaces::NodeBase::NodeBase(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptrrclcpp::Context, rcl_node_options_t const&, bool) (librclcpp.dylib:x86_64+0x90c75)
[occupancy_grid_node-2] 17 0x101aa37fd in rclcpp::node_interfaces::NodeBase::NodeBase(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::shared_ptrrclcpp::Context, rcl_node_options_t const&, bool) (librclcpp.dylib:x86_64+0x917fd)
[occupancy_grid_node-2] 18 0x101a87440 in rclcpp::Node::Node(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, rclcpp::NodeOptions const&) (librclcpp.dylib:x86_64+0x75440)
[occupancy_grid_node-2] 19 0x101a872b2 in rclcpp::Node::Node(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&, rclcpp::NodeOptions const&) (librclcpp.dylib:x86_64+0x752b2)
[occupancy_grid_node-2] 20 0x10083da58 in cartographer_ros::(anonymous namespace)::OccupancyGridNode::OccupancyGridNode(double, double) occupancy_grid_node_main.cc:56
[occupancy_grid_node-2] 21 0x10083d458 in cartographer_ros::(anonymous namespace)::OccupancyGridNode::OccupancyGridNode(double, double) occupancy_grid_node_main.cc:58
[occupancy_grid_node-2] 22 0x10083d311 in std::__1::__compressed_pair_elem<cartographer_ros::(anonymous namespace)::OccupancyGridNode, 1, false>::__compressed_pair_elem<double&, double&, 0ul, 1ul>(std::__1::piecewise_construct_t, std::__1::tuple<double&, double&>, std::__1::__tuple_indices<0ul, 1ul>) memory:2153
[occupancy_grid_node-2] 23 0x10083cec3 in std::__1::__compressed_pair<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>, cartographer_ros::(anonymous namespace)::OccupancyGridNode>::__compressed_pair<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>&, double&, double&>(std::__1::piecewise_construct_t, std::__1::tuple<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>&>, std::__1::tuple<double&, double&>) memory:2255
[occupancy_grid_node-2] 24 0x10083c4f3 in std::__1::__compressed_pair<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>, cartographer_ros::(anonymous namespace)::OccupancyGridNode>::__compressed_pair<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>&, double&, double&>(std::__1::piecewise_construct_t, std::__1::tuple<std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>&>, std::__1::tuple<double&, double&>) memory:2256
[occupancy_grid_node-2] 25 0x10083bd35 in std::__1::__shared_ptr_emplace<cartographer_ros::(anonymous namespace)::OccupancyGridNode, std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode> >::__shared_ptr_emplace<double&, double&>(std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>, double&&&, double&&&) memory:3668
[occupancy_grid_node-2] 26 0x10083b244 in std::__1::__shared_ptr_emplace<cartographer_ros::(anonymous namespace)::OccupancyGridNode, std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode> >::__shared_ptr_emplace<double&, double&>(std::__1::allocator<cartographer_ros::(anonymous namespace)::OccupancyGridNode>, double&&&, double&&&) memory:3669
[occupancy_grid_node-2] 27 0x10083adb6 in std::__1::shared_ptr<cartographer_ros::(anonymous namespace)::OccupancyGridNode> std::__1::shared_ptr<cartographer_ros::(anonymous namespace)::OccupancyGridNode>::make_shared<double&, double&>(double&&&, double&&&) memory:4327
[occupancy_grid_node-2] 28 0x10082b9a3 in std::__1::enable_if<!(is_array<cartographer_ros::(anonymous namespace)::OccupancyGridNode>::value), std::__1::shared_ptr<cartographer_ros::(anonymous namespace)::OccupancyGridNode> >::type std::__1::make_shared<cartographer_ros::(anonymous namespace)::OccupancyGridNode, double&, double&>(double&&&, double&&&) memory:4706
[occupancy_grid_node-2] 29 0x10082b659 in main occupancy_grid_node_main.cc:193
[occupancy_grid_node-2] 30 0x7fff5be663d4 in start (libdyld.dylib:x86_64+0x163d4)
[occupancy_grid_node-2]
[occupancy_grid_node-2] SUMMARY: AddressSanitizer: heap-buffer-overflow (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x596de) in __asan_memcpy
[occupancy_grid_node-2] Shadow bytes around the buggy address:
[occupancy_grid_node-2] 0x1c0400000800: fa fa 00 00 fa fa 00 00 fa fa 00 00 fa fa 00 fa
[occupancy_grid_node-2] 0x1c0400000810: fa fa 00 00 fa fa 00 00 fa fa 00 fa fa fa 00 00
[occupancy_grid_node-2] 0x1c0400000820: fa fa 00 00 fa fa 00 fa fa fa 00 00 fa fa 00 00
[occupancy_grid_node-2] 0x1c0400000830: fa fa 00 fa fa fa 00 00 fa fa 00 00 fa fa 00 fa
[occupancy_grid_node-2] 0x1c0400000840: fa fa 00 00 fa fa 00 00 fa fa 00 fa fa fa fd fa
[occupancy_grid_node-2] =>0x1c0400000850: fa fa fd fa fa fa fd fa fa fa[05]fa fa fa 00 00
[occupancy_grid_node-2] 0x1c0400000860: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] 0x1c0400000870: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] 0x1c0400000880: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] 0x1c0400000890: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] 0x1c04000008a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[occupancy_grid_node-2] Shadow byte legend (one shadow byte represents 8 application bytes):
[occupancy_grid_node-2] Addressable: 00
[occupancy_grid_node-2] Partially addressable: 01 02 03 04 05 06 07
[occupancy_grid_node-2] Heap left redzone: fa
[occupancy_grid_node-2] Freed heap region: fd
[occupancy_grid_node-2] Stack left redzone: f1
[occupancy_grid_node-2] Stack mid redzone: f2
[occupancy_grid_node-2] Stack right redzone: f3
[occupancy_grid_node-2] Stack after return: f5
[occupancy_grid_node-2] Stack use after scope: f8
[occupancy_grid_node-2] Global redzone: f9
[occupancy_grid_node-2] Global init order: f6
[occupancy_grid_node-2] Poisoned by user: f7
[occupancy_grid_node-2] Container overflow: fc
[occupancy_grid_node-2] Array cookie: ac
[occupancy_grid_node-2] Intra object redzone: bb
[occupancy_grid_node-2] ASan internal: fe
[occupancy_grid_node-2] Left alloca redzone: ca
[occupancy_grid_node-2] Right alloca redzone: cb
[occupancy_grid_node-2] Shadow gap: cc
[occupancy_grid_node-2] ==37032==ABORTING
@wjwwood and I were talking yesterday and realised that the default QoS profile for services might lead to undesired behaviour.
The default durability is RMW_QOS_POLICY_TRANSIENT_LOCAL_DURABILITY
. @wjwwood pointed out that this was set before wait_for_service
was implemented, as a way to compensate for race conditions in server/client bringup.
With this setting, in the case of a server that starts, dies, and restarts, it will receive the service request from the client twice. While the client is protected from receiving multiple service responses, the server is not protected from side effects of receiving multiple service calls.
If a service does anything like write to file, increment a counter, toggle a value, etc, the transient_local setting is probably not a good idea.
A better default might be volatile, and we should be careful about exposing the transient_local setting to users.
rmw / fastrtps appears to throw an assert when another node becomes unreachable. This makes for some hard to notice "hmmm why did it stop working?" scenarios
Required Info:
Start TB3 latest package which is split between three CPU's - wheel control on OpenCR, robot state publisher on Raspberry Pi, and rviz / main workload processing on Mac or Ubuntu PC
Debugging some issue that I saw when running rviz on Ubuntu, wanted to switch over to Mac and see if maybe that issue stopped happening. Shut down Ubuntu, and the instance of rmw/fast-RTPS throws an exception while the Ubuntu side is shutting down.
Unfortunately, I don't notice the exception on Raspberry Pi side until several minutes later when I'm scratching my head trying to figure out why I can't run basic teleop on Mac anymore... oh yeah, rmw threw an error and my bot is basically dead.
if one node goes down, the other nodes shouldn't crash and burn
crashing and burning
[robot_state_publisher-1] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[robot_state_publisher-1] what(): failed to publish message: cannot publish data, at /tmp/binarydeb/ros-dashing-rmw-fastrtps-shared-cpp-0.7.5/src/rmw_publish.cpp:52, at /tmp/binarydeb/ros-dashing-rcl-0.7.6/src/rcl/publisher.c:257
rmw_time_t
is defined as:
typedef struct RMW_PUBLIC_TYPE rmw_time_t
{
uint64_t sec;
uint64_t nsec;
} rmw_time_t;
This differs from rcutils_duration_value_t
which admits a value as just a int64_t
holding nanoseconds. It probably makes sense to replace rmw_time_t
with a type alias for rcutils_duration_value_t
, and to make clear that it is a duration, not a time point.
Given that namespaces are supported by now, we should refactor the rmw interface to not only give one accumulated list of all topics, but rather distinguish them between publishers, subscribers, services and clients.
Currently, the macro RMW_CHECK_TYPE_IDENTIFIERS_MATCH
has an error message of the format:
" implementation '%s'(%p) does not match rmw implementation '%s'(%p)"
However, in the rmw implementations this macro is used for checking the types of typesupport identifiers, not only rmw implementation identifiers, leading to confusing and incorrect error messages such as:
could not create publisher: type support handle implementation 'connext_static'(0x2b54f95e9671) does not match rmw implementation 'introspection'(0x2b54f97eb691)
This macro should be reworded to be more general, or a different macro should be made for matching typesupport identifiers.
I believe it's a good idea to shift the error handling from rmw
to c_utilities
to avoid duplication and typedefs everywhere and have clean dependencies.
This issue tracks the progression of rmw
to Quality Level 1 and a 1.0 version level. It follows the outline described in REP 2004.
Documentation
endian.hpp
, filesystem_helper.hpp
, find_and_replace.hpp
, find_library.hpp
, join.hpp
, split.hpp
, pointer_traits.hpp
, visibility_control.hpp
)endian.hpp
, filesystem_helper.hpp
, thread_safety_annotations.hpp
, visibility_control.hpp
)join.hpp
)Testing:
Dependencies:
Platform Support
Since a subscriber can receive a topic message only if its QOS profile is compatible with the QOS profile of the publisher a method to retrieve this information from the available topics should be very useful.
I think that the command
$ ros2 topic info /topic
can be a good place to introduce this important feature.
Ref: https://answers.ros.org/question/304946/ros2-retrieving-qos-settings-for-a-topic/
Draft overview and background is ros2/design#282,
Introduce dds contentfiltertopic
to ROS
, dds specification is DDS v1.4 2.2.2.3.3 ContentFilteredTopic Class. ContentFilteredTopic can be created on parent topic and DataReader only receives the messages that matches with filtering expression and expression parameters. If implementation does not support this feature, it returns not support
.
Implementation | Supported | Comment |
---|---|---|
RTI Connext | Yes | https://community.rti.com/examples/content-filtered-topic, Not sure if filtering is done on writer or reader side. |
eProsima Fast-DDS | No | |
ADLINK cyclonedds | No | |
ADLINK OpenSplice | Yes | E.O.L foxy or newer |
expected user for this feature.
rmw
and rcl
implementation
rcl
and rclcpp
parameter_events
filtering which calls ContentFilteredTopic.rcl
based when rmw does not support ContentFilteredTopic.Right now, when requesting endpoint info from rmw implementations, e.g. rmw_get_publishers_info_by_topic
, we receive inconsistent results for publishers created with the same QoS settings. This makes it hard to create subscriptions adaptively when using queried QoS profiles. It also means that the reporting is noisy/lossy, we do not know what settings were really given to the endpoints.
rmw_get_publishers_info_by_topic
and rmw_get_subscriptions_info_by_topic
that these infinity values will be reported by RMW implementationsThe DDS Spec's OMG IDL PSM (Platform Specific Model) defines the following (page 139)
struct Duration_t {
long sec;
unsigned long nanosec;
};
const long DURATION_INFINITE_SEC = 0x7fffffff;
const unsigned long DURATION_INFINITE_NSEC = 0x7fffffff;
const long DURATION_ZERO_SEC = 0;
const unsigned long DURATION_ZERO_NSEC = 0;
We should do something similar. However, since we use a uint64_t
for both fields of rmw_time_t
, I'd suggest the infinity value being "uint64_max" for both seconds and nanoseconds.
We have a lot of different formats for the names of QoS policy settings at the moment. The inconsistency makes it inconvenient for users to change the options without looking them up.
If we unify the names, what should the format be?
RMW_QOS_POLICY_{SETTING}_{CATEGORY}
e.g. how we have RMW_QOS_POLICY_KEEP_LAST_HISTORY (closest to DDS naming e.g. RELIABLE_RELIABILITY_QOS)RMW_QOS_POLICY_{CATEGORY}_{SETTING}
e.g. how we have RMW_QOS_POLICY_HISTORY_SYSTEM_DEFAULTRMW_QOS_POLICY_{SETTING}
e.g. how we have RMW_QOS_POLICY_RELIABLERMW_QOS_{CATEGORY}_POLICY_{SETTING}
e.g. how we have rmw_qos_history_policy_tor perhaps a different (shorter) option
Closed in favor of ros2/design#259 (comment)
Some of the include guards are still using ROS_MIDDLEWARE_INTERFACE
rather than RMW
.
The current API only passes a topic name which can only return the number of publishers / subscribers independently if they are already matched (due to timing) or will never match (due to incompatible QoS settings).
The new API needs to be coupled to a publisher / subscriber handle since it needs that context to decide if another entity is actually matching this entities QoS.
See discussion: #311 (comment)
The corresponding parameters and variables upstream/downstream of rmw
should also be renamed to be consistent. API docs will also need to be updated.
For reusability purposes, the rmw_serialized_message_t
type should be excluded from this rmw
package. The data type shall be placed in rcutils
as it is basically only a vector of characters.
Required Info:
colcon build --packages-up-to rmw
colcon test --packages-up-to rmw
Tests on test/test_names_and_types.cpp:131 and test/test_names_and_types.cpp:137 pass.
13: [ RUN ] rmw_names_and_types.rmw_names_and_types_fini
13: /home/jenkins-agent/workspace/nightly_linux-aarch64_debug/ws/src/ros2/rmw/rmw/test/test_names_and_types.cpp:131: Failure
13: Expected equality of these values:
13: rmw_names_and_types_fini(&names_and_types)
13: Which is: 0
13: 11
13: /home/jenkins-agent/workspace/nightly_linux-aarch64_debug/ws/src/ros2/rmw/rmw/test/test_names_and_types.cpp:137: Failure
13: Expected equality of these values:
13: rmw_names_and_types_fini(&names_and_types)
13: Which is: 0
13: 11
13: [ FAILED ] rmw_names_and_types.rmw_names_and_types_fini (0 ms)
I was looking at vcs exact output between today and yesterday and I have no inkling as to why these tests which were passing yesterday are failing today. These should be shot for shot tests of code here and here.
I'm trying to determine roughly what the diff is between QoS functionality available via rmw interfaces vs dds itself. From a read through rmw/{types, qos_profiles, rmw}.h
it looks like there are some diffs:
i think there are others as well, but wanted to check to see what is intended/planned.
We see two ways to implement this: 1) creating rmw_pre_init() and rmw_post_init() functions or 2) Passing a parameter to rmw_init that would be used to specify the topology.
e.g,:
Option 1
rmw_ret_t rmw_pre_init(rmw_init_options_t * context_ptr) //not necessarily void *
{
//pre-init stuff here
return RMW_RET_OK;
}
rmw_ret_t rmw_post_init(rmw_init_options_t * context_ptr) //not necessarily void *
{
// post-init stuff here
return RMW_RET_OK;
}
int main(void)
{
rmw_init_options_t * context_ptr = ...;
rmw_ret_t retval = rmw_pre_init(context_ptr);
// Check for and handle errors
retval = rmw_init();
if (ret == RMW_RET_OK) {
retval = rmw_post_init(context_ptr);
// Check for and handle errors
} else {
// error handling...
}
return 0;
}
rmw_ret_t entry_callback(void * caller_context_ptr)
{
// do pre-init stuff
return RMW_RET_OK;
}
rmw_ret_t exit_callback(void * caller_context_ptr, rmw_ret_t current_rmw_init_status)
{
// do post-init stuff
return RMW_RET_OK;
}
int main(void)
{
struct rmw_init_callbacks_t init_callbacks {
.m_caller_context_ptr = NULL,
.m_callback_at_rmw_init_entry = &entry_callback,
.m_callback_at_rmw_init_exit = &exit_callback
};
rmw_ret_t ret = rmw_init(init_callbacks); // new rmw_init
// error handling...
return 0;
}
rmw_ret_t rmw_init(const rmw_init_callbacks_t * callback_ptr) // new rmw_init
{
rmw_ret_t retval = RMW_RET_ERROR;
retval = callback_ptr->m_callback_at_rmw_init_entry(callback_ptr->m_caller_context_ptr);
// check for and handle errors
retval = old_rmw_init(); //old rmw_init
// check for and handle errors
retval = callback_ptr->m_callback_at_rmw_init_exit(callback_ptr->m_caller_context_ptr, retval);
// check for and handle errors
return retval
}
** Credit to @wjwwood @serge-nikulin for the original examples (which I may have bastardized when I elaborated on them)
Option 1
Option 2
At a high-level for the static topology, you would need to specify a graph of how your nodes are connected and what kind of resources they need to communicate (on an edge-by-edge basis). e.g., :
See discussion in:
ros2/rcl#30 (comment)
Every now and then there is a change to the RMW interface that necessitates updating all the RMW implementations โย for example #187, which gives the application a way to control options offered by the underlying implementation that are not otherwise exposed in the regular RCL interfaces. This then obviously results in the updated RMW implementations no longer being compatible with older versions of ROS2.
Having a version number for the RMW interface allows an implementation to be compatible with a larger range of ROS2 versions using a single source version.
One could also do the same in reverse: let RMW implementations advertise the version they implement, then provide compatibility with older RMW implementations where possible with a small effort. (#187 would have been pretty trivial.) The more different implementations of the RMW interface exist outside the ROS2 sources, the more valuable this becomes.
In my view the implementation could be as simple as defining an RMW_VERSION
macro in rmw.h
, expanding to an integer that gets incremented whenever there is an interface change. That would have a minimal impact on managing the RMW interface but would give RMW implementers options.
While letting RMW implementations advertise the interface version they implement and providing compatibility with older RMW implementations would definitely be very nice for the RMW implementers, it would also require some real effort in maintaining the RMW interface layer. That might be a bit too much to ask for ๐ .
I don't know the full scope of doing this, but I know for sure that in order to get persistent durability, we'll need to add a new member to the durability_policy_t here: https://github.com/ros2/rmw/blob/master/rmw/include/rmw/types.h#L191 . I'm sure there is other work to get it down to the RMW implementation properly.
This is not an issue but rather a question about ROS2 & FastRTPS.
https://answers.ros.org/question/358361/fastrtps-communication-over-other-network/
Required Info:
The fast (1kHz) publisher and slow (10Hz) subscriber
https://gist.github.com/cleitner/93decfa79a99a8a3b59a795df02b99e7
https://gist.github.com/cleitner/e5eb35f6bcf6639425c06c03dd13fbff
expose problems with buffer bloat and, more relevant here, issues with the rmw_publish
interface.
As described in #176, the RMW implementations can cause rmw_publish
to block when history is set to KEEP_ALL
Calling the two scripts with
$ ./pub_back_pressure.py
$ ./sub_back_pressure.py
will sometimes exhibit blocking times > 20ms in rmw_publish
and more often RMW_RET_ERROR
when the internal RESOURCE_LIMITS setting is hit.
...
Failed to publish at 10032 with Failed to publish: cannot publish data, at .../src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_publish.cpp:53, at .../src/ros2/rcl/rcl/src/rcl/publisher.c:290
Failed to publish at 10032 with Failed to publish: cannot publish data, at .../src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_publish.cpp:53, at .../src/ros2/rcl/rcl/src/rcl/publisher.c:290
Slowed down at 10032 for 41.456 ms
Slowed down at 10035 for 101.639 ms
Failed to publish at 10039 with Failed to publish: cannot publish data, at .../src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_publish.cpp:53, at .../src/ros2/rcl/rcl/src/rcl/publisher.c:290
Failed to publish at 10032 with Failed to publish: cannot publish data, at .../src/ros2/rmw_fastrtps/rmw_fastrtps_shared_cpp/src/rmw_publish.cpp:53, at .../src/ros2/rcl/rcl/src/rcl/publisher.c:290
...
With
$ RMW_IMPLEMENTATION=rmw_cyclonedds_cpp ./pub_back_pressure.py
$ RMW_IMPLEMENTATION=rmw_cyclonedds_cpp ./sub_back_pressure.py
the publisher blocks with (seemingly) random timeouts, but no errors:
...
Slowed down at 3205 for 21.131 ms
Slowed down at 3207 for 21.206 ms
Slowed down at 3209 for 209.976 ms
Slowed down at 3210 for 275.831 ms
Slowed down at 3211 for 173.350 ms
Slowed down at 3212 for 97.646 ms
Slowed down at 3213 for 43.757 ms
...
The subscriber crashes with OoM because CycloneDDS doesn't seem to limit the incoming buffer.
At least consistent behavior regarding returning errors or blocking.
The IMHO correct result would be an indication of a need for blocking, akin to the taken
parameter to rmw_take
to allow pure polling operation of rmw_publish
, i.e. in cyclic realtime context.
No crashing in the subscriber with default settings.
rmw_publish
blocks when some unknown internal (and external) limit is hit. Fast-RTPS also returns RMW_RET_ERROR
, without having experiences a real error.
CycloneDDS subscriber can be crashed with OoM if the publisher exceeds the subscribers processing bandwidth.
The write
function of the DDS DataWriter interface has a OUT_OF_RESOURCES
return code and specifies a max_block_time
parameter for the RELIABILITY
QoS, which would preferrably be set to 0.
The interface of rmw_publish
could be made symmetric to rmw_take
:
rmw_ret_t
rmw_publish(
const rmw_publisher_t * publisher,
const void * ros_message,
bool * published,
rmw_publisher_allocation_t * allocation);
This will make debugging much easier.
From discussion on ros2/rmw_connext#43
typedef struct CustomParticipantInfo
{
eprosima::fastrtps::Participant * participant;
ReaderInfo * secondarySubListener;
WriterInfo * secondaryPubListener;
rmw_guard_condition_t * graph_guard_condition;
} CustomParticipantInfo;
What is the purpose of the secondarySubListener in the above strucutre, which is implemented in custom_participant_info.hpp ?
Reconsidering 1-to-1 mapping of ROS nodes to DDS participants
As a result of the research published here:
https://github.com/nobleo/ros2_performance
A ROS discourse discussion was started here:
https://discourse.ros.org/t/reconsidering-1-to-1-mapping-of-ros-nodes-to-dds-participants/10062
at this discussion it was requested that this feature be posted as an issue.
From the discourse discussion:
dirk-thomas:
"Instead of introducing an option the current idea is to associate the DDS participant with the context created during rmw_init. That would imply that common applications using a single init / context will only use a single DDS participant - even if they are composed of multiple ROS nodes."
Argumentation for change: Making every Node in rclcpp a DDS participant creates significant CPU overhead. Use cases exists where all Nodes run on a single piece of hardware. The CPU overhead can be reduced by creating 1 participant per process space.
From the discourse discussion:
dirk-thomas:
"... Any help is appreciated. It will likely start with a design article to discuss the side affects of the intended change. E.g. the ROS node name is currently being used for the DDS participant name. When that mapping goes away there needs to be a replacement mechanism to communicate the node name."
kyrofa:
"Not trying to debate the performance gains of this approach, but I think itโs worth pointing out that SROS 2 (and indeed DDS-Security in general) currently only supports security at the domain participant level. Using the same participant for multiple nodes will make security very difficult, as all nodes will have effectively the same identity and thus the same access control."
In order to reduce dynamic memory allocations during publish and receipt of messages during the operation of a Node, this API would ideally allow for users of the RMW API to request the middleware to preallocate memory for predefined message types and sizes.
Implement a struct that hides the implementation of the buffer allocation that future calls to publish
will take as an argument. If a publisher is called with the argument, then it will use the previously-allocated buffer rather than allocating a new one as necessary.
typedef struct RMW_PUBLIC_TYPE rmw_publisher_allocation_t
{
const char * implementation_identifier;
void * data;
} rmw_publisher_allocation_t;
typedef struct RMW_PUBLIC_TYPE rmw_subscription_allocation_t
{
const char * implementation_identifier;
void * data;
} rmw_subscription_allocation_t;
These can then be used by the rmw_publish
and rmw_subscription
methods to provide a reserved space for serializing/deserializing incoming/outgoing messages.
Before any messages were publish
or take
, these would need to be initialized via an API as follows:
rmw_init_publisher_allocation_t(rmw_publisher_allocation_t*, const rosidl_message_typesupport_t*, const rosidl_message_bounds_t*);
rmw_init_subscription_allocation_t(rmw_publisher_allocation_t*, const rosidl_message_typesupport_t*, const rosidl_message_bounds_t*)
rmw_fini_publisher_allocation_t(rmw_publisher_allocation_t*);
rmw_fini_subscription_allocation_t(rmw_subscription_allocation_t*)
This will also required additional information from the typesupport to get the serialized message sizes, also taking into account the potential size of unbounded fields. To account for unbounded fields, I propose adding a second structure rosidl_message_bounds_t
that will give the user a way of specifying known fixed bounds for unbounded messages.
For fixed and bounded sized messages, the API should take a const rosidl_message_typesupport_t*
which will then be allocated by the RMW implementation.
The rosidl_message_bounds_t
structure would be on a per message basis and describe any of the unbounded fields. Besides the nominal case, there are three main cases to handle.
# Message with unbounded primitive
uint8[] data
struct Msg__bounds {
size_t data__length = 0;
};
# Message with string array
string[] data
# special structure for delimiting string bounds
struct string__bounds {
size_t bounds = 0;
};
struct Msg__bounds {
size_t data__length = 0;
string__bounds data__bounds;
};
# Message with unbounded array of complex structures
std_msgs/Header header[]
# special structure for delimiting string bounds
struct string__bounds {
size_t bounds = 0;
};
struct std_msgs__msg__Header__bounds {
string__bounds frame_id__bounds;
};
struct Msg__bounds {
size_t header__length = 0;
std_msgs__msg__Header__bounds header__bounds;
};
Serialized length can be retrieved via:
rmw_get_serialized_length(const rosidl_message_typesupport_t*, const rosidl_message_bounds_t*);
These structures can then be used in conjunction with rmw_publisher
:
rmw_publisher_allocation_t alloc;
rmw_init_publisher_allocation(&alloc, ts, bounds);
# TODO: What happens when ros_message violates bounds given?
rmw_publish(publisher, ros_message, alloc);
# Also rmw_publish_serialized_message()
Or with rmw_subscriber
:
rmw_subscription_allocation_t alloc;
rmw_init_subscription_allocation(&alloc, ts, bounds);
rmw_create_subscription(node, ts, topic, qos, ignore_local, alloc)
# TODO: What happens when incoming message violates bounds?
rmw_take()
Open Questions:
Line 16 in 28f9185
here the event-related function is declared in a isolated event.h
header file, but it is not included in the rmw.h.
when we implement a rmw-xxx, we have to include another rmw/event.h, why the event
is so special.
The DDS Security standard includes a Logging plugin that handles logging of security events to a special topic. This is implemented and available in RTI Connext, but currently must be enabled via source code, and the corresponding configuration is not exposed in any way by RMW. This should be changed to allow users to enable either DDS Security logging to the topic or a specified file through startup configuration.
One reasonable way to present this to the user would be to make this configurable via environment variables which, if set, can configure whether logging to the secure topic, to a file, or both are to be enabled. For example, we could have:
RMW_SECURITY_LOG_TO_TOPIC=true/false
RMW_SECURITY_LOG_TO_FILE= <path to file>
The default setting in the absence of an override should be to log to the topic, but not to a file.
A couple of destruction methods have signatures that make it impossible to obtain an allocator from an active context, which means that any of the members those methods are meant to deallocate cannot use a user defined allocator to do so.
An example is the rmw_destroy_guard_condition
I suspect there are a couple more, such as rmw_destroy_wait_set
, etc.
This could be done by either sticking the context or node pointer in the relevant structs, or passing it into the method like how rmw_destroy_publisher
does it. Though this might be API breaking for all existing rmw implementations.
On the other hand, I'm not sure if the recommended way to allocate or deallocate dynamic memory is via rmw_allocate
and rmw_free
. It looks like those just boil down to a malloc
and free
/delete
call.
I presume that using allocators (that is, rcutils_allocator_t
) would be preferable?
Acceptance Criteria:
Expose resource limit QoS. If not, KEEP_ALL history policy is not configurable at all.
See comment ros2/rclcpp#727 (comment).
Also, it will be needed if we want to implement KEEP_ALL policy with intraprocess communication (see ros2/rclcpp#727).
Initial context: ros2/rosbag2#773
Services can't be recorded and replayed. This prevents actions from being replayed which is an important feature for offline debugging/testing.
As of now checks for both the service names and topic names are same, but there should be separate length checks for service names as the service names are suffixed with a Request
or a Reply
in rmw_dds implementations which limits the actual length of the service names that can be used. Note: special consideration for rmw_connext implementation which suffixes the GUID of the participant to the service_name along with Reply
. See: ros2/rmw_connext#285
Required Info:
Run in different terminals
ros2 run examples_rclcpp_minimal_subscriber subscriber_lambda
ps aux | grep subscriber_lambda
ros2 run examples_rclcpp_minimal_client client_main
ps aux | grep subscriber_lambda
The memory used by the subscriber_lambda
process does not change while creating an additional node that does not share any pub/sub/client/service with it.
The memory used by the subscriber_lambda
process increases.
Before:
RSS: 21192 VRT: 433292
After:
RSS: 21452 VRT: 498828
The increase in memory is proportional to the number of pub/sub/client/service created in the new node.
For example, slightly changing the code for examples_rclcpp_minimal_client client_main
in order to create 12 clients instead of only 1 causes the following:
After:
RSS: 21720 VRT: 498828
This behavior is observed with both Fast-RTPS 1.7.0, 1.7.2 and OpenSplice (version shipped with Crystal patch3).
Note that with OpenSplice, this should not happen if nodes are run in the same process.
Moreover, creating a FastRTPS example and simulating this behavior, there is only an increase in VRT.
It looks like the RMW is allocating some memory whenever a new end-point is discovered.
However, this aggressive approach does not scale well (especially if every node contains a ParameterServer and a ParameterClient).
For example, in Fast-RTPS v1.7.0, considering a single process application with multiple nodes.
If we have 1 node and we add a second one, the memory overhead is approximately 10Mb.
If we have 20 nodes and we add a 21st one, the memory overhead is 23Mb.
I guess that the reason for this behavior is the following scenario:
Node A publishes to topic 1.
Node B is created and discovers Node A and its publisher.
Later, Node B creates a subscriber to topic 1 and it can immediately receive messages as everything is already set-up.
In the presence of multiple publishers, with potentially different QoS settings, it is useful to automatically setup a subscription with a QoS that is compatible with the majority (if not all) of the publishers. Presumably, what it means for two QoS profiles to be compatible depends on the RMW implementation. Therefore, it makes sense to offer an API that, given one or more endpoints, returns a QoS profile that is compatible with the most endpoints.
Immediately useful in the following tools:
Given the non-symmetric nature of publisher and subscription QoS policies, it may make sense to add two API's: one for getting a publisher QoS and one for getting a subscription QoS.
There already exists an implementation for DDS RMWs in rosbag2, see related issue about moving logic to a common place: ros2/rosbag2#601
I'm proposing we actually move the logic to the RMW layer, since it may vary depending on the RMW implementation.
Perhaps moving this logic to RMW is overkill, as the existing QoS policies are fairly specific to DDS. Alternatively, we could hard-code the compatibility rules in rcl
.
Hello !
I am developing a custom implementation of RMW, and I can't figure what rmw_take and rmw_take_with_info should do.
Especially the interaction with rclcpp callbacks.
This comment could apply to other undocumented functions.
... currently only ensures that the history depth of the data reader and writer is at least the specified size.
While this allow the user to use vendor specific configuration to override the QoS parameters it prevents the user from setting a not-to-be-changed queue size.
We might want to consider offering both options to the developer and decide which behavior should be the default.
The auto-populated seq
field was dropped from Header in transition from ROS1 to ROS2 (ros2/common_interfaces#2). This data was useful in measuring lost messages, and I imagine will be even more helpful with ROS2's focus on unreliable communication channels.
Some implementation hints were provided in https://discourse.ros.org/t/foxy-message-api-review/13105/17
To avoid the current compiler warnings: http://ci.ros2.org/job/ros2_batch_ci_linux/70/warnings17Result/
Nodes are creating with a name
as well as a namespace
. The function rmw_create_node
only returns the name though. It should include the namespace as well - either as separate return values or as a combined value.
Follow up of #95.
We make some investigations to find out the dependencies of transport latencies between different nodes. Therefore we made an set up with one sender-receiver-computer (IOM) and one repeater-computer (CPM). The IOM publishes defined data-packages on a constant frequency f
with the sending timestamp t1
to the network. The CPM subscribes this, takes the receiving time t2
, copies the data and publishes the data on time t3
. The IOM subscribes at t4
and logs the data.
Afterwards the transport times between the sender-receiver-computer and the repeater-computer and vice versa shall be calculated. We assume them to be equal (Dt = t2-t1 = t4-t3
) and use the formula
Dt = ( (t4-t1) - (t3-t2) ) / 2
which means, that the clocks of both computers don't have to be synchronized.
We assume the transport time Dt
to be independent from the sending frequency f
, as long as the network is not overloaded. But our results show an linear increasing transport time Dt
on linear decreasing frequencies f
(as you can see in the following section). In the same time, our timestamps t1
and t2
show that the sending frequency seems to be correct.
Below you will see the results of our investigation. Information to our set up you will find in the next section. We do some tests with LAN and WLAN. In both cases there are some strange behaviours. Regarding the latency-issue it's on both the same. The ordinate axis is limited to 20 ms for a better readability. Latencies bigger than 20 ms are counted as packages out of bounds. So lets take a look to the WLAN test results:
For us the results are not as expected. Can you help us to find out the reason for it? Is there something like a fixed buffer or something else we don't see at the moment???
The computers are Raspberry Pi 2 with Raspbian installed on. We did our investigation with openSplice and FastRTPS. But there are no significant differences related to this behaviour. The ROS2 build is alpha7.
Below you will find the code for this two nodes:
#include "rclcpp/rclcpp.hpp"
#include <string>
#include "test_lat_msg/msg/test_lat_msg.hpp"
// reference time stamp
static std::chrono::steady_clock::time_point t_start;
static FILE *fd_log;
void cb_subscriber(const test_lat_msg::msg::LatMsgCpm::SharedPtr msg)
{
// calc receive time
uint64_t t4 = (std::chrono::steady_clock::now()-t_start).count();
// store received data to logfile
fwrite (&msg->id, sizeof(msg->id), 1, fd_log);
fwrite (&msg->t1, sizeof(msg->t1), 1, fd_log);
fwrite (&msg->t2, sizeof(msg->t2), 1, fd_log);
fwrite (&msg->t3, sizeof(msg->t3), 1, fd_log);
fwrite (&t4, sizeof(t4), 1, fd_log);
fwrite (&(msg->payload[0]), sizeof(uint8_t), 32, fd_log);
fflush(fd_log);
}
int main(int argc, char * argv[])
{
// aircraft number with default value - 0 should not use as valid value
int flzNo = 0;
// frequency of data output [Hz]
int freq = 10;
// replace default values if some arguments given
// first argument: aircraft number
if( argc > 1)
{
flzNo = std::stoi(argv[1]);
}
// second argument: frequency of data output
if( argc > 2)
{
freq = std::stoi(argv[2]);
}
// create a log-file with a unique name
...
// create names for node and topics
...
// initialize ROS2
rclcpp::init(argc, argv);
// initialize a node
auto node = rclcpp::node::Node::make_shared(flzNodeName);
// initialize a puplisher for IOM-messages
auto publisher = node->create_publisher<test_lat_msg::msg::LatMsgIom>
(iomTopicName, rmw_qos_profile_sensor_data);
auto subscriber = node->create_subscription<test_lat_msg::msg::LatMsgCpm>
(cpmTopicName, cb_subscriber, rmw_qos_profile_sensor_data);
// take a reference time stamp
t_start = std::chrono::steady_clock::now();
// declare a publisher message
test_lat_msg::msg::LatMsgIom pupmsg;
// add aircraft number as IOM-ID to the puplisher message
pupmsg.iomid = flzNo;
// create a payload
for(uint64_t i=0; i<32; i++)
{
pupmsg.payload[i] = i;
}
// set loop rate for publishing messages
rclcpp::WallRate loop_rate(freq);
// a counter for the message id
uint64_t cnt = 0;
while ( rclcpp::ok() )
{
pupmsg.id = cnt++;
// put sending time to the message
pupmsg.t1 = (std::chrono::steady_clock::now()-t_start).count();
publisher->publish(pupmsg);
// iterate callbacks
rclcpp::spin_some(node);
// time control of the loop
loop_rate.sleep();
}
return 0;
}
#include "rclcpp/rclcpp.hpp"
#include <string>
#include "CpmRelay.h"
int main(int argc, char * argv[]){
// count of aircrafts or rather count of relays provided by this CPM
// the default value is 5
uint8_t flzCnt = 5;
// replace default value if some arguments given
// first argument: aircraft count
if( argc > 1){
flzCnt = std::stoi(argv[1]);
}
// initialize ROS2
rclcpp::init(argc, argv);
// initialize a node
auto node = rclcpp::node::Node::make_shared("test_cpm_test");
// create an instance for each relay
std::vector<test::CpmRelay*> cpmRelay;
for(int i=1; i<=flzCnt; i++){
cpmRelay.push_back(new test::CpmRelay(node, i));
}
std::cout << "spin" << std::endl;
rclcpp::spin(node);
return 0;
}
#ifndef CPMRELAY_H_INCLUDED
#define CPMRELAY_H_INCLUDED
#include "rclcpp/rclcpp.hpp"
#include <string>
#include "test_lat_msg/msg/test_lat_msg.hpp"
namespace test
{
class CpmRelay
{
private:
uint8_t flzNo;
std::string cpmTopicName;
std::string iomTopicName;
rclcpp::node::Node::SharedPtr node;
rclcpp::publisher::Publisher<test_lat_msg::msg::LatMsgCpm>::SharedPtr
publisher;
rclcpp::subscription::Subscription<test_lat_msg::msg::LatMsgIom>::SharedPtr
subscriber;
test_lat_msg::msg::LatMsgCpm pupmsg;
/// offset-time / start time
std::chrono::steady_clock::time_point t_start;
public:
CpmRelay( rclcpp::node::Node::SharedPtr node, uint8_t flzNo );
private:
void cb_subscriber(const test_lat_msg::msg::LatMsgIom::SharedPtr msg);
};
}
#endif // CPMRELAY_H_INCLUDED
#include "rclcpp/rclcpp.hpp"
#include <string>
#include "test_lat_msg/msg/test_lat_msg.hpp"
#include "CpmRelay.h"
test::CpmRelay::CpmRelay( rclcpp::node::Node::SharedPtr pnode,
uint8_t pflzNo)
{
this->flzNo = pflzNo;
this->node = pnode;
// get start time
this->t_start = std::chrono::steady_clock::now();
cpmTopicName = "t_flz" + std::to_string(flzNo) + "_cpm";
iomTopicName = "t_flz" + std::to_string(flzNo) + "_iom";
std::cout << "Relay no. " << std::to_string(flzNo) << " listen to "
<< iomTopicName << " and writes to " << cpmTopicName << std::endl;
publisher = node->create_publisher<test_lat_msg::msg::LatMsgCpm>
(cpmTopicName, rmw_qos_profile_sensor_data);
using namespace std::placeholders; // for `_1`
subscriber = node->create_subscription<test_lat_msg::msg::LatMsgIom>
(iomTopicName,
std::bind(&CpmRelay::cb_subscriber, this,_1),
rmw_qos_profile_sensor_data);
}
void
test::CpmRelay::cb_subscriber(const test_lat_msg::msg::LatMsgIom::SharedPtr msg)
{
// get relative receiving time
pupmsg.t2 = (std::chrono::steady_clock::now()-t_start).count();
// copy data
pupmsg.iomid = msg->iomid;
pupmsg.id = msg->id;
pupmsg.t1 = msg->t1;
pupmsg.payload = msg->payload;
// get relative sending time
pupmsg.t3 = (std::chrono::steady_clock::now()-t_start).count();
publisher->publish(pupmsg);
}
Feel free to ask for further information.
Kind regards,
Alex
Design document here.
The idea is to implement everything in the rmw
layer, and avoid changes in upstream layers as much as possible.
Tasks already done:
rmw_fastrtps
.rmw_dds_common
, to allow easiest porting of remaining rmw implementations.To do:
rmw_connext
(not needed, we changed to rmw_connextdds
which is one-participant-per-context)rmw_cyclonedds
Meta-ticket to track the progress in exposing more QoS settings.
Please update the tasklist below when a setting has been decided to be exposed.
Similar to previously added events like ros2/ros2#822 or on liveliness lost
, it would be good to add support to on sample lost
events (which might be really helpful when debugging communication issues).
For ROS
, on message lost
might be a more appropriate name.
For DDS based rmw implementations, implementing this should be pretty straightforward:
Detailed list of tasks: #226 (comment)
For rmw implementations that don't support the feature, they can return RMW_RET_UNSUPPORTED, as for example fastrtps already does when the creation of an ON_*_INCOMPATIBLE_QOS
event is requested.
DDS also support a status callback called on sample rejected. The difference with the previous one, is that this one reflects a reader not having resources to receive the sample and not a communication problem. I think it would be interesting to add support to that one too, maybe both can be wrapped in the same event?
Developer can use this interface to confirm reader receive all sent messages. For some use cases, this confirmation is very important.
wait_for_acknowledgments
is defined in DDS spec.
The description in DDS spec
2.2.2.4.1.12 wait_for_acknowledgments
This operation blocks the calling thread until either all data written by the reliable DataWriter entities is acknowledged by all
matched reliable DataReader entities, or else the duration specified by the max_wait parameter elapses, whichever happens
first. A return value of OK indicates that all the samples written have been acknowledged by all reliable matched data readers;
a return value of TIMEOUT indicates that max_wait elapsed before all the data was acknowledged.
If DDS exposes this interface, we can directly use (Such as FastDDS). If not, return unsupported.
https://github.com/ros2/rmw/blob/master/rmw/include/rmw/serialized_message.h is an extremely thin wrapper around rcutils_uint8_array_*
. The problem with this is that users expect that these functions return RMW_*
error codes, when in reality they return RCUTILS_*
error codes. Even the documentation is confused about this; it is documented as returning RMW_*
error codes, which is just incorrect.
My suggestion is that we make these static inline
functions instead, with a conversion between the RCUTILS
error codes that are returned from the rcutils_uint8_array
functions and the RMW
error codes that they are documented to return.
Currently, the API does not provide (or documents) a generic mechanism to wait for publishers (subscriptions) to match subscriptions (publishers) -- only the current match count is available for query. Thus, code that monitors these counts has to sleep and retry. That's inefficient and potentially brittle. It'd be nice to enable such waits.
It seems to me that a new pub/sub event would be the simplest solution. All Tier 1 RMW implementations have a form of pub/sub matching listener that can be leveraged:
rmw_fastrtps_cpp
and rmw_fastrtps_dynamic_cpp
already use it (see here and here).rmw_connext_cpp
already uses it (see here and here).rmw_cyclonedds_cpp
doesn't use it yet (see here).It's worth noting that some rcl
tests appear to rely on pub/sub matching triggering the graph guard condition. Regardless of that not being true for any of the current Tier 1 RMW implementations, it seems to me that would create too many unwarranted awakes. Also, pub/sub matching does not imply a graph change.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.