gazebosim / gz-transport

Transport library for component communication based on publication/subscription and service calls.

Gazebo Transport

Maintainer: caguero AT openrobotics DOT org

Build	Status
Test coverage
Ubuntu Jammy
Homebrew
Windows

Gazebo Transport, a component of Gazebo, provides fast and efficient asynchronous message passing, services, and data logging.

Table of Contents

Features

Install

Usage

Documentation

Testing

Folder Structure

Contributing

Code of Conduct

Versioning

License

Features

Gazebo Transport is an open source communication library that allows exchanging data between clients. In our context, a client is called a node. Nodes might be running within the same process in the same machine or in machines located in different continents. Gazebo Transport is multi-platform (Linux, Mac OS X, and Windows), so all the low level details, such as data alignment or endianness are hidden for you.

Gazebo Transport uses Google Protocol buffers as the data serialization format for communicating between nodes. Users can define their own messages using the Protobuf utils, and then, exchange them between the nodes. Gazebo Transport discovers, serializes and delivers messages to the destinations using a combination of custom code and ZeroMQ.

Install

See the installation tutorial.

Usage

See tutorials and the example directory in the source code.

Known issue of command line tools

In the event that the installation is a mix of Debian and from source, command line tools from gz-tools may not work correctly.

A workaround is to define the environment variable GZ_CONFIG_PATH to point to the location of the Gazebo library installation, where the YAML file for the package is found, such as

export GZ_CONFIG_PATH=/usr/local/share/gz

This issue is tracked here.

Documentation

Visit the documentation page.

Folder Structure

gz-transport
├── conf        Configuration file for the integration with the `gz` CLI tool.
├── docker      Dockerfile with gz-transport installed and scripts to build and run the code.
├── example     Example programs that use most of the Gazebo Transport API.
├── include     Header files that get installed.
├── log         All the code related with Gazebo Transport logging.
├── src         Source code of the core library.
├── test        A directory of integration, performance and regression tests.
└── tutorials   A set of tutorials about Gazebo Transport features.

Contributing

Please see CONTRIBUTING.md.

Code of Conduct

Please see CODE_OF_CONDUCT.md.

Versioning

This library uses Semantic Versioning. Additionally, this library is part of the Gazebo project which periodically releases a versioned set of compatible and complimentary libraries. See the Gazebo website for version and release information.

License

This library is licensed under Apache 2.0. See also the LICENSE file.

gz-transport's Issues

Remove ign-msgs dependency

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

It's used for testing but we can remove ign-msgs if we create a few custom protobuf messages.

protobuf ParseFromString function return value not checked

Original report (archived issue) by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

---

Migrated from gazebo issue 1022:

I noticed that the protobuf ParseFromString function is used frequently without checking its return value (bool) to see if it was successful.

Service requests without input parameter

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

Sometimes we want to receive some result but don't have any input parameter to send. We could use the msgs::Empty for not having to modify the wire format when sending a service request. We'd create new Request functions in the Node class without the input parameter (both synchronous and asynchronous versions make still sense).

Problem compiling on Homebrew/Mac

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

@caguero Is there any conflict between zeromq and czmq versions?

====== Finding 3rd Party Packages ======
-- Operating system is Apple MacOS X
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.28") 
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE  
-- Found PROTOBUF: /usr/local/lib/libprotobuf.dylib  
-- checking for module 'robot_msgs'
--   found robot_msgs, version 0.1.0
-- checking for module 'libzmq>=3.2.0'
--   found libzmq, version 3.2.4
-- Looking for zmq pkgconfig file - found
-- checking for module 'libczmq>=1.4.0'
--   found libczmq, version 2.0.3
-- Looking for czmq pkgconfig file - found
-- 	ronn not found, manpages won't be generated
-- ----------------------------------------

-- Checking ignition build type
-- Enable host CFlags
-- 
SSE4 disabled.

-- Enable upstream CFlags
-- Performing Test GCC_SUPPORTS_VISIBILITY
-- Performing Test GCC_SUPPORTS_VISIBILITY - Success
-- Performing Test R-Wall
-- Performing Test R-Wall - Success
-- Performing Test R-Wextra
-- Performing Test R-Wextra - Success
-- Performing Test R-Wno-long-long
-- Performing Test R-Wno-long-long - Success
-- Performing Test R-Wno-unused-value
-- Performing Test R-Wno-unused-value - Success
-- Performing Test R-Wfloat-equal
-- Performing Test R-Wfloat-equal - Success
-- Performing Test R-Wshadow
-- Performing Test R-Wshadow - Success
-- Performing Test R-Winit-self
-- Performing Test R-Winit-self - Success
-- Performing Test R-Wswitch-default
-- Performing Test R-Wswitch-default - Success
-- Performing Test R-Wmissing-include-dirs
-- Performing Test R-Wmissing-include-dirs - Success
-- Performing Test R-pedantic
-- Performing Test R-pedantic - Success
-- BUILD WARNINGS
-- 	ronn not found, manpages won't be generated
-- END BUILD WARNINGS

-- Custom CFlags:  -O3 -DNDEBUG -mssse3 -msse3 -msse -msse2  -std=c++11 -fvisibility=hidden -Wall -Wextra -Wno-long-long -Wno-unused-value -Wno-unused-value -Wno-unused-value -Wno-unused-value -Wfloat-equal -Wshadow -Winit-self -Wswitch-default -Wmissing-include-dirs -pedantic
-- Build Type: Release
-- Install path: /usr/local/Cellar/ign-transport/HEAD
-- Configuration successful. Type make to compile
  ignition_transport
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/ign-transport-hbXb/build
==> make install
Scanning dependencies of target ignition_transport
Scanning dependencies of target gtest
[  4%] [  9%] [ 14%] [ 19%] [ 23%] [ 28%] [ 33%] [ 38%] Building CXX object src/CMakeFiles/ignition_transport.dir/DiscoveryPrivate.cc.o
Building CXX object test/CMakeFiles/gtest.dir/gtest/src/gtest-all.cc.o
Building CXX object src/CMakeFiles/ignition_transport.dir/Discovery.cc.o
Building CXX object src/CMakeFiles/ignition_transport.dir/Node.cc.o
Building CXX object src/CMakeFiles/ignition_transport.dir/NodeShared.cc.o
Building CXX object src/CMakeFiles/ignition_transport.dir/TopicUtils.cc.o
Building CXX object src/CMakeFiles/ignition_transport.dir/TopicStorage.cc.o
Building CXX object src/CMakeFiles/ignition_transport.dir/Packet.cc.o
[ 42%] Building CXX object src/CMakeFiles/ignition_transport.dir/Uuid.cc.o
/tmp/ign-transport-hbXb/src/DiscoveryPrivate.cc:48:18: error: no matching function for call to 'zbeacon_new'
  this->beacon = zbeacon_new(this->ctx, this->DiscoveryPort);
                 ^~~~~~~~~~~
/usr/local/Cellar/czmq/2.0.3/include/zbeacon.h:39:5: note: candidate function not viable: requires single argument 'port_nbr', but 2 arguments were provided
    zbeacon_new (int port_nbr);
    ^
/tmp/ign-transport-hbXb/src/DiscoveryPrivate.cc:299:3: error: use of undeclared identifier 'zstr_free'; did you mean 'zstr_recv'?
  zstr_free(&srcAddr);
  ^~~~~~~~~
  zstr_recv
/usr/local/Cellar/czmq/2.0.3/include/zstr.h:36:5: note: 'zstr_recv' declared here
    zstr_recv (void *socket);
    ^
/tmp/ign-transport-hbXb/src/DiscoveryPrivate.cc:566:20: error: no matching function for call to 'zbeacon_new'
    zbeacon_t *b = zbeacon_new(this->ctx, this->DiscoveryPort);
                   ^~~~~~~~~~~
/usr/local/Cellar/czmq/2.0.3/include/zbeacon.h:39:5: note: candidate function not viable: requires single argument 'port_nbr', but 2 arguments were provided
    zbeacon_new (int port_nbr);
    ^
3 errors generated.
In file included from /tmp/ign-transport-hbXb/src/Node.cc:28:
/tmp/ign-transport-hbXb/include/ignition/transport/Node.hh:352:11: error: non-void function 'Request' should return a value [-Wreturn-type]
          return;
          ^
make[2]: *** [src/CMakeFiles/ignition_transport.dir/DiscoveryPrivate.cc.o] Error 1

Windows support

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

To be able to compile and successfully test the library in Windows.

It would be nice to provide a simple file easy to install.

Crash if IGN_IP is bad

Original report (archived issue) by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

---

I had originally ticketed this in handsim, but I think it's more related to ign-transport. But it might actually be not a problem with ignition itself, but a problem on how Gazebo is handling ignition.

If you set a bad IGN_IP, for example:

echo "export IGN_IP=123.456.789" >> ~/.bashrc

Then open Gazebo and try to insert the MPL arm, or open arat.world from handsim, and Gazebo crashes with this message:

Error setting socket option (IP_MULTICAST_IF).
NodeShared() Error: No such device

Proper implementation of PIMPL

Original report (archived issue) by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

---

It looks like the Discover class is aiming to use the PIMPL idiom to make it easier to preserve ABI. I've noticed a couple problems:

• DiscoveryPrivate.hh should not be included by Discovery.hh
• DiscoveryPrivate.hh should not be installed

I'll try fixing this when the Windows pull requests are merged.

Topic info command line doesn't list subscribers

Original report (archived issue) by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

---

To reproduce:

1. Compile examples

2. Start publisher, it publishes HELLO messages

3. Start subscriber, it receives HELLO messages

4. Check topic info with ign topic -i -t /foo and we see the publisher, but not the subscriber.

ign-transport doesn't exit cleanly in MATLAB in Windows

Original report (archived issue) by Brian Gerkey (Bitbucket: Brian Gerkey, GitHub: gerkey).

---

Originally a haptix-comm issue, worked around for now via PR58.

We should eventually figure out and fix the underlying issue.

Discovery test fails in the presence of other nodes

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

This test assumes an isolated environment. This is causing test fails in local machines or Jenkins when other Ignition Transport nodes are on the same LAN.

Pull request #61 solves this issue.

Create a deb package

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

For different versions of Ubuntu. It will be useful for testing the library easily.

Consider making topic name optional for Publish

Original report (archived issue) by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

---

To publish a topic, one needs to first advertise the topic:

node.Advertise<msg_type>(topic_name);

And then publish:

node.Publish(topic_name, msg)

This means you have to input the same topic_name in two or more different places, maybe needing to keep an extra variable to hold that topic name, for example.

Not sure if it is a good idea, but it could be convenient to have the topic be optional for publish, and the node automatically publishes to the first advertised topic that matches the message type. One drawback is that when a node advertises many topics with the same message type, it might be complicated to figure out that you need to specify the topic name.

A workaround for users advertising a single topic and not wanting to repeatedly write it right now could be to use Node::TopicList to get the topic every time, but unfortunately that's a blocking call.

Lots of compiler warnings in downstream code

Original report (archived issue) by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

---

When building haptix_comm, I get a lot of warnings about some unknown pragmas:

In file included from devel_isolated/ign-transport/include/ignition/transport.hh:3:0,
                 from src/haptix_comm/src/haptix.cc:20:
devel_isolated/ign-transport/include/ignition/transport/Discovery.hh:21:0: warning: ignoring #pragma warning  [-Wunknown-pragmas]
 #pragma warning(push, 0)
 ^

In file included from devel_isolated/ign-transport/include/ignition/transport.hh:3:0,
                 from src/haptix_comm/src/haptix.cc:20:
devel_isolated/ign-transport/include/ignition/transport/Discovery.hh:25:0: warning: ignoring #pragma warning  [-Wunknown-pragmas]
 #pragma warning(pop)
 ^

Support service responses beyond the service callback

Original report (archived issue) by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).

---

The current implementation only allows services to send a response directly from the callback function.

But there might be use cases when the service provider wants to, for example, put the request in a queue to be processed later. In this case, we'd want the response to be sent only after the request is fully processed.

This could be achieved with something like ROS actions.

Another idea (which might not make sense) is to let the user "end" the response anywhere in the code, calling response.end(), like for http requests.

consider asio networking

Original report (archived issue) by Andrew Hundt (Bitbucket: ahundt).

---

asio (also the same as boost.asio) is expected to be the basis for the C++ networking standard.

• latest standard proposal revision from nov 2014
• non-boost asio
• boost asio
• official asio zeromq implementation
• asio task priority queue example

asio provides a cross platform network interface, which means that the variety of networking headers currently included can be dropped down to just a couple. asio can be used to precisely control the number of threads, thread priority, message priority, and tasks that are being executed. asio also provides a deadline_timer which can be used to wait for exactly the right amount of time without stalling a threads' ability to execute other outstanding tasks.

There are currently places in the code where bugs such as the atomic-sleep problem occur. Search for sleep_for in the code to find most instances of this problem. The deadline_timer can be used to avoid those issues.

Thanks for taking this into consideration.

Message throttling on the subscriber

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

Implement message throttling in the subscriber to be able to control the number of messages delivered to the user.

Use logging system from ign_common when available

Original report (archived issue) by Nate Koenig (Bitbucket: Nathan Koenig).

---

Replace std::cout and std::cerr with logging functionality in ign_common

ccmake can't get past ignition-tools warning

Original report (archived issue) by Deanna Hood (Bitbucket: d_hood).

---

Calling ccmake .. without ignition-tools installed gives a warning which you can't get past to generate the makefile. Calling cmake directly displays the warning, but then makes fine.

Improve `NetUtils::determineHost`

Original report (archived issue) by Jackie K (Bitbucket: jacquelinekay).

---

I'm trying to use NetUtils::determineHost to get the current machine's public IP. If IGN_IP is not set, it keeps returning my private IP on interface eth1, which is on the subnet 192.168.0.0/16. determineHost should be changed to prefer IPs outside of this subnet, since it is reserved for private networks.

Visualize network topology

Original report (archived issue) by Nate Koenig (Bitbucket: Nathan Koenig).

---

Create a graphical tool to visualize the network topology.

Restrict maximum strings' length in a discovery message

Original report (archived issue) by nampi (Bitbucket: nampi).

---

This is a task similar to issue_43: Restrict maximum topic length (#43). We can add a restriction for all strings that are send in a discovery message.

Consider flatbuffers

Original report (archived issue) by Andrew Hundt (Bitbucket: ahundt).

---

I know you guys have been developing for over a year and this is probably too little, too late, but I wanted to point out some problems with protocol buffers and the replacement that was created to solve some of the exact problems that are encountered in robotics.

One of the major issues with protocol buffers in high performance systems is the substantial CPU overhead involved in packing and unpacking protobufs. Google themselves ran into this exact problem in high performance situations such as games, which have many of the same real-time requirements as robotics, particularly for multiplayer games. For this reason they created their own alternative to protobuf called flatbuffers.

https://google.github.io/flatbuffers/

Much of flatbuffers is very similar to protobuf, so it might actually be less unreasonable to move to flatbuffers than it would be for any other alternative. However, I realize less than extremely unreasonable may still be unreasonable. :-)

I'd like to hear any thoughts on this or if there is anything else useful I could add here.

Fail to communicate some protobufs

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

In this library, a protobuf message is serialized with msg.SerializeToString, and then, it's sent over the wire using zeromq. The C++ SerializeToString method might contain null bytes (\0) and as long as we store the serialized data into a std::string or we don't interpret the stream of bytes as a c-string, we are safe.

In several points of the code we were reconstructing a stream of bytes by using:

std::string data = std::string(reinterpret_cast<char *>(msg.data()));

This line was not storing the whole stream of bytes into data, it was storing only bytes up to the first \0.

The solution is to use an alternative std::string constructor that accepts two arguments (a pointer and size):

std::string data = std::string(reinterpret_cast<char *>(msg.data()), msg.size());

This problem was not appearing with protobuf messages containing a single field. However, if the message contains multiple fields, the bug was causing that incorrect messages were received by the subscribers. Probably protobuf uses \0 as an internal separator in its serializer.

Pull request #15 solves this problem.

Doxygen doc building fails

Original report (archived issue) by Javier V Gómez (Bitbucket: jvgomez).

---

Hello everyone,

I found a couple of problems when building doc with make doc. Extract of the terminal output:

[100%] Generating API documentation with Doxygen
**Warning: Tag `SYMBOL_CACHE_SIZE' at line 341 of file /home/jabuntu14/Desktop/ign-transport/build/ignition.dox has become obsolete.**
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
**Warning: doxygen no longer ships with the FreeSans font.**
You may want to clear or change DOT_FONTNAME.
Otherwise you run the risk that the wrong font is being used for dot generated graphs.
**Warning: tag INPUT: input source `/home/jabuntu14/Desktop/ign-transport/ignition/transport' does not exist**
Searching for include files...
Searching for files in directory /home/jabuntu14/Desktop/ign-transport/build
....

The last error causes index.html to be mostly useless. The fixes are easy. I was about to create a pull request. However, since I am new in this library, I want to be sure that this is a bug (probably I'm just missing something, actually I think it was designed to have all ignition packages together). The fixes I did to ignition.in are:

INPUT                  = @CMAKE_SOURCE_DIR@/doc/mainpage.html \
                         @CMAKE_SOURCE_DIR@/include/ignition/@IGN_PROJECT_NAME@
DOT_FONTNAME           = Helvetica
# SYMBOL_CACHE_SIZE      = 0

Additionally, I propose to set GENERATE_LATEX = NO.

Compressed data when publishing topics

Original report (archived issue) by nampi (Bitbucket: nampi).

---

We could add a new option to the AdvertiseOptions class for specifying that the content is compressed. We should use a compression library (zlib for example) and compress the data. I believe we should modify the wire format of the message when is published for adding a new flags field and set it the compressed flag. We should also decompress the message in the reception side.

P.D: idea and text from @caguero

Implement a service request from the command line using "ign"

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

Similar to this pull request used for publishing a message from the command line.

Communication problem when multiple network interfaces are up and using Scope::Host

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

This problem is located in the Discovery code. When binding to a socket for receiving discovery updates we use the function NetUtils::DetermineHost(), which works as expected. However, when sending discovery updates, the source IP address is chosen by the kernel and sometimes the IP address is different than the IP used in the bind(). This causes a problem when using Scope::Host because the IPs of sender/receiver don't match and we conclude the message comes from a different host discarding it.

Branch 'multicast' solves this issue and uses multicast instead of broadcast for the discovery messages.

Windows compilation broken on default branch

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

http://build.osrfoundation.org/job/ignition_transport-ci-default-windows7-amd64/57/

#!msvc


[ 14%] Building CXX object src/CMakeFiles/ignition-transport2.dir/Publisher.cc.obj
Publisher.cc
D:\Jenkins\workspace\ignition_transport-ci-default-windows7-amd64\workspace\ign-transport\src\Publisher.cc(118) : warning C4267: 'initializing' : conversion from 'size_t' to 'uint16_t', possible loss of data
[ 16%] Building CXX object src/CMakeFiles/ignition-transport2.dir/TopicUtils.cc.obj
TopicUtils.cc
D:\Jenkins\workspace\ignition_transport-ci-default-windows7-amd64\workspace\ign-transport\include\ignition/transport/TopicUtils.hh(74) : error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
D:\Jenkins\workspace\ignition_transport-ci-default-windows7-amd64\workspace\ign-transport\include\ignition/transport/TopicUtils.hh(74) : error C2146: syntax error : missing ';' before identifier 'kMaxNameLength'
D:\Jenkins\workspace\ignition_transport-ci-default-windows7-amd64\workspace\ign-transport\include\ignition/transport/TopicUtils.hh(74) : error C2143: syntax error : missing ';' before '='
D:\Jenkins\workspace\ignition_transport-ci-default-windows7-amd64\workspace\ign-transport\include\ignition/transport/TopicUtils.hh(74) : error C2238: unexpected token(s) preceding ';'
D:\Jenkins\workspace\ignition_transport-ci-default-windows7-amd64\workspace\ign-transport\src\TopicUtils.cc(29) : error C2065: 'kMaxNameLength' : undeclared identifier
D:\Jenkins\workspace\ignition_transport-ci-default-windows7-amd64\workspace\ign-transport\src\TopicUtils.cc(113) : error C2065: 'kMaxNameLength' : undeclared identifier
NMAKE : fatal error U1077: 'C:\PROGRA~2\MICROS~1.0\VC\bin\X86_AM~1\cl.exe' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.
Failed with error #2.

Are log files in scope?

Original report (archived issue) by Andrew Hundt (Bitbucket: ahundt).

---

Is the topic of log files in scope here? Since the data going over the transport is already serialized I expect it is very likely to be logged.

Consider appveyor for free Windows CI

Original report (archived issue) by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

---

We are using it successfully in ignition-math:

• https://github.com/ignitionrobotics/ign-math/blob/master/appveyor.yml
• https://osrf-migration.github.io/ignition-gh-pages/#!/ignitionrobotics/ign-math/commits/branch/default

It's also automatically displayed in the bitbucket build status API.

WaitForSubscribers in ign.cc

Original report (archived issue) by Nate Koenig (Bitbucket: Nathan Koenig).

---

Pull request #130 introduced a sleep here.

The sleep should be replace with a WaitForSubscribers call that is under development.

Write tutorials

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

Covering installation and usage.

Test suite fails when running in parallel

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

While building packages, test suite is run in parallel and fails:

To reproduce:

mkdir build
cd build
cmake ..
/usr/bin/ctest --force-new-ctest-process -j5

We would need to find the way of defining dependencies between check_ and real tests.

Compilation is broken on default branch for Xenial/Wily

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

Something in this list of changes broke the compilation on Ubuntu starting from Wily (including Xenial). Trusty is ok so it is probably a problem related to the lack of include headers in cmake and triggered by new versions of cmake or g++.

#!c++


[ 11%] Built target gtest
Scanning dependencies of target gtest_main
[ 12%] Building CXX object test/CMakeFiles/gtest_main.dir/gtest/src/gtest_main.cc.o
In file included from /usr/include/ignition/msgs0/ignition/msgs.hh:7:0,
                 from /var/lib/jenkins/workspace/ignition_transport-ci-default-xenial-amd64/ign-transport/src/ign.cc:25:
/usr/include/ignition/msgs0/ignition/msgs/Utility.hh:20:36: fatal error: ignition/math/Vector3.hh: No such file or directory
compilation terminated.
src/CMakeFiles/ignition-transport2.dir/build.make:134: recipe for target 'src/CMakeFiles/ignition-transport2.dir/ign.cc.o' failed
make[2]: *** [src/CMakeFiles/ignition-transport2.dir/ign.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 14%] Linking CXX static library libgtest_main.a

Mac OS X support

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

To be able to compile and successfully test the library in Mac OS.

It would be nice to provide a .dmg easy to install.

Communication blocked after some time doing request-response

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

If you request service calls in a loop, after 20k-50k calls the communications blocks. This is because I have been using a zeromq DEALER-DEALER pattern with a temporary socket used only for sending the requests. Each time there's a service request a new temporary DEALER socket is created and that was causing problems in zeromq. The solution is to redesign the way the service calls were requested by using ROUTER-ROUTER sockets and no temporary sockets. Now, there's one ROUTER socket for sending requests, one ROUTER socket for receiving service requests and sending responses, and one ROUTER socket for receiving the responses.

ROUTER sockets need the zeromq identity of the destination, so this information has been added into the discovery information.

Pull request #23 addresses the problem.

Compilation error advertising a service call using a member function as callback

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

Due to templates and the lack of a test for this method, we didn't realize that there is a compilation problem.

#!c++

/home/caguero/workspace/ign-transport/include/ignition/transport/Node.hh:273:55: error: no matching function for call to ‘ignition::transport::RepHandler<ignition::msgs::StringMsg, ignition::msgs::StringMsg>::RepHandler(std::string&)’
           new RepHandler<T1, T2>(this->dataPtr->nUuid));

Pull request #16 solves the problem.

Restrict maximum topic length

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

Right now there's no restriction on the maximum length of a topic. There's a potential attack if you use a huge topic name in any operation that accepts a string. We could limit the maximum string size to 256 (as DDS does).

http://stackoverflow.com/questions/27970321/limit-on-dds-topic-names

We also have to check the topic lengths when reading from the network. This will involve to modify some of the Unpack() methods in the Packet class.

P.D: idea and text from @caguero

`request()` fails sporadically

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

While running multiple stress tests in parallel doing request/response I found that sometimes Request() fails.

It fails at the beginning of the test and successive calls to request() on the same topic continue failing.

This bug was introduced in the NodeShared constructor. NodeShared is the object that creates the Discovery object. Once the discovery has been created, NodeShared registers the callbacks to receive discovery notifications.

The problem appears if the discovery learns the information about the service to be requested before NodeShared registers its discovery callbacks. During Request(), the address of the node that offers the service is known but the zmq socket of the requester is not connected to the zmq socket of the responser. This connection occurs in the service callback that NodeShared missed.

One possible solution is to decouple the object discovery creation from its activation. NodeShared can instantiate the discovery object, register its callbacks, and then, start the service.

Type checking

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

When a node advertises a topic, it should specify a type:

#!c++

node.Advertise<TYPE>("topic"). 

When the same node publishes a message with:

#!c++

node.Publish("topic", msg)

We should check that msg matches before publishing the message.

On the other side of the communication, the subscriber also registers a callback that has a specific type. Ex:

#!c++

void cb(const std::string &_topic, const robot_msgs::StringMsg &_msg)
{
 ...
}

node.Subscribe("topic", cb)

When a new message arrives to the subscriber, we should check that the message received and the message expected in the callback matches. As a bonus, we could send a hash string (MD5 or SHA) of the message definition to make sure that the message version is the same between publisher and subscriber.

Allow node communication outside the LAN

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

Right now, the discovery is using UDP broadcast. An an option (environment variable), the user might include an additional list of remote IP addresses that the discovery should take into account.

Publisher Subscriber Example

Original report (archived issue) by Robert Walenta (Bitbucket: TwoBit01).

---

Hi Ignition Team,

making my first steps with ign-transport lib. After installing it and trying to build the pub/sub example from source, it had first some issues with the proto file that could not be build. After installing all other binary dependencies, I guess libignition-msgs* was missing, this error was solved.

After a new compile run the Publisher method is throwing now an error and not excepting the msg type, here the compile output:

#!c++
Scanning dependencies of target protobuf_compilation
[ 33%] Running C++ protocol buffer compiler on stringmsg.proto
[ 33%] Built target protobuf_compilation
Scanning dependencies of target publisher
[ 66%] Building CXX object CMakeFiles/publisher.dir/publisher.cc.o
/home/twobit/gazebo_ws/socket_comm/ign_transport/publisher.cc: In function ‘int main(int, char**)’:
/home/twobit/gazebo_ws/socket_comm/ign_transport/publisher.cc:66:33: error: no matching function for call to ‘ignition::transport::Node::Publish(bool&, example::msgs::StringMsg&)’
     if (!node.Publish(pubId, msg))
                                 ^
/home/twobit/gazebo_ws/socket_comm/ign_transport/publisher.cc:66:33: note: candidate is:
In file included from /usr/include/ignition/transport1/ignition/transport.hh:9:0,
                 from /home/twobit/gazebo_ws/socket_comm/ign_transport/publisher.cc:24:
/usr/include/ignition/transport1/ignition/transport/Node.hh:125:20: note: bool ignition::transport::Node::Publish(const string&, const ProtoMsg&)
       public: bool Publish(const std::string &_topic,
                    ^
/usr/include/ignition/transport1/ignition/transport/Node.hh:125:20: note:   no known conversion for argument 1 from ‘bool’ to ‘const string& {aka const std::basic_string<char>&}’
make[3]: *** [CMakeFiles/publisher.dir/publisher.cc.o] Error 1
make[2]: *** [CMakeFiles/publisher.dir/all] Error 2
make[1]: *** [CMakeFiles/publisher.dir/rule] Error 2
make: *** [publisher] Error 2

After that I tried to use a predefine message and included:

#!c++
  #include <ignition/msgs0/ignition/msgs/stringmsg.pb.h>

But after using the ignition::msgs::StringMsg Class the compiler is complaining about a missing <ignition/msgs/System.hh>

I would really like to work with ign-transport but the tutorial:
[http://ignition-transport.readthedocs.io/en/latest/installation/installation.html](Link URL)
should be refactored.

Implement data encryption

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

Use the 0MQ encryption capabilities to be able to publish encrypted messages.

Service requests without response

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

Not all the service requests require a response. We'd add some logic for removing the service request handler after sending the request, instead of waiting for the response. We could use the msgs::Empty for not having to modify the wire format when sending a service request. We'd create a new Request function in the Node class with only one parameter (the input parameter).

P.D: idea and text from Carlos Agüero

Circular Buffers

Original report (archived issue) by Andrew Hundt (Bitbucket: ahundt).

---

One key element for communication in robotics applications that I have substantial experience with is the use of circular buffers rather than queues when processing messages. Typically the latest sensor data coming in matters more than the oldest, so if you can only process half the images or laser scans, the most recent are more important.

A circular buffer facilitates this by automatically dropping data that is too old to matter. Additionally it has the side effect that it becomes impossible for the data to fill up all of the memory and cause a crash, which is something that is very possible with a queue.

This is a field tested technique that has been proven crucial across several generations of high performance robots developed for multiple commercial customers.

Here are a couple pages with images I found that illustrate the idea. The linked library is not actually something you would want to utilize because it appears to be a Java library. Nonetheless, the images are useful to convey the point.

https://fabric3.atlassian.net/wiki/display/FABRIC/Ring+Buffer+%28Disruptor%29+Channels

https://fabric3.atlassian.net/wiki/display/FABRIC/ZeroMQ+and+Ring+Buffer+(Disruptor)+Channels

The fundamentals of the system you are implementing look amazing at a high level. However circular buffers, a higher performance protocol, and prioritization are killer features for a system that can span beyond research projects into high performance commercial products on inexpensive hardware.

twoProcSrvCall.ThousandCalls test is failing from time to time

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

Comes from pull request #23. While running the test on Ubuntu Trusty, sometimes is failing with this message:

[ RUN      ] twoProcSrvCall.ThousandCalls
/home/jrivero/code/ignition/ign-transport/test/integration/twoProcessesSrvCallStress.cc:61: Failure
Value of: node.Request(topic, req, timeout, response, result)
  Actual: false
Expected: true
/home/jrivero/code/ignition/ign-transport/test/integration/twoProcessesSrvCallStress.cc:65: Failure
Value of: response.data()
  Actual: 5646
Expected: i
Which is: 5647
/home/jrivero/code/ignition/ign-transport/test/integration/twoProcessesSrvCallStress.cc:61: Failure
Value of: node.Request(topic, req, timeout, response, result)
  Actual: false
Expected: true
/home/jrivero/code/ignition/ign-transport/test/integration/twoProcessesSrvCallStress.cc:65: Failure
Value of: response.data()
  Actual: 5646
Expected: i
Which is: 5648

Implement compression

Original report (archived issue) by Carlos Agüero (Bitbucket: caguero, GitHub: caguero).

---

Give the option to compress the message when Publish().

Failing test in auxiliary testing file

Original report (archived issue) by Jose Luis Rivero (Bitbucket: Jose Luis Rivero, GitHub: j-rivero).

---

As noted by @scpeters, there is one failure in the auxiliary file used for testing.

A part from the failing test, we need to find a way of making the result of the test to fail in presence of errors in auxiliary functions.

Need new release

Original report (archived issue) by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).

---

The most recent release (0.8.1) doesn't work with the latest version of zeromq in homebrew (version 4.1.3). Since this was just added as a dependency of gazebo7, it is causing the gazebo homebrew builds to fail. It would help to make a new release as soon as possible just to fix the gazebo build.

Message Delivery and handling unreliable connections robustly

Original report (archived issue) by Andrew Hundt (Bitbucket: ahundt).

---

Is the design being built to handle unreliable and/or time delayed connections robustly?

Another crucial component of a messaging system is handling unreliable connections robustly. If connections begin to drop packets it is often undesirable to resend the data because the new data will be arriving anyway.

If data transfer becomes constricted, particularly due to lost packets, it is important to allow prioritization so that small messages containing the most important data can get through, such as motion commands and heartbeats. Images and other large data can be critical, but depending on the application the priorities really matter and systems can often operate in a reduced functionality mode.

The best combination for this is a circular buffer for receiving messages and a priority queue (or really the circular buffer equivalent would be better) for sending messages.

Are you using buffer disassembly and reassembly via UDP? TCP breaks the requirements of many robotics applications and fails in the wrong way when packets are dropped and the throughput drops below the rate data is being sent. The resending of packets can also cause a connection to collapse when if it simply dropped packets the way UDP does later data would get through successfully.

Also, don't simply take my word for all of this search the literature on robust communication in robotics and you will be able to reach a conclusion independently as well. As mentioned in the other issue the fundamentals of the system you are implementing look amazing at a high level. However circular buffers, a higher performance protocol, and prioritization are killer features for a system that can span beyond research projects into high performance commercial products on inexpensive hardware.

Related issues:
https://osrf-migration.github.io/ignition-gh-pages/#!/ignitionrobotics/ign-transport/issues/24/circular-buffers (#24)
https://osrf-migration.github.io/ignition-gh-pages/#!/ignitionrobotics/ign-transport/issues/23/consider-flatbuffers (#23)