facontidavide / ros_type_introspection Goto Github PK
View Code? Open in Web Editor NEWDeserialize ROS messages that are unknown at compilation time
License: MIT License
Deserialize ROS messages that are unknown at compilation time
License: MIT License
I am trying to use ros_type_introspection to get a header of different image messages.
Found that Parser::applyVisitorToBuffer() method is slow for sensor_msgs/Image message type when image have high resolution (1920x1080).
Code snippet for subscriber callback:
void TopicMonitor::topicCallback(const topic_tools::ShapeShifter::ConstPtr& msg, const string
&topicName, const unsigned int topicIndex)
{
const string& datatype = msg->getDataType();
const string& definition = msg->getMessageDefinition();
parser_.registerMessageDefinition(topicName, RosIntrospection::ROSType(datatype), definition);
static vector<uint8_t> buffer;
buffer.resize(msg->size());
ros::serialization::OStream stream(buffer.data(), buffer.size());
msg->write(stream);
std_msgs::Header header;
const RosIntrospection::Parser::VisitingCallback deserializeHeader =
[&header](const RosIntrospection::ROSType&, absl::Span<uint8_t>& buffer)
{
ros::serialization::IStream is( buffer.data(), buffer.size() );
ros::serialization::deserialize(is, header);
};
const RosIntrospection::ROSType header_type(ros::message_traits::DataType<std_msgs::Header>::value());
static absl::Span<uint8_t> buffer_span(buffer);
parser_.applyVisitorToBuffer(topicName, header_type, buffer_span, deserializeHeader);
...
}
Thanks.
Hello @facontidavide! I appreciate you for this useful library and tutorials which let me use your tool without any problem.
I'm trying to building a type agnostic bridge to publish a vector of subscribed topics of interest without disturbing the type of the subscribed message. I have problems to deserialize and move the subscribed data without knowing the type in advance to my publishers. FlatContainers are good but I would like to make my topic publications practically easier to use in other functional components.
Thanks.
Curious any plans to release it to noetic
distro? https://github.com/ros/rosdistro/tree/master/noetic
The ROS wiki example for creating a generic subscriber does not work for ROS Kinetic. I ended up using the example code provided in type_introspection_tests
Here's the question I asked on the issue before posting here. The example code on ROS Wiki needs to be updated for Kinetic (and newer I guess)
ReadFromBuffer does not know the size of buffer_ptr
ros_type_introspection/src/parser.cpp
Lines 116 to 210 in 81fd041
If the string_size
is larger than the remaining contents of buffer_ptr
, this should be an error. It is not safe to trust the user supplied data.
ros_type_introspection/src/deserializer.cpp
Lines 64 to 65 in 81fd041
when building, if you change the headers, you can run into the project being compiled against the installed headers rather than the current source headers.
I think this is more robust:
include_directories(SYSTEM
${catkin_INCLUDE_DIRS}
)
include_directories(BEFORE
include
)
Is there also a way to serialize a message from RenamedValues
, i.e. the reverse process of parsing?
The StringTree inside ROSTypeFlat is updated every time the BuildRosFlatType is called. Actually it is unique for any kind of messagesm, therefore there is no reason to build it every time.
Move the creation of that StringTree into BuildROSTypeMapFromDefinition
Parser::applyVisitorToBuffer
is an extremely useful function.
If we can control the traversal from within the visiting callback and decide whether it should continue or terminate, that will be awesome!
I think if the callback has bool as the return value, the implementation will be easy.
Any idea?
ros_type_introspection/src/deserializer.cpp
Lines 163 to 166 in 81fd041
stderr
would be better, but I don't want to be forced to see that either.
If this is about the buffer being too small and so it only partially parsed, that seems like possibly desirable behavior, so a warning is kind of annoying.
Maybe it would be better to return the required array_size instead? Possibly return bool regarding whether it was completely processed or not.
buildRosFlatType(...., array_size, &required_array_size);
// at least the user can control the output now
LOG_IF(WARNING, required_array_size > array_size) << "buffer too small, results will be truncated";
For instance, I'm mostly interested in only parsing the Header
and not the entire message.
Non-templated methods in shape_shifter.hpp
should be marked inline or moved to a module, otherwise, can result in multiple definitions if included by multiple files in a single project.
https://github.com/facontidavide/ros_type_introspection/tree/dev
@ibtaylor @mehditlili , you are the users that contributed in the past, you might want to take a look before I merge into master.
I rewrote the library using a different interface.
Originally I used a stateless API with 3 C like function.
The new approach use a class named Parser because in this way it is easier to keep a cache of some computed values as private members of the class.
https://github.com/facontidavide/ros_type_introspection/blob/dev/include/ros_type_introspection/ros_introspection.hpp
The new implementation outperform the old one by a factor 2X.
Still the most expensive step is applyNameTransform, that takes much more than deserializeIntoFlatContainer.
The new implementation was tested successfully against all the rosbags I commonly parse for testing (and the unit tests, of course).
I was trying to compile PlotJuggler from source after cloning the latest versions of PlotJuggler, ros_type_introspection and abseil-cpp (your catkinized version) and got errors for this package. I am using catkin_tools
to build the workspace and the one "non-standard" thing that I did was to enable the install option (catkin config --install
) for the workspace. The output I got is as follows:
----------------------------------------------------------------------------------
Profile: default
Extending: [env] /opt/ros/melodic
Workspace: /home/kartikmohta/programs/ros/plotjuggler_ws
----------------------------------------------------------------------------------
Source Space: [exists] /home/kartikmohta/programs/ros/plotjuggler_ws/src
Log Space: [missing] /home/kartikmohta/programs/ros/plotjuggler_ws/logs
Build Space: [exists] /home/kartikmohta/programs/ros/plotjuggler_ws/build
Devel Space: [exists] /home/kartikmohta/programs/ros/plotjuggler_ws/devel
Install Space: [missing] /home/kartikmohta/programs/ros/plotjuggler_ws/install
DESTDIR: [unused] None
----------------------------------------------------------------------------------
Devel Space Layout: linked
Install Space Layout: merged
----------------------------------------------------------------------------------
Additional CMake Args: -DCMAKE_BUILD_TYPE=RelWithDebInfo
Additional Make Args: None
Additional catkin Make Args: None
Internal Make Job Server: True
Cache Job Environments: False
----------------------------------------------------------------------------------
Whitelisted Packages: None
Blacklisted Packages: None
----------------------------------------------------------------------------------
Workspace configuration appears valid.
NOTE: Forcing CMake to run for each package.
----------------------------------------------------------------------------------
[build] Found '3' packages in 0.0 seconds.
Starting >>> catkin_tools_prebuild
Finished <<< catkin_tools_prebuild [ 2.8 seconds ]
Starting >>> abseil_cpp
Finished <<< abseil_cpp [ 13.6 seconds ]
Starting >>> ros_type_introspection
__________________________________________________________________________________________________________________________________________________________________________________________________________________
Errors << ros_type_introspection:cmake /home/kartikmohta/programs/ros/plotjuggler_ws/logs/ros_type_introspection/build.cmake.000.log
CMake Error at /home/kartikmohta/programs/ros/plotjuggler_ws/install/share/abseil_cpp/cmake/abseil_cppConfig.cmake:148 (message):
Project 'ros_type_introspection' tried to find library 'absl_container'.
The library is neither a target nor built/installed properly. Did you
compile project 'abseil_cpp'? Did you find_package() it before the
subdirectory containing its code is included?
Call Stack (most recent call first):
/opt/ros/melodic/share/catkin/cmake/catkinConfig.cmake:76 (find_package)
CMakeLists.txt:7 (find_package)
cd /home/kartikmohta/programs/ros/plotjuggler_ws/build/ros_type_introspection; catkin build --get-env ros_type_introspection | catkin env -si /usr/bin/cmake /home/kartikmohta/programs/ros/plotjuggler_ws/src/ros_type_introspection --no-warn-unused-cli -DCATKIN_DEVEL_PREFIX=/home/kartikmohta/programs/ros/plotjuggler_ws/devel/.private/ros_type_introspection -DCMAKE_INSTALL_PREFIX=/home/kartikmohta/programs/ros/plotjuggler_ws/install -DCMAKE_BUILD_TYPE=RelWithDebInfo; cd -
..................................................................................................................................................................................................................
Failed << ros_type_introspection:cmake [ Exited with code 1 ]
Should I not be using the latest abseil-cpp
from https://github.com/Eurecat/abseil-cpp?
Abseil, a C++ library built by Google, has been released a couple of days ago.
https://github.com/abseil/abseil-cpp
It has a lot of interesting features that will make the code of this library more expressive.
I am planning to build a catkinized version of abseil and depend on it
Another strange performance point (when profiling with perf) seemed to be in buildRosFlatTypeImpl
.
Two things I would do would be to remove the usage of std::function
Use a stack to keep up with the tree parent (think that's all that would be needed) rather than using recursion.
I'm working on a patch, but thought I would make an issue in case I am too slow.
Hi, I have problems with some message types.
terminate called after throwing an instance of 'std::runtime_error'
what(): unsupported builtin type value
I used the generic_subscriber code and tested it on ros melodic with the packaged version 1.3.1-0bionic.20190325.150517 and version 1.4.0 from github.
Hi,
Regarding the function
Note the const qualifier.
I suddenly had trouble compiling PlotJuggler with ROS, giving me build errors of the form:
In file included from <..>/plotjuggler/ws_plotjuggler/src/PlotJuggler/plugins/ROS/RosoutPublisher/rosout_publisher.cpp:3:
In file included from <..>/plotjuggler/ws_plotjuggler/src/PlotJuggler/plugins/ROS/RosoutPublisher/../shape_shifter_factory.hpp:4:
/opt/ros/kinetic/include/ros_type_introspection/utils/shape_shifter.hpp:207:31: error:
no matching constructor for initialization of
'ros::serialization::IStream'
ros::serialization::IStream s(msgBuf_.data(), msgBuf_.size() );
^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/opt/ros/kinetic/include/ros/serialization.h:749:3: note: candidate constructor
not viable: 1st argument ('const unsigned char *') would lose const
qualifier
IStream(uint8_t* data, uint32_t count)
^
Which was odd, as I had it compiling fine before, without any obvious changes to my setup. As far as I can understand though, the compiler correctly asserts that accessing the vector data in the implementation
through the vector<>::data()
function from within the const method instantiate
indeed returns the const pointer const uint8_t*
:
https://en.cppreference.com/w/cpp/container/vector/data
Which constructs an IStream object which (for kinetic at least), in serialization.h is not const:
http://docs.ros.org/kinetic/api/roscpp_serialization/html/serialization_8h_source.html#l00709
Removing the const
qualifier from instantiate
resulted in a working build.
Now, obviously there is something with my setup that triggers this now as I had it building before and no one else has had this issue compiling introspection or plotjuggler. However, should this method really be const?
Best,
Kristian
Hey,
first off, thank you for working on this!
However, I ran into a problem. I'm trying to build a tree from the deserialized message in FlatMessage
. I assumed the tree member would be a perfect fit and it is partly.
I can build my tree by traversing it but I don't see how I can retrieve the values from the leaves.
The values are stored as pairs of StringTreeLeaf
and the value in FlatMessage
.
But how do I obtain the StringTreeLeaf
representing the StringTreeNode
.
It looks like I have to go through the values vector for each leaf node to obtain the value which is far from optimal.
To be honest, I don't really see the point of the StringTreeLeaf
struct.
For example, why does it have a fixed size index array?
Is there a limit for ros messages that they can't be more than 8 layers deep?
Why isn't the mapping in FlatMessage
just from StringTreeNode
's that are leaves to values?
And why is it a vector of pairs instead of a map?
Best,
Stefan
These builtin types are not given actual values like (BOOL=0,BYTE=1,...
)
maybe better to use runtime asserts, and an unordered map (instead of compares)
inline const char* toStr(const BuiltinType& c)
{
switch (c) {
case BOOL: return "BOOL";
case BYTE: return "BYTE";
case INT8: return "INT8";
case CHAR: return "CHAR";
case UINT8: return "UINT8";
case UINT16: return "UINT16";
case UINT32: return "UINT32";
case UINT64: return "UINT64";
case INT16: return "INT16";
case INT32: return "INT32";
case INT64: return "INT64";
case FLOAT32: return "FLOAT32";
case FLOAT64: return "FLOAT64";
case TIME: return "TIME";
case DURATION: return "DURATION";
case STRING: return "STRING";
case OTHER: return "OTHER";
}
LOG(FATAL) << "unsupported builtin type value:" << static_cast<int>(c);
}
// this should only be called if BuiltinType has a size, otherwise is a programmer error
inline std::size_t builtinSize(const BuiltinType c) {
DCHECK(HasSize(c));
switch (c) {
case BOOL:
case BYTE:
case INT8:
case CHAR:
case UINT8: return 8;
case UINT16:
case INT16: return 16;
case UINT32:
case INT32:
case FLOAT32: return 32;
case UINT64:
case INT64:
case FLOAT64:
case TIME:
case DURATION: return 64;
case STRING:
case OTHER:
}
LOG(FATAL) << "invalid call to builtinSize for type:" << toStr(c);
}
inline BuiltinType toBuiltinType(const std::string& s) {
static std::unordered_map<std::string, BuiltinType> string_to_builtin_map {
{ "bool", BOOL },
{ "byte", BYTE },
{ "char", CHAR },
{ "uint8", UINT8 },
{ "uint16", UINT16 },
{ "uint32", UINT32 },
{ "uint64", UINT64 },
{ "int8", INT8 },
{ "int16", INT16 },
{ "int32", INT32 },
{ "int64", INT64 },
{ "float32", FLOAT32 },
{ "float64", FLOAT64 },
{ "time", TIME },
{ "duration", DURATION },
{ "string", STRING },
};
CHECK(!s.empty());
auto it = string_to_builtin_map.find(s);
return it == string_to_builtin_map.cend() ? OTHER : it->second;
}
Compiling
void printMessage(const RosIntrospection::FlatMessage& flat_container)
{
for (auto& it : flat_container.value)
if (it.second.convert<std::uint8_t>()) // error value, of type franka_msgs::Errors::_tau_j_range_violation_type
ROS_ERROR(" - %s", it.first.toStdString().c_str()); // error name
}
with gcc 7.4.0 -Wall -O3 returns:
<...>/source.cpp: In function ‘void ns::printMessage(const RosIntrospection::FlatMessage&)’:
<...>/source.cpp:892:5: warning: ‘target’ may be used uninitialized in this function [-Wmaybe-uninitialized]
if (it.second.convert<std::uint8_t>())
Looking into it, I found -Wmaybe-uninitialized's cause and tried to see if I'm in that case, but I can't see any issues.
Now I'm not discounting being wrong, but if I add a default case to the convert function in variant.hpp, the warning disappears:
case OTHER: throw TypeException("Variant::convert -> cannot convert type" + std::to_string(_type)); break;
default: throw TypeException("Variant::convert -> cannot convert type" + std::to_string(_type)); break;
}
return target;
Also, I hadn't realized until this last copy-paste, but target
is exactly the return variable of the convert function.
I'd actually suggest changing the OTHER
type (actually most of the times I see enums, the last one is the total number of enumerated cases, i.e. an ALL_TYPES
in this case. It'd probably be useless though) with the default
case.
EDIT: no, not changing the OTHER
type, but introducing the default
where you anyways throw
Starting with the removal of abseil, if an executable is compiled that includes, for example, helper_functions.hpp
I now get warnings about comparisons between signed and unsigned integers (assuming -Wsign-compare
is passed to catkin_make
):
In file included from /home/jarvis/catkinws/src/ros_type_introspection/src/ros_type.cpp:37:0:
/home/jarvis/autoyard/src/ext/ros_type_introspection/include/ros_type_introspection/helper_functions.hpp: In function ‘void RosIntrospection::ReadFromBuffer(RosIntrospection::Span<unsigned char>&, size_t&, T&) [with T = std::__cxx11::basic_string<char>; RosIntrospection::Span<unsign
ed char> = nonstd::span_lite::span<unsigned char, -1l>; size_t = long unsigned int]’:
/home/jarvis/catkinws/src/ros_type_introspection/include/ros_type_introspection/helper_functions.hpp:95:28: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if( offset + string_size > buffer.size())
As far as I can see, the message parser needs registration of messages with full-fledged definitions, including all definitions of sub messages - as they are provided by ros::message_traits::Definition<ros_type>::value()
. So far, the use case of this lib seems to process arbitrary messages received externally together with this full-fledged message definition.
I intended to use the lib to parse serialized messages, where I only knew the ros type (in form of pkg_name/type_name
) and I was disappointed that this doesn't work out of the box.
I think, by looking up msg definitions from the (ROS) file system, utilizing
roslib::getPath(), you can easily handle this use case as well. Probably it will be a valuable extension to augment the existing API with a registerMessageDefinition(const ROSType &ros_type)
.
Hi,
I just came across this repo, great stuff !
Is there any way this can be used to write a function fileToMsg
, that would basically do in c++ what
rostopic pub /topic type_of/msg -f /to/file.yml
does ?
In the same fashion as RosIntrospection::SubstitutionRule
it would be nice to have a RosIntrospection::FilterRule
to filter out unwanted variables.
Use case 1
I get a Twist Stamped but I am only interested on the twist, then I apply a filter to remove the header part. Once filtered, I can apply Substitution Rules to rename the vars if wanted.
Use case 1
I get a sensor_msgs/BatteryState but I am only interested in the percentage. I apply the filter and can use only the data I am interested in.
Ideal features
I wanted to do some changes to the source code and switched from using the deb package to master. However, my ros node depending on ros_type_introspection did not compile anymore.
I took a closer look and noticed that the renamed_values vector doesn't exist anymore in the ROSTypeFlat structure. The class VarNumber doesn't overload the operator double() anymore and buildRosFlatType now requires a fifth parameter max_array_size.
Is there a reason for those changes?
I have a use case that I want to check a message if it has Header
field, then deserialize only this part to get the timestamp in it. Can I use this library to implement this efficiently?
Check if this helps to speed up computation....
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.