ros / genpy Goto Github PK

genpy

EmberScript 1.44% Python 98.15% CMake 0.42%

genpy's Introduction

Robot Operating System (ROS)
===============================================================================

ROS is a meta-operating system for your robot.  It provides
language-independent and network-transparent communication for a
distributed robot control system.

Installation Notes
------------------

For full installation instructions, including system prerequisites and
platform-specific help, see:

  http://wiki.ros.org/ROS/Installation

genpy's People

Contributors

Stargazers

Watchers

Forkers

mkjaergaard gerkey kwc severin-lemaignan kargm yasushi3 flixr pierrickkoch aurelienballier hidof-forks ayrton04 requesttimedout k-okada abencz haas85 furushchev aballier jpetereit masasin lamby iwanders dhood eric-wieser metamemoryt asmodehn dorianscholz robbeofficial pschillinger gavanderhoorn aambrosano jspricke snrkiwi kartikmohta chenrui2014 ros-hunter hunter-packages ernestmc cclauss mgrrx nnsuite haraisao mcerdeiro stan-quinn tzickel pedromduarte bromeara jimwelagu tsukuyomih2 mikepurvis thekobithirdparty mintar tailosinc sloretz georgealbertson-bc gbiggs matejbartosovic peci1 at-wat belorenz bigolol woz4tetra

genpy's Issues

Problems with Message definitions containing backslashes in comments

Given you have some message definition like the following (SomeMessage.msg):

# \brief Some description
# \note Something I really want to stress
float64 x

... genpy will not escape the slashes when writing the _full_text member for the message serialization / deserialization classes.

In fact, a bagfile will contain the invalid message definition:

Header header
#rief Some description
# 
ote Something I really want to stress
float64 x

... invalidating the stored message and md5hash.

For the given example you can try this out with:

from your_package.msg import SomeMessage
constructed_msg = SomeMessage()
# output the message definition: without the fix you should see \b and especially \n screwing up the message definition,
# introducing a weird new "ote" field for the "# \note" comment
print(constructed_msg._full_text)
with rosbag.Bag("/tmp/test.bag", "w") as bag:
  bag.write("/some_topic", constructed_msg, genpy.Time.from_sec(0))
with rosbag.Bag("/tmp/test.bag") as bag:
  # attempt to read the message again. You will get an empty message and a md5sum-error without the fix
  next(bag.read_messages(topics=["/some_topic"])

On 32-bit systems, rostime.Time.from_sec() fails on times later than january 19 2038.

This error can be reproduced in python like this:

genpy.rostime.Time.from_sec(2**31 + 0.1)

or on the command line like this:

rostopic pub /test std_msgs/Header "{stamp: 2147483648000001000}"

The error message looks like:

File "/opt/ros/groovy/lib/python2.7/dist-packages/genpy/rostime.py", line 217, in from_sec
    return Time(secs, nsecs)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/genpy/rostime.py", line 191, in init
    super(Time, self).__init__(secs, nsecs)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/genpy/rostime.py", line 67, in init
    raise ValueError("if secs is a float, nsecs cannot be set")
ValueError: if secs is a float, nsecs cannot be set

It looks like the problem arises in the TVal.__init__() function of rostime.py:

def init(self, secs=0, nsecs=0):
        """ 
        :param secs: seconds. If secs is a float, then nsecs must not be set or 0, ``int/float``
        :param nsecs: nanoseconds, ``int``
        """ 
        if type(secs) != int:
        # float secs constructor
            if nsecs != 0:
                raise ValueError("if secs is a float, nsecs cannot be set")
...

Here the assumption is that secs is never a long. Time.from_sec(), however, calls this code on a long:

        secs = int(float_secs)
        nsecs = int((float_secs - secs) * 1000000000)
        return Time(secs, nsecs)

Note that secs is a long on a 32-bit system if float_secs is greater than or equal to 2**31.

pip install fails with latest version of pip

Using the latest version of pip v23.1, it seems that pip install from this repo now fails:

# start in a venv with the required packages like catkin_pkg already installed
pip install --upgrade pip # upgrade to latest pip
pip install git+https://github.com/ros/genpy

This fails with the error

ModuleNotFoundError: No module named 'catkin_pkg'

On the other hand

# start in a venv with the required packages like catkin_pkg already installed
pip install pip==23.0.1 # use the previous version of pip
pip install git+https://github.com/ros/genpy

works.

Not sure what is going on.

rospy.rostime.Time and Duration have inaccurate representations

The introspection is odd:

In [8]: d=rospy.Duration(0)

In [9]: d
Out[9]: genpy.Duration[0]

In [10]: rospy.Time(1)
Out[10]: genpy.Time[1000000000]

In [11]: t = rospy.Time(1)

In [12]: rospy.Time(1,1)
Out[12]: genpy.Time[1000000001]

In [13]: rospy.Time(1,1) - rospy.Time(1)
Out[13]: genpy.Duration[1]

In [14]: type(t)
Out[14]: rospy.rostime.Time

In [15]: "%s"%t
Out[15]: '1000000000'

It would be nice to fix this. It's also represented in rqt topic introspection.

improve serialization/deserialization struct error

To reproduce, using two shells(e.g.):

rostopic echo /footest
rostopic pub /footest  "{x: 0.0, y: 0.0, z: 0.0.1}" 
[WARN] [WallTime: 1377942406.224566] Inbound TCP/IP connection failed: required argument is not a float

Note the topic listener is required, else no packing happens. (Sidenote: I am not sure whether this should be a warning, seems more like an error to me.)

The problem here is that the message does not contain a hint to what value was not a float. I looked at the code genpy generates for messages / service, which looks like this:

 for val1 in ...:
    _v1 = ...
    _v2 = _v1....
    _x = _v2
    buff.write(_struct...(_x....)

At least the value of _x should IMO be used in the case of errors, to indicate the writing of what failed. In my case (not using geometry_msgs/Point but a more complex message) that would have reduce me looking for the error a lot.

So a change to genpy would be to change the generated code something like this:

- except struct.error as se: self._check_types(se)
- except TypeError as te: self._check_types(te)
+ except struct.error as se: self._check_types(self._check_types(ValueError("%s: %s when writing %s" % (type(se), str(se), str(_x))))
+ except TypeError as te: self._check_types(self._check_types(ValueError("%s: %s when writing %s" % (type(te), str(te), str(_x)))))

Also see:
http://stackoverflow.com/questions/696047/re-raising-exceptions-with-a-different-type-and-message-preserving-existing-inf

Problem with customized message

I meet a bug when using rosbag filter with a customized message. I've located the bug to this function: https://github.com/ros/genpy/blob/indigo-devel/src/genpy/dynamic.py#L81

def _gen_dyn_modify_references(py_text, types):

I have a customized message like this:

my_message_package/Odometry
  std_msgs/uint8 flags
  std_msgs/string info
  nav_msgs/Odometry odom
  ......

Unfortunately, as is shown above, my customized message is my_message_package/Odometry, while there is a standard nav_msgs/Odometry. The package names are different, but the type names are the same.

The generated .py script is attached:

"""autogenerated by genpy from my_message_package/Odometry.msg. Do not edit."""
import sys
python3 = True if sys.hexversion > 0x03000000 else False
import genpy
import struct

class _nav_msgs__Odometry(genpy.Message):
  _md5sum = "94d99f86002b25504a5d3354fa1ad709"
  _type = "my_message_package/Odometry"
  _has_header = False #flag to mark the presence of a Header object
.............

The function def _gen_dyn_modify_references(py_text, types) just simply substitutes the class name. In my situation, the class Odometry(genpy.Message) should be substituted with _my_message_package__Odometry, but the nav_msgs/Odometry is first touched during iteration, so it is substituted with _nav_msgs__Odometry

In order to solve the problem, I add few lines to tell this function what is the current working package. See this pull request. That means that after the comment line """autogenerated by genpy from my_message_package/Odometry.msg. Do not edit.""", now_pkg is my_message_package, not nav_msgs/Odometry, such that it would not substitute this Odometry with _nav_msgs__Odometry. When the next comment line is detected, the now_pkg will change. The detection is simply a re-expression.

But the solution is not so "elegant". I'd like to see if anyone has more elegant way to fix this.

genpy.Time comparison problem in Python 3 (Melodic)

With Melodic and Python3 (and Python 2 with newer versions of Numpy), I'm having problems with genpy.Time instances initialized from (sec, nano) values which are of type numpy.int64. Comparisons (<, >, <=, >=) fail with a TypeError except for ==.

This works fine in Python 2 and older versions (1.13.3 tested) of Numpy. However, newer versions (1.16.6 tested) fail with a different TypeError that states a deprecation of boolean subtraction in numpy.

Also, initializing with a numpy.float64 works just fine.

Same issue with rospy.Time.

Code:

import numpy as np
import genpy 

a = np.array([1583968688, 8586883])
print('Type of a[0]: {}'.format(type(a[0])))

# Fails in Python 3 with np.int64 type
t = genpy.Time(a[0], a[1])
print('\nt: {}'.format(t))
print('Compare t < t: {}'.format(t < t))

# Works in Python 3 with regular int type
t2 = genpy.Time(int(a[0]), int(a[1]))
print('\nt2: {}'.format(t2))
print('Compare t2 < t2: {}'.format(t2 < t2))

Python 3.6.9 / numpy 1.18.1:

Type of a[0]: <class 'numpy.int64'>

t: 1583968688008586883
Traceback (most recent call last):
  File "rosTimeProblem.py", line 9, in <module>
    print('Compare t < t: {}'.format(t < t))
TypeError: '<' not supported between instances of 'Time' and 'Time'

Python 2.7.17 / numpy 1.13.3:

Type of a[0]: <type 'numpy.int64'>

t: 1583968688008586883
Compare t < t: False

t2: 1583968688008586883
Compare t2 < t2: False

Python 2.7.17 / numpy 1.16.6:

Type of a[0]: <type 'numpy.int64'>

t: 1583968688008586883
Traceback (most recent call last):
  File "rosTimeTest.py", line 10, in <module>
    print('Compare t < t: {}'.format(t < t))
  File "/opt/ros/melodic/lib/python2.7/dist-packages/genpy/rostime.py", line 270, in __cmp__
    return (a > b) - (a < b)
TypeError: numpy boolean subtract, the `-` operator, is deprecated, use the bitwise_xor, the `^` operator, or the logical_xor function instead.

constructor for rosbridge_suite objects

I think it'll be useful to add constructors which can instantiate the python object from a json object.
When we work with rosbridge_suite we lose the byte array.

memoryview / buffer inputs should be accepted as serializable data

Let's say I have a large numpy array that represents an std_msgs/Image.data

Today if I try to publish such a message it will fail in this part of the serialize function:

      if type(_x) in [list, tuple]:
        buff.write(struct.pack('<I%sB'%length, length, *_x))
      else:
        buff.write(struct.pack('<I%ss'%length, length, _x))

So I would have to to .tostring() or something which will do a copy (and then another copy in struct.pack).

I think that a check should be added if it's a memoryview (py3) or buffer (py2), and then only struct.pack the length, and then simply buff.write the object itself directly.

misleading ROSSerializationException in some cases

If you e.g. have a message like this:

string foo
float64 bar

and you try to serialize {"foo": "foo_string", "bar": "not_a_float"} you get a misleading exception like

<class 'struct.error'>: 'required argument is not a float' when writing 'foo_string'

instead of what would be correct:

<class 'struct.error'>: 'required argument is not a float' when writing 'not_a_float'

This seems to be because the local var _x is not set before every new field, but this is printed since f42fa96

The solution could be as simple as always setting _x before the next field is serialized.

Python 3.11 compatibility with unittest methods

The deprecated unittest aliases were removed in python/cpython#28268 . The following replacements can be made for Python 3.11 compatibility.

assertEquals -> assertEqual
failIf -> assertFalse

test/test_genpy_rostime.py:        self.assertEquals(0, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.failIf(v) # test __zero__
test/test_genpy_rostime.py:        self.assertEquals('0', str(v))
test/test_genpy_rostime.py:        self.assertEquals(0, v.to_nsec())
test/test_genpy_rostime.py:        self.assertEquals(0, v.to_sec())
test/test_genpy_rostime.py:        self.assertEquals(v, v)
test/test_genpy_rostime.py:        self.assertEquals(v, TVal())
test/test_genpy_rostime.py:        self.assertEquals(v, TVal(0))
test/test_genpy_rostime.py:        self.assertEquals(v, TVal(0, 0))
test/test_genpy_rostime.py:        self.assertEquals(v.__hash__(), TVal(0, 0).__hash__())
test/test_genpy_rostime.py:        self.assertEquals(0, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(1000000000, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.failIf(v.is_zero())
test/test_genpy_rostime.py:        self.assertEquals('1000000000', str(v))
test/test_genpy_rostime.py:        self.assertEquals(1000000000, v.to_nsec())
test/test_genpy_rostime.py:        self.assertEquals(v, v)
test/test_genpy_rostime.py:        self.assertEquals(v, TVal(1))
test/test_genpy_rostime.py:        self.assertEquals(v, TVal(1, 0))
test/test_genpy_rostime.py:        self.assertEquals(v, TVal(0,1000000000))
test/test_genpy_rostime.py:        self.assertEquals(v.__hash__(), TVal(0,1000000000).__hash__())
test/test_genpy_rostime.py:        self.assertNotEquals(v, TVal(0, 0))
test/test_genpy_rostime.py:        self.assertNotEquals(v.__hash__(), TVal(0, 0).__hash__())
test/test_genpy_rostime.py:        self.assertEquals(NotImplemented, v.__ge__(0))
test/test_genpy_rostime.py:        self.assertEquals(NotImplemented, v.__gt__(Foo()))
test/test_genpy_rostime.py:        self.assertEquals(NotImplemented, v.__ge__(Foo()))
test/test_genpy_rostime.py:        self.assertEquals(NotImplemented, v.__le__(Foo()))
test/test_genpy_rostime.py:        self.assertEquals(NotImplemented, v.__lt__(Foo()))
test/test_genpy_rostime.py:        self.failIf(v.__eq__(Foo()))
test/test_genpy_rostime.py:        self.assertEquals(0, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.nsecs)
test/test_genpy_rostime.py:        self.failIf(v.is_zero())
test/test_genpy_rostime.py:        self.assertEquals('1', str(v))
test/test_genpy_rostime.py:        self.assertEquals(1, v.to_nsec())
test/test_genpy_rostime.py:        self.assertEquals(v, v)
test/test_genpy_rostime.py:        self.assertEquals(v, TVal(0,1))
test/test_genpy_rostime.py:        self.assertNotEquals(v, TVal(0, 0))
test/test_genpy_rostime.py:        self.assertEquals(2, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(2, v.to_sec())
test/test_genpy_rostime.py:        self.assertEquals(2000000000, v.to_nsec())
test/test_genpy_rostime.py:        self.assertEquals(2, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(2.000000001, v.to_sec())
test/test_genpy_rostime.py:        self.assertEquals(2000000001, v.to_nsec())
test/test_genpy_rostime.py:        self.assertEquals(0, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(0.000000001, v.to_sec())
test/test_genpy_rostime.py:        self.assertEquals(1, v.to_nsec())
test/test_genpy_rostime.py:            self.assertEquals(-2, v.secs)
test/test_genpy_rostime.py:            self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:            self.assertEquals(-2, v.to_sec())
test/test_genpy_rostime.py:            self.assertEquals(-2000000000, v.to_nsec())
test/test_genpy_rostime.py:            self.assertEquals(-1, v.secs)
test/test_genpy_rostime.py:            self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:            self.assertEquals(-1, v.to_sec())
test/test_genpy_rostime.py:            self.assertEquals(-1000000000, v.to_nsec())
test/test_genpy_rostime.py:        self.assertEquals(TVal(1).__hash__(), TVal(1).__hash__())
test/test_genpy_rostime.py:        self.assertEquals(TVal(1,1).__hash__(), TVal(1,1).__hash__())
test/test_genpy_rostime.py:        self.assertNotEquals(TVal(1).__hash__(), TVal(2).__hash__())
test/test_genpy_rostime.py:        self.assertNotEquals(TVal(1,1).__hash__(), TVal(1,2).__hash__())
test/test_genpy_rostime.py:        self.assertNotEquals(TVal(1,1).__hash__(), TVal(2,1).__hash__())
test/test_genpy_rostime.py:        self.failIf(failed, "should have failed to compare")
test/test_genpy_rostime.py:        self.failIf(failed, "should have failed to compare")
test/test_genpy_rostime.py:        self.failIf(failed, "negative time not allowed")
test/test_genpy_rostime.py:        self.failIf(failed, "negative time not allowed")
test/test_genpy_rostime.py:        self.assertEquals(v.to_sec(), t)
test/test_genpy_rostime.py:        self.assertEquals(Time.from_sec(0), Time())
test/test_genpy_rostime.py:        self.assertEquals(Time.from_sec(1.), Time(1))
test/test_genpy_rostime.py:        self.assertEquals(Time.from_sec(v.to_sec()), v)
test/test_genpy_rostime.py:        self.assertEquals(v.from_sec(v.to_sec()), v)
test/test_genpy_rostime.py:        self.assertEquals(v.to_sec(), v.to_time())
test/test_genpy_rostime.py:        self.failIf(failed, "Time + Time must fail")
test/test_genpy_rostime.py:        self.assertEquals(Time(2, 0), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(2, 0), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(2, 2), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(2, 2), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(2), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(2), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(400, 100), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(400, 100), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(400, 400), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(400, 400), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(399, 999999999), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(399, 999999999), v)
test/test_genpy_rostime.py:        self.failIf(failed, "Time - non Duration must fail")
test/test_genpy_rostime.py:        self.failIf(failed, "Time - non TVal must fail")
test/test_genpy_rostime.py:        self.assertEquals(Time(), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(2, 2), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(1), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(100, 100), v)
test/test_genpy_rostime.py:        self.assertEquals(Time(99, 999999999), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(0,100), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(-100), v)
test/test_genpy_rostime.py:        self.assertEquals(Time.from_sec(0.5), Time(0.5))
test/test_genpy_rostime.py:        self.assertEquals(0, t.secs)
test/test_genpy_rostime.py:        self.assertEquals(500000000, t.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(v, Duration.from_sec(v.to_sec()))
test/test_genpy_rostime.py:        self.assertEquals(v, v.from_sec(v.to_sec()))
test/test_genpy_rostime.py:        self.assertEquals(v, Duration.from_sec(v.to_sec()))
test/test_genpy_rostime.py:        self.assertEquals(v, v.from_sec(v.to_sec()))
test/test_genpy_rostime.py:        self.assertEquals(-1, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(1, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(999999999, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(Duration(1,0) + Time(1, 0), Time(2, 0))
test/test_genpy_rostime.py:        self.failIf(failed, "Duration + int must fail")
test/test_genpy_rostime.py:        self.assertEquals(2, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(Duration(2, 0), v)
test/test_genpy_rostime.py:        self.assertEquals(0, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(Duration(), v)
test/test_genpy_rostime.py:        self.assertEquals(2, v.secs)
test/test_genpy_rostime.py:        self.assertEquals(0, v.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(Duration(2), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(400, 100), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(400, 400), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(399, 999999999), v)
test/test_genpy_rostime.py:        self.failIf(failed, "Duration - non duration must fail")
test/test_genpy_rostime.py:        self.failIf(failed, "Duration - Time must fail")
test/test_genpy_rostime.py:        self.assertEquals(Duration(), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(-3, 999999998), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(1), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(-200, 100), v)
test/test_genpy_rostime.py:        self.assertEquals(Duration(-201, 999999999), v)
test/test_genpy_rostime.py:        self.assertEquals(abs(Duration()), Duration())
test/test_genpy_rostime.py:        self.assertEquals(abs(Duration(1)), Duration(1))
test/test_genpy_rostime.py:        self.assertEquals(abs(Duration(-1)), Duration(1))
test/test_genpy_rostime.py:        self.assertEquals(abs(Duration(0,-1)), Duration(0,1))
test/test_genpy_rostime.py:        self.assertEquals(abs(Duration(-1,-1)), Duration(1,1))
test/test_genpy_rostime.py:        self.assertEquals(abs(Duration(0,1)), Duration(0,1))
test/test_genpy_rostime.py:        self.assertEquals(Duration.from_sec(0.5), Duration(0.5))
test/test_genpy_rostime.py:        self.assertEquals(0, t.secs)
test/test_genpy_rostime.py:        self.assertEquals(500000000, t.nsecs)
test/test_genpy_rostime.py:        self.assertEquals(Duration(4), Duration(2) * 2)
test/test_genpy_rostime.py:        self.assertEquals(Duration(4), Duration(2) * 2.)
test/test_genpy_rostime.py:        self.assertEquals(Duration(4), 2 * Duration(2))
test/test_genpy_rostime.py:        self.assertEquals(Duration(4), 2. * Duration(2))
test/test_genpy_rostime.py:        self.assertEquals(Duration(10), Duration(4) * 2.5)
test/test_genpy_rostime.py:        self.assertEquals(Duration(4, 8), Duration(2, 4) * 2)
test/test_genpy_rostime.py:        self.assertEquals(Duration(4), Duration(8) / 2)
test/test_genpy_rostime.py:        self.assertEquals(Duration(4), Duration(8) / 2.)
test/test_genpy_rostime.py:        self.assertEquals(Duration(4, 2), Duration(8, 4) / 2)

Integer division of durations

Today I was surprised to find that integer division of Duration converts to floats internally, possibly loosing precision. My use case for using Duration over "seconds as a float" is to have arithmetic accurate to the ns. This is for example an issue when averaging durations. For example, I have a bag file where I want to manipulate some topics. Some recorded timestamps are from an arduino clock, i.e. in the order of a few thousand seconds. I want to map this to the recorded timestamps (~ current unix time), and for this I am averaging a bunch of time differences that have a large absolute value, but are within milliseconds and I am very much interested in their ns level precision for the avarage.

The following illustrates the issue:

In [50]: s = Duration(100000000,1)

In [51]: s
Out[51]: rospy.Duration[100000000000000001]

In [52]: (s+s)/2
Out[52]: genpy.Duration[100000000000000000]

In [53]: (s+s)//2
Out[53]: genpy.Duration[100000000000000000]

In [54]: np.mean([s, s])
Out[54]: genpy.Duration[100000000000000000]

The current implementation comes from #59 with previous discussion in #49.

Would it make sense to do this arithmetic with int nanoseconds instead float at least when divisor is an integral type?

Don't log full message on serialization errors?

We recently had a bug in a python script which made the serialization of a sensor_msgs/Image fail. The result was that the full image was logged to rosout, making the logs grow by multiple megabytes per second.

  File "/[...]/python3.8/site-packages/genpy/message.py", line 385, in _check_types
    raise SerializationError(str(exc))
genpy.message.SerializationError: <class 'TypeError'>: 'string argument expected, got 'bytes'' when writing 'header: 
  seq: 234328
  stamp: 
    secs: 1670536260
    nsecs: 942930698
  frame_id: "camera_down_infra1_frame"
height: 240
width: 320
encoding: "mono8"
is_bigendian: 0
step: 320
data: [4, 3, 3, 4, 3, 4, 3, 4, 4, 4, 3, 3, 4, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 4, 3, 3, 4, 3, 3, 3, 3, 4, 3, 4, 3, 4, 3, 3, 3, 3, 3, 3, 4, 4, 3, 3, 4, 3, 4, 3, 3, 3, 3, 3, 4, 4, 3, 3, 4, 3, 3, 3, 4, 4, 3, 3, 4, 4, 3, 4, 3, 3, 3, 3, 3, 3, 4, 3, 4, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 4, 4, 4, 4, 4, ...

This is bad for log management and also bad for SSDs.

I'd like to have the option to not log full messages to rosout on serialization errors.
Alternatively, the logging module in rospy could be changed to be able to enforce a maximum log message size.
What do you think?

Support for special float values

I think this genmsg issue is more appropriate here.

In cases where constants in messages are set to special floats, for example:
float32 a = NaN
float32 p = Inf
float32 n = -Inf 
The successfully generated Python script declare those as:
a = nan
p = inf
n = -inf
However, those causes NameError as nan, inf don't exist.

The following works in Python:
a = float('nan')
p = float('inf')
n = float('-inf')

A suggestion is to add the lines

nan = float('nan')
inf = float('inf')

at the top of the output of msg_generator:

genpy/src/genpy/generator.py

Lines 817 to 821 in 5008399

    
           yield '# This Python file uses the following encoding: utf-8' 
        
           yield '"""autogenerated by genpy from %s.msg. Do not edit."""' % spec.full_name 
        
           yield 'import sys' 
        
           yield 'python3 = True if sys.hexversion > 0x03000000 else False' 
        
           yield 'import genpy\nimport struct\n'

Another solution would be to have special handling of these types of values at

genpy/src/genpy/generator.py

Line 864 in 5008399

yield ' %s = %s' % (c.name, c.val)

Can't generate message CameraInfo

The message type CameraInfo depends on the sensor_msgs, however, in the function in dynamic.py

def _gen_dyn_modify_references(py_text, current_type, types):
...
        for t in types:
                pkg, base_type = genmsg.package_resource_name(t)
                gen_name = _gen_dyn_name(pkg, base_type)
                # - remove any import statements
                py_text = py_text.replace('import %s.msg' % pkg, '')
...

This line will erase the statement of importing sensor_msgs.msg, because the variable pkg here is sensor_msg. Then the message fails to generate.
Does this aim to avoid importing some message type repeatedly?

check_type does not check float

Problem: The check_type function doesn't check float32 or float64 fields. Here's a comparison with the (correct) int64 behavior:

>>> import genpy.message

################## Floats ##################

# F1. This correctly passes
>>> genpy.message.check_type('x', 'float64', 42.0)

# F2. This should fail ('foobar' is not serializable as float64); during serialization, a generic SerializationError will be thrown:
# genpy.message.SerializationError: <class 'struct.error'>: 'required argument is not a float' when writing 'header: ...'
>>> genpy.message.check_type('x', 'float64', 'foobar')

# F3. This should fail (although np.float64(42) *is* serializable as float64, so unless the user explicitly calls check_type, serialization will work)
>>> genpy.message.check_type('x', 'float64', np.float64(42.0))

################## Ints ##################

# I1. This correctly passes
>>> genpy.message.check_type('x', 'int64', 42)

# I2. This correctly fails ('foobar' is not serializable as int64); during serialization, check_type throws a specific SerializationError:
# genpy.message.SerializationError: field header.seq must be unsigned integer type
>>> genpy.message.check_type('x', 'int64', 'foobar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/genpy/message.py", line 206, in check_type
    raise SerializationError('field %s must be an integer type'%field_name)
genpy.message.SerializationError: field x must be an integer type

# I3. This correctly fails (although np.int64(42) *is* serializable as int64, so unless the user explicitly calls check_type, serialization will work)
>>> genpy.message.check_type('x', 'int64', np.int64(42))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/genpy/message.py", line 206, in check_type
    raise SerializationError('field %s must be an integer type'%field_name)
genpy.message.SerializationError: field x must be an integer type

Solution: There's a elif field_type in ['float32', 'float64']: ... missing about here:

genpy/src/genpy/message.py

Line 264 in e27b6fd

elif field_type == 'bool':

Discussion: I believe adding a check for floats would be the correct behavior. However, I don't believe that many people use the check_type function directly. The normal workflow for most users is this:

user creates message object and fills fields
user calls publish(), which causes the message to be serialized
if serialization fails, check_type is automatically called to print a specific error message.

The impact of this bug in this scenario is only that check_type throws a very unspecific error for floats, while it does print a specific error for all other types.

A different scenario would be if a user directly calls check_type to determine if a message can be serialized. In that case, this bug doesn't detect type errors in float fields. In other words, currently, not all messages that pass check_type can be correctly serialized. On the other hand, check_type currently also rejects some messages (with numpy types) that actually can be serialized.

If we fix this bug, check_type will be consistent between floats and the other types, and it will no longer accept messages that cannot be serialized. However, it will reject even more messages that could be serialized. What do you think? Do you want me to provide a PR for this, or do we leave it as it is? I don't have any strong opinions either way.

Duration.rmul is not implemented

>>> 2 * Duration(3.2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for *: 'int' and 'Duration'

rospy.rostime.Time.from_seconds() can produce wrong conversion

When generating the msg header stamp with the Time.from_seconds() the converted result can be wrong, the following is one example

from rospy.rostime import Time
from sensor_msgs.msg import Imu
timestamp_ns = 10117469579
imu_msg = Imu() 
imu_msg.header.stamp = Time.from_seconds(timestamp_ns / 1e9)
assert Time.to_nsec(imu_msg.header.stamp) == timestamp_ns, '{} vs. {}'.format(Time.to_nsec(imu_msg.header.stamp), timestamp_ns)
---
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: 10117469578 vs. 10117469579

Time.nonzero is ill-defined

What's special about 1970? Why is 1970-1-1 00:00 considered to be false, and everything else true?

I'd argue that __nonzero__ (and consequently, __bool__, is_zero) should only be implemented on Duration objects

Time message doc/code inconsistency

I just noticed :

http://wiki.ros.org/msg says : time | secs/nsecs unsigned 32-bit ints | ros::Time | rospy.Time
https://github.com/ros/genpy/blob/kinetic-devel/src/genpy/rostime.py#L191 and https://github.com/ros/genpy/blob/kinetic-devel/src/genpy/rostime.py#L59 which seems to indicate that Time is implemented with signed integers... at least for python.

So what is it in reality ? should I expect time secs max value to be 2147483647 (signed max 32 bits) or 4294967295 (unsigned max 32 bits).

nested types - AttributeError: 'dict' object has no attribute '...'

Hi!

we are currently trying to use nested types for service responses.
we have the values as a dict and it looks like, the auto-generated "_type.py" has problems with nested dicts..

I get following error:
File ".../groovy_dev_workspace/devel/lib/python2.7/dist-packages/rosbridge_library/srv/_MirorCalculateTipfollowingTrajectoryTest.py", line 300, in serialize
buff.write(_struct_d.pack(self.snake_arm_config_trajectory.data))
AttributeError: 'dict' object has no attribute 'data'

if I change that line to:
buff.write(_struct_d.pack(self.snake_arm_config_trajectory["data"]))
everything seems to work.

I did not have much time to check into this, but I thought I should ask here at first..
I checked some ways to make the nested dict accessable by "dot"-operator but did not yet find any good solution.

the srv-file I am using uses std_msgs/Float64 as response type. this defines a "data" field (/object) for the value that I want to store. this nested object is not accessable, as you can see in the error message above.

Is there any known solution, how I could convert my nested dict to make it work?
or do you think this will need changes within genpy?
or am I doing something completely wrong?

if you need to, you can see my code at:
https://github.com/ipa-fxm-db/rosbridge_suite/tree/genpy_problem

to see the problem "in action":
you can start the rosbridge instance with:

roslaunch rosbridge_server rosbridge_tcp.launch

and the test that is causing this error:

first: rosbridge_suite/rosbridge_library/test/tcp+complex_srv/test_non-ros_service_server_complex-srv.py
and second: rosbridge_suite/rosbridge_library/test/tcp+complex_srv/test_non-ros_service_client_complex-srv.py

the problematic nested dict is passed in line 358 of file .../advertise_service.py

Thanks for reading!

PyPI release ?

How about doing a pypi release of genpy ?

It would allow a user from python to generate ROS message classes by calling genpy.genpy_main.genmain directly from any python script (ROS or not), and would make interfacing with ROS from other python frameworks easier, I think...

My current usecase is that I want to run test on generated python messages classes using tox.

Depending on a debian package, that need to be installed on a specific system, and then accessed through a bash script that setup a environment variable, is quite overkill...

If I had genpy as a pip installable package, generating messages just before testing would be much easier, and platform independent...

message classes with many fields contain syntax error

When a message contains more than 255 consecutive simple fields the generate code contains a SyntaxError: more than 255 arguments since all fields are passes as arguments to the pack function.

generate_dynamic creates millions of tmp directories

I'm creating an application that reads in ros messages from a ros bag.

During the process of reading messages, generate_dynamic(core_type, msg_cat) is being called to get the message type definition.

In the function generate_dynamic, temp directories are being created as a temporary location for dynamically generated message classes.

# Create a temporary directory
tmp_dir = tempfile.mkdtemp(prefix='genpy_')

then, uses atexit to remove the directory when the main module terminates.

# Afterwards, we are going to remove the directory so that the .pyc file gets cleaned up if it's still around
atexit.register(shutil.rmtree, tmp_dir)

but.....

my application is not meant to terminate and must run indefinitely.

In addition, atexit does not get called if the following happens:

the program dies because of a signal
os._exit() is invoked directly
a Python fatal error is detected (in the interpreter)

As a result, millions of genpy_XXXXXX directories are being created for each message my application receives in the \tmp directory of my machine which later causes my machine to crash.

Do you have any suggestions on how to solve this issue?
I'm thinking that there should be a check if there is an existing message type module before creating a new msg type module.

genpy tests can't use std_msgs in their tests as it's not in the dependency list

Error output:
http://jenkins.willowgarage.com:8080/view/groovy_devel/job/devel-groovy-genpy-precise-amd64/8/testReport/junit/test.test_genpy_message/MessageTest/test_Message_check_types/

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 327, in run
    testMethod()
  File "/tmp/2012-09-11_17.35.54.905907/genpy/test/test_genpy_message.py", line 57, in test_Message_check_types
    from std_msgs.msg import String, UInt16MultiArray, MultiArrayLayout, MultiArrayDimension
ImportError: No module named std_msgs.msg

Message._fill_message_args() fails on messages with fields of type 'time[]'

(on genpy version 0.4.10, ros_comm version 1.9.44)

To reproduce: define a message "rosbugs/msg/TimeArray.msg" with a single field:

time[] stamps

Build it and then try:

rostopic pub /test ros_bugs/TimeArray "{stamps: []}"

This fails with the error:

Traceback (most recent call last):
  File "/opt/ros/groovy/bin/rostopic", line 35, in <module>
    rostopic.rostopicmain()
  File "/opt/ros/groovy/lib/python2.7/dist-packages/rostopic/__init__.py", line 1654, in rostopicmain
    _rostopic_cmd_pub(argv)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/rostopic/__init__.py", line 1330, in _rostopic_cmd_pub
    argv_publish(pub, msg_class, pub_args, rate, options.once, options.verbose)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/rostopic/__init__.py", line 1354, in argv_publish
    publish_message(pub, msg_class, pub_args, rate, once, verbose=verbose)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/rostopic/__init__.py", line 1224, in publish_message
    genpy.message.fill_message_args(msg, pub_args, keys=keys)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/genpy/message.py", line 513, in fill_message_args
    _fill_message_args(msg, msg_args[0], keys, '')
  File "/opt/ros/groovy/lib/python2.7/dist-packages/genpy/message.py", line 461, in _fill_message_args
    _fill_val(msg, f, v, keys, prefix)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/genpy/message.py", line 425, in _fill_val
    list_msg_class = get_message_class(base_type)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/genpy/message.py", line 570, in get_message_class
    cls = _get_message_or_service_class('msg', message_type, reload_on_error=reload_on_error)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/genpy/message.py", line 532, in _get_message_or_service_class
    raise ValueError("message type is missing package name: %s"%str(message_type))
ValueError: message type is missing package name: time

It looks like this could be fixed by inserting a new case around line 420 of message.py, handling the case when base_type is a time type.

exception in serialize uses undefined _x variable

Filled originally as ros/ros_comm#705 by @felixduvallet:

Several of the std_msgs implementations use an undefined _x variable in serialize(). For example, _Int32.py has:

def serialize(self, buff):
    """
    serialize message into buffer
    :param buff: buffer, ``StringIO``
    """
    try:
      buff.write(_struct_i.pack(self.data))
    except struct.error as se: self._check_types(struct.error("%s: '%s' when writing '%s'" % (type(se), str(se), str(_x))))
    except TypeError as te: self._check_types(ValueError("%s: '%s' when writing '%s'" % (type(te), str(te), str(_x))))

Any exception in publishing the message will try to access str(_x), where _x is undefined. As a comparison, _String.py defines:

_x = self.data

A minimal failing example is a publisher that attempts to send a message without casting it properly:

import rospy
import std_msgs

rospy.init_node('talker')
pub = rospy.Publisher('/foo', std_msgs.msg.Int32, queue_size=1)
rospy.sleep(1)
msg = std_msgs.msg.Int32('1')  # NOTE: the type is incorrect
pub.publish(msg)  # This should raise a struct.error

The equivalent call for a String message will correctly raise an exception:
rospy.exceptions.ROSSerializationException: field data must be of type str

ImportError: No module named yaml - ROS NOETIC

Hello!
I got the following error using ros noetic on ubuntu20

File "/opt/ros/noetic/lib/python3/dist-packages/genpy/message.py", line 48, in
import yaml
ImportError: No module named yaml

=============================
I tryed install pyyaml usisng pip install pyyaml, pip3 install pyyaml and the error persists.
I tried a lot to find a solution for it in stackoverflow and etc but nothing works.
Thanks!

Generate type hints for ROS messages and services

I'm sorry if this is a duplicated issue.

Hello.
As you may know, Python has the type hints system, which allows us to hint the type of objects, and it is available for not only Python3 but Python2.
By adding type hints, we can use some great tools which make our Python development better, e.g., static type checking by mypy.
I personally add type hints to my code, and if genpy generates type hints for ROS messages and services, that would be much appreciated.

Is there any plan to add type hints to the generate code?
If no one is working on this feature, I would love to work on it.

Header default to rospy.Time.now()

If the stamp of a header is left unspecified it defaults to genpy.Time() which sets secs and nsecs to 0.

Would it not be better/possible to have it default to the rosmaster time?

Ideally, I think it would be best if ROS recorded when a message was published and each callback received the message and the publish time, but I guess that's a separate issue.

Fixed size custom message list - All entries have reference to same object

CustomMsgEntry.msg

float32 foo
float32 bar

CustomMsgEntryList.msg

CustomMsgEntry[50] list

When using this in python:

entry_list = CustomMsgEntryList()
entry_list[0].foo = 2
print entry_list

You will see that 2 is assigned to foo of all objects in the list. You could just instantiate a new CustomEntry object and assign this as a work around.

Error: the - operator, is deprecated (rostime)

Hi everyone,
Not sure if this is a bug or if I am doing something wrong.
I was using the rosbag api to save a bag (output_bag.write(topic, msg, t)) when I came across this error:
Traceback (most recent call last): (...) File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosbag/bag.py", line 382, in write if t > self._curr_chunk_info.end_time: File "/opt/ros/kinetic/lib/python2.7/dist-packages/genpy/rostime.py", line 270, in __cmp__ return (a > b) - (a < b) TypeError: numpy boolean subtract, the -operator, is deprecated, use the bitwise_xor, the^operator, or the logical_xor function instead.
I ended up editing this line in my rostime.py file. Just replaced the - by a ^ and it worked.
Thank you.

rospy absolute duration bug

Hello,
I use ROS hydro and I think I found an error in the code of the rospy.Duration class

When I do :

print abs(rospy.Duration.from_sec(0.5))

it prints -500000000 instead of 500000000. That's because the code only takes into account the number of seconds.
cf : https://github.com/ros/genpy/blob/indigo-devel/src/genpy/rostime.py

    def __abs__(self):
        """
        Absolute value of this duration
        :returns: positive :class:`Duration`
        """
        if self.secs > 0:
            return self
        return self.__neg__()

It should be :

    def __abs__(self):
        """
        Absolute value of this duration
        :returns: positive :class:`Duration`
        """
        if self.secs < 0 or (self.secs == 0 and self.nsecs<0):
            return self.__neg__()
        return self

Message hashes don't change with contents

Eg:

$ python
Python 2.7.11 (default, Jan 22 2016, 08:29:18) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from std_msgs.msg import Int8
>>> foo = Int8()
>>> foo.data = 1
>>> hash(foo)
275653405
>>> foo.data = 2
>>> hash(foo)
275653405  # Should be different!
>>> bar = Int8()
>>> bar.data = 1
>>> hash(bar)
275876165  # Should be the same as the first one!

The default hash method supplied with object is based on identity rather than fields, hence the above behaviour.

Instead, generated messages should supply a __hash__ function which hashes the data, possibly as simple as just passing the message's string representation to hash.

regression on kinetic due to #127

The changes in #127 (https://github.com/ros/genpy/blob/kinetic-devel/src/genpy/generator.py#L761) adds the line:

codecs.lookup_error("rosmsg").msg_type = self._type

to deserialization. On kinetic this line throws LookupException with the message: unknown error handler name 'rosmsg'

The issue can be reproduced by using any package that has regenerated its messages since #127. The package where I discovered it was with rosbridge_suite.

This is how to reproduce it:

# Boot up the minimal ros image in docker
docker run -it -p 11311:11311 ros:kinetic-ros-core

# Update apt and install ros-kinetic-rosbridge-suite
apt update
apt install ros-kinetic-rosbridge-suite

# Start roscore
roscore

On the host computer run docker ps to get the container id.

Log into the container and start rosapi (from rosbridge_suite):

docker exec -it <CONTAINER ID> bash

# Inside the container
source /opt/ros/kinetic/setup.bash
rosrun rosapi rosapi_node

Log into the container and call service:

docker exec -it <CONTAINER ID> bash

# Inside the container
source /opt/ros/kinetic/setup.bash
rosservice call /rosapi/get_param what ever

The service call will fail when trying to deserialize the message.

This can probably also be done on your local machine, but I choose docker to make sure we have the same environment.

UnicodeDecodeError with non-ASCII characters in message definition

I have custom message definitions that contain UTF-8 encoded characters in the comments. When I try to filter topics from bagfiles that contain these messages, it fails with UnicodeDecodeError.

I already described it here:

ros/ros_comm#143 (comment)

And linked to a related entry in the ROS Q&A:

https://answers.ros.org/question/240612/rosbag-unicodedecodeerror/

On my machine I could work around this by removing the call to .encode() in the line highlighted in the Traceback:

https://github.com/ros/genpy/blob/kinetic-devel/src/genpy/dynamic.py#L168

I don't know what other side effects this may produce (maybe with other file encodings of the message definition?), but so far it seems to work...

Python 3: _check_types() fails on binary uint8[]

import rospy
from std_msgs.msg import UInt8MultiArray
m = UInt8MultiArray(data=bytes([0, 1, 2]))
m._check_types()

The check passes on melodic, but fails on noetic:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/ros/noetic/lib/python3/dist-packages/genpy/message.py", line 362, in _check_types
    check_type(n, t, getattr(self, n))
  File "/opt/ros/noetic/lib/python3/dist-packages/genpy/message.py", line 281, in check_type
    raise SerializationError('field %s must be a list or tuple type' % field_name)
genpy.message.SerializationError: field data must be a list or tuple type

Is assigning bytes no longer accepted?

struct.pack / struct.unpack / struct.calcsize on fields with variable size leaks memory in Python 2

Originally caused by https://bugs.python.org/issue14596

TL;DR: The python struct module caches struct instances.

All calls of the module functions struct.pack, struct.unpack and struct.calcsize make use of this cache.
https://github.com/ros/genpy/blob/kinetic-devel/src/genpy/generator.py

We debugged this issue with a simple script like this and a huge map published on /map:

import rospy
import nav_msgs.msg
rospy.init_node("test")
def noop(msg):
    print "msg arrived"
s = rospy.Subscriber("/map", nav_msgs.msg.OccupancyGrid, noop)
rospy.sleep(5)
s.unregister()
rospy.spin()

The result is that the node still consumes 370 MB after the print of "msg arrived" and even after unregistering the subscriber (to rule out latched topic msg caching).

Calling struct._clearcache() frees the RAM and we get down to 52 MB.

Solution

Calls of struct.pack, struct.unpack and struct.callback need to be substituted by the respective calls of the Struct class.

Negative range limit is wrong for signed ints

Using ROS Kinetic, in rqt topic message publisher I get an error when trying to set an int32 to -2147483648...
-2147483648 # error: field data exceeds specified width [int32]
That's wrong, the range should be 2147483647 to -2147483648.

I think the issue is here...

genpy/src/genpy/message.py

Line 215 in 0ab1f40

if field_val >= maxval or field_val <= -maxval:

It should be...
if field_val >= maxval or field_val < -maxval:
...note, change from <= to <.

genpy should generate serialize/deserialize as private names

They pollute the actual field namespace and aren't technically reserved.

Since it would affect public API it is not likely to be addressed before a new major version.

(Copied from https://code.ros.org/trac/ros/ticket/3091)

Undefined names: 'package' and 'name' in generator.py

flake8 testing of https://github.com/ros/genpy on Python 3.6.3

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./src/genpy/generator.py:743:76: F821 undefined name 'package'
        raise MsgGenerationException("Cannot generate .msg for %s/%s: %s"%(package, name, str(e)))
                                                                           ^
./src/genpy/generator.py:743:85: F821 undefined name 'name'
        raise MsgGenerationException("Cannot generate .msg for %s/%s: %s"%(package, name, str(e)))
                                                                                    ^
./src/genpy/message.py:231:35: F821 undefined name 'unicode'
            if type(field_val) == unicode:
                                  ^
./src/genpy/message.py:604:17: F821 undefined name 'reload'
                reload(pypkg)
                ^
4     F821 undefined name 'package'
4

release 0.6.10 into melodic

Please release 0.6.10 into melodic as well.

This is needed in the official melodic distro, because all message packages need to regenerate the serialization code with this genpy version (to get fixes for reproducibility and correct exception messages).

reserved Python keywords in an array of custom message type

Usually message fields with a name that equals a reserved Python keyword like "from" are renamed to e.g. "from_". Unfortunately this does not seem to work when the keyword name is used in a sub field of an array of custom messages.
This results in a SyntaxError exception when importing the generated Python message wrapper.

The easiest way to reproduce this is to change "genpy/test/msg/TestString.msg" from:
string data
to:
string from

Regenerate the test messages:

./scripts/genmsg_py.py -p genpy -Igenpy:`pwd`/test/msg -o src/genpy/msg `pwd`/test/msg/TestFillEmbedTime.msg
./scripts/genmsg_py.py -p genpy -Igenpy:`pwd`/test/msg -o src/genpy/msg `pwd`/test/msg/TestFillSimple.msg
./scripts/genmsg_py.py -p genpy -Igenpy:`pwd`/test/msg -o src/genpy/msg `pwd`/test/msg/TestManyFields.msg
./scripts/genmsg_py.py -p genpy -Igenpy:`pwd`/test/msg -o src/genpy/msg `pwd`/test/msg/TestMsgArray.msg
./scripts/genmsg_py.py -p genpy -Igenpy:`pwd`/test/msg -o src/genpy/msg `pwd`/test/msg/TestPrimitiveArray.msg
./scripts/genmsg_py.py -p genpy -Igenpy:`pwd`/test/msg -o src/genpy/msg `pwd`/test/msg/TestString.msg

And then run the genpy test suite:

catkin_make -DCATKIN_ENABLE_TESTING=1
catkin_make run_tests

This produces the following error (3 times):

ERROR: test_fill_message_args_embed_time (test.test_genpy_message.MessageTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dorian/src/ros-stacks/catkin/src/genpy/test/test_genpy_message.py", line 185, in test_fill_message_args_embed_time
    from genpy.msg import TestFillEmbedTime
  File "/home/dorian/src/ros-stacks/catkin/src/genpy/src/genpy/msg/__init__.py", line 1, in <module>
    from ._TestFillEmbedTime import *
  File "/home/dorian/src/ros-stacks/catkin/src/genpy/src/genpy/msg/_TestFillEmbedTime.py", line 87
    _x = val1.from
                 ^
SyntaxError: invalid syntax

Duration.floordiv(Duration, int) is ill-defined

Floor division makes no sense, because floor makes no sense for duration, as it requires some sense of a base unit.

Two possible sets of semantics could be:

Duration // int = Duration.from_nsecs(d.to_nsecs() // int)
Duration // int = Duration.from_secs(d.to_secs() // int)

Unfortunately, the current semantics are neither of these!

genpy.message.DeserializationError: unpack requires a string argument of length 76

I'm using ROS Kinetic， and I try to read a rosbag using python package rosbag . But I just got this error when started to read it.

File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosbag/bag.py", line 2340, in read_messages
    yield self.seek_and_read_message_data_record((entry.chunk_pos, entry.offset), raw)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rosbag/bag.py", line 2490, in seek_and_read_message_data_record
    msg.deserialize(data)
  File "/tmp/genpy_1LKntB/tmphfDta4.py", line 916, in deserialize
    raise genpy.DeserializationError(e)  # most likely buffer underfill
genpy.message.DeserializationError: unpack requires a string argument of length 76

My code is :

for topic, msg, t in self.bag.read_messages():
            if topic == '/mapserver/refined_pose':
                poses.append(msg)
                t1_vec.append(t)
                cnt1 += 1
            if topic == '/pita/info/obs_map':
                map_area = self.compute_mapping_area(msg)
                self.maps_area_vec.append(map_area)
                t2_vec.append(t)
                cnt2 += 1
            print('cnt1 = ', cnt1, 'cnt2 = ', cnt2)
        self.bag.close()

DeserializationError cannot deserialize: unknown error handler name 'rosmsg'

The latest ROS Melodic update upgraded the genpy package from 0.6.14 to 0.6.15. It causes an error (DeserializationError cannot deserialize: unknown error handler name 'rosmsg') to occur in a system that worked well before the upgrade. I was not able to detect the exact cause of the error, but it seems to be very similar to the previously known bug discussed in:
PR #127
https://answers.ros.org/question/360537/unknown-error-handler-name-rosmsg/?answer=360643#post-id-360643

Downloading and building the genpy package (version tag 0.6.14) from source fixes the problem (0.6.15 built from source causes the problem, too).

	yield '# This Python file uses the following encoding: utf-8'
	yield '"""autogenerated by genpy from %s.msg. Do not edit."""' % spec.full_name
	yield 'import sys'
	yield 'python3 = True if sys.hexversion > 0x03000000 else False'
	yield 'import genpy\nimport struct\n'