Workaround native library (un)availability for Python 3 about catkin_virtualenv

[paulbovbel]

Not on my part. Since ROS will be making a hard cutover to python 3, I'm going to just let this sort itself out.

[mikepurvis]

I think you could get most of what you want by teaching catkin_python_setup to optionally build libraries for a specified Python2 as well as Python3 version:

https://github.com/ros/catkin/blob/kinetic-devel/cmake/catkin_python_setup.cmake
https://github.com/ros/catkin/blob/kinetic-devel/cmake/templates/python_distutils_install.sh.in

You're still only going to have one value for PYTHON_EXECUTABLE, and that'll be what ends up in shebangs and so-on, but building a second of bindings libs or whatever, to permit the same modules to be used in either version should be possible.

The main question then is, is there a sane way to integrate that change into ros/catkin (and an API to expose the "build both" intention), and is it something that @dirk-thomas would want to merge into mainline? If not, an alternative approach would be to double-build as part of an external wrapper, for example in catkin_tools or something else. Doing that would probably look like:

• Detecting that a package has installed a file which matches lib/python2.7/(.*)\.so
• Re-building that package for python3, pointed at a different installspace.
• Copying lib/python3/* into the the original installspace.

Potentially there'd be a way for specific packages to opt-into this behaviour by calling another CMake function, but I feel like part of the point is to get all your dependencies in line, and that doesn't really work if you need to fork them all to shove this in.

[dirk-thomas]

Coming up with a clean way to support building with Python 2 or 3 is only the list for ROS Melodic. I can't tell yet how it will look like but we hope to support that as well as provide a way to test ROS packages with Python 3. Currently you can already override the Python interpreter but this needs to get further in term of how to map the rosdep keys, etc.

[paulbovbel]

Thanks for the comments!

@mikepurvis: This seems like quite an extension, since AFAIK catkin_python_setup doesn't have anything to do with creating library targets. At least looking at tf2_py as an example, I see that happening here: https://github.com/ros/geometry2/blob/indigo-devel/tf2_py/CMakeLists.txt#L92

Are you proposing additional catkin commands to define the libraries, which would then be adapted into py2/py3 library targets via catkin_python_setup?

Wrt to catkin_tools, we are using ros_buildfarm internally, and I'm hoping this module will be useful to the community at large for releasing on public buildfarm. If Locus migrates to bespoke build infrastructure, I imagine I'll have much better options in terms of bundling virtualenvs with application deployments.

Another issue is that catkin_virtualenv currently keeps the underlying /opt/ros/kinetic/lib/python2.7/dist-packages PYTHONPATH, regardless of which PYTHON_EXECUTABLE is being used. So even if I got tf2_py to build into python2.7 and python3.x with the appropriate underlying PythonLibs, there is still the matter of getting the virtualenv to pick it up. I suppose I'm already doing enough regex hacks with dh_virtualenv that one more won't hurt...

---

@dirk-thomas I am very much looking forward to that. Do you think that would address the issue I'm having here, which is that a package like tf2_py will build native libraries targeting py2 OR py3, but not both?

[mikepurvis]

You dug deeper than I did. I definitely thought that native/compiled extensions were being handled via the Python build infrastructure, as it is in more conventional PyPI/distutils packages, eg: https://docs.python.org/3/extending/building.html

However, given the current realities, it seems like it's something which will indeed require changes in both the build infrastructure and also the individual packages which are supplying binary extensions. Here's our exposure on this front (some of these are indeed built by non-catkin python packages, so that would be addressed via changes on our side to catkin_tools_python):

$ find lib/python2.7 | grep \.so$
lib/python2.7/dist-packages/roslz4/_roslz4.so
lib/python2.7/dist-packages/PyKDL.so
lib/python2.7/dist-packages/wrapt/_wrappers.so
lib/python2.7/dist-packages/tornado/speedups.so
lib/python2.7/dist-packages/cv_bridge/boost/cv_bridge_boost.so
lib/python2.7/dist-packages/tf2_py/_tf2.so
lib/python2.7/dist-packages/cv2.so
lib/python2.7/dist-packages/matplotlib/_contour.so
lib/python2.7/dist-packages/matplotlib/_path.so
lib/python2.7/dist-packages/matplotlib/ft2font.so
lib/python2.7/dist-packages/matplotlib/_png.so
lib/python2.7/dist-packages/matplotlib/_tri.so
lib/python2.7/dist-packages/matplotlib/_delaunay.so
lib/python2.7/dist-packages/matplotlib/ttconv.so
lib/python2.7/dist-packages/matplotlib/backends/_tkagg.so
lib/python2.7/dist-packages/matplotlib/backends/_backend_agg.so
lib/python2.7/dist-packages/matplotlib/_image.so
lib/python2.7/dist-packages/matplotlib/_cntr.so
lib/python2.7/dist-packages/matplotlib/_qhull.so
lib/python2.7/dist-packages/twisted/test/raiser.so
lib/python2.7/dist-packages/twisted/python/_sendmsg.so
lib/python2.7/dist-packages/camera_calibration_parsers/camera_calibration_parsers_wrapper.so
lib/python2.7/dist-packages/qt_gui_cpp/libqt_gui_cpp_shiboken.so
lib/python2.7/dist-packages/qt_gui_cpp/libqt_gui_cpp_sip.so
lib/python2.7/dist-packages/rviz/librviz_sip.so

Either way, the good news is that Melodic is only a few months away, so it's likely that OSRF will be planning for and beginning to implement their vision of the support for the catkin+python side sooner rather than later, and I'm sure would be delighted to collaborate on it.

[dirk-thomas]

Do you think that would address the issue I'm having here, which is that a package like tf2_py will build native libraries targeting py2 OR py3, but not both?

I don't know enough about this package to answer your question.

REP 151 will only aim to support one Python version at a time (which the user can choose when building their ROS distro from source).

[gavanderhoorn]

Trying to package something that tries to build Cython extensions: has there been any progress on this front? It's OK if there isn't, just wanted to make sure I'm not going down the wrong path (ie: something that doesn't and cannot ever work).

[paulbovbel]

Well, consider this one solved!