Build failures on O(S)R(F) buildfarm: "no attribute '__legacy__'" about catkin_virtualenv

[gavanderhoorn]

Hm, not sure what happened, but now they are green again.

I'll keep an eye on the builds.

[gavanderhoorn]

Just did a source /path/to/catkin_ws/build/haros_catkin/venv/bin/activate and then pip list | grep setuptools. This returns setuptools 20.7.0 for me (on a local system, I don't have access to the workspace the buildfarm uses).

The buildfarm seems to run /tmp/binarydeb/ros-kinetic-haros-catkin-0.1.1/obj-x86_64-linux-gnu/venv/bin/python /tmp/binarydeb/ros-kinetic-haros-catkin-0.1.1/obj-x86_64-linux-gnu/venv/bin/pip install -qq -U pip==19.3.1. "The internet" seems to suggest that pip >= 19 does not play nice with such old versions of setuptools.

The next command the buildfarm runs is /tmp/binarydeb/ros-kinetic-haros-catkin-0.1.1/obj-x86_64-linux-gnu/venv/bin/python /tmp/binarydeb/ros-kinetic-haros-catkin-0.1.1/obj-x86_64-linux-gnu/venv/bin/pip install -qq setuptools<45.

Could this be reversed and should setuptools be updated/installed first and then the new pip version? virtualenv --python /usr/bin/python2 --system-site-packages --no-setuptools ./venv suggests there should be no setuptools yet in the venv when it's created.

(if it wasn't clear: I'm not into Python too much)

[paulbovbel]

I'm not into Python too much

This is voodoo to me as well. I've never used this package on ARM, and I'm never surprised to see issues, especially related to various pip and setuptools versions.

Could this be reversed and should setuptools be updated/installed first and then the new pip version?

That certainly sounds reasonable, if you find it fixes the issue. I also don't have direct access to those platforms for testing - QEMU might help there. I'm happy with whatever change helps you move forward as long as the tests pass :)

Please keep in mind that 0.6 works a bit differently (https://github.com/locusrobotics/catkin_virtualenv/blob/master/catkin_virtualenv/src/catkin_virtualenv/venv.py#L79)

[gavanderhoorn]

Seems I can reproduce this with a prerelease test for Kinetic+Xenial on i386.

Unfortunately 0.6 fails with a similar error message :(

I'll try to figure out whether the order of things matters.

---

Edit: setting USE_SYSTEM_PACKAGES FALSE makes things work again. But has the side-effect of increasing package size by 30MB+. From what I understand from the issues I found it could be that either setuptools is not installed at all when venv/pip tries to use it, or because the only requirement passed to pip is <45, any version of setuptools will satisfy that requirement, leading to very old versions getting used.

---

Edit 2: changing setuptools<45 to setuptools==44.0.0 as a test (with USE_SYSTEM_PACKAGES back to its default (ie: FALSE)) and no other changes also seems to work.

Output from pip when run here:

Collecting pip==20.1
  Downloading pip-20.1-py2.py3-none-any.whl (1.5 MB)
Collecting pip-tools==5.1.2
  Downloading pip_tools-5.1.2-py2.py3-none-any.whl (43 kB)
Collecting setuptools==44.0.0
  Downloading setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
Collecting click>=7
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Requirement already satisfied: six in /usr/lib/python2.7/dist-packages (from pip-tools==5.1.2) (1.10.0)
Installing collected packages: pip, click, pip-tools, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.2.2
    Uninstalling pip-20.2.2:
      Successfully uninstalled pip-20.2.2
  Attempting uninstall: setuptools
    Found existing installation: setuptools 20.7.0
    Not uninstalling setuptools at /usr/lib/python2.7/dist-packages, outside environment /tmp/ws/build_isolated/haros_catkin/venv
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed click-7.1.2 pip-20.1 pip-tools-5.1.2 setuptools-44.0.0

So it seems indeed the case that the 20.7.0 version of setuptools on i386 Xenial is just too old for pip==20.1 which then causes the failure.

Now to see which minimum version of setuptools will work in this case, to get a setuptools>=X,<45 requirement.

[gavanderhoorn]

Now to see which minimum version of setuptools will work in this case, to get a setuptools>=X,<45 requirement.

What about settling on a 44.x version of setuptools? That would still mean <45, but would automatically install updates to the 44.x series of setuptools (not sure those are still released).

[gavanderhoorn]

Ok. Changing:

catkin_virtualenv/catkin_virtualenv/src/catkin_virtualenv/venv.py

Line 79 in 8cda360

preinstall += ['setuptools<45']

To:

preinstall += ['setuptools>=44,<45']

works for me in a prerelease run with i386 Kinetic on Xenial.

This installs:

Collecting pip==20.1
  Downloading pip-20.1-py2.py3-none-any.whl (1.5 MB)
Collecting pip-tools==5.1.2
  Downloading pip_tools-5.1.2-py2.py3-none-any.whl (43 kB)
Collecting setuptools<45,>=44
  Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB)
Collecting click>=7
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
Requirement already satisfied: six in /usr/lib/python2.7/dist-packages (from pip-tools==5.1.2) (1.10.0)
Installing collected packages: pip, click, pip-tools, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.2.2
    Uninstalling pip-20.2.2:
      Successfully uninstalled pip-20.2.2
  Attempting uninstall: setuptools
    Found existing installation: setuptools 20.7.0
    Not uninstalling setuptools at /usr/lib/python2.7/dist-packages, outside environment /tmp/ws/build_isolated/haros_catkin/venv
    Can't uninstall 'setuptools'. No files were found to uninstall.
Successfully installed click-7.1.2 pip-20.1 pip-tools-5.1.2 setuptools-44.1.1

I'll submit a PR.

[gavanderhoorn]

See #71.

Afaict tests also still pass like this.

[gavanderhoorn]

Re-opening as this is still an issue unfortunately.

Problem is I can't reproduce it locally. I've tried using the aarch64 Docker images to run a Debian Stretch + Melodic prerelease, but those succeed ..

---

Edit: oh, seems I can't re-open my own issue?

[gavanderhoorn]

It fails on arm64, armhf and i386, but not on amd64. This makes it difficult to diagnose, as I don't have access to any of those platforms (or at least: not easily).

Final comment (for future readers perhaps also): multi-arch support via QEMU and binfmt in the end allowed me to run prerelease tests for arm64 and armhf architectures.

They took "forever" (due to emulation), but it was better than nothing.

On some versions of Ubuntu you need to fiddle a little bit with the binfmt configuration. I took inspiration from docker/for-linux#56 and the linked computermouth/qemu-static-conf (copying only the files I needed though, and to /etc/binfmt.d instead of creating a new directory under /lib).