Installing Python bindings when `pip` in `PYTHONPATH` but not in Python prefix about mlpack

[wdconinc]

huge amount of complexity in Python environments

Yeah... I am hardly an expert on Python packaging, let alone Python bindings packaging in CMake projects that is compliant with PEP517. A modernization of a similar Python bindings installation happened recently in another project I follow (but am not involved in), xrootd/xrootd#2025. Maybe there is a use in looking at their approach. In particular they allow for a set(PIP_OPTIONS "" CACHE STRING "pip options used during the Python bindings install.") cache variable to pass --no-deps.

Do you want to open a PR with that patch?

I will do so today.

[wdconinc]

@rcurtin See #3502.

[barak]

I am having similar issues packaging 4.2.0 for Debian. With the added caveat that network access during the build is strictly forbidden, so pip should not be invoked. (Well, maybe pip show, although that wouldn't really be definitive since python packages might be installed in other ways.) Perhaps a build option NET_ACCESS=OFF could be added to turn off anything that might access the internet, and just check for whether the thing it wants to download is already available on the system and if not give a cogent error message.

(This NET_ACCESS=OFF option might also be useful on my kids, but I digress.)

[rcurtin]

Thank you for opening this issue @wdconinc! The Python installation step has always been something that I've been dubious about due to the huge amount of complexity in Python environments, so I'm hopeful that we can figure out a robust way forward here (and handle @barak's issue too).

First things first: @barak, what does the Debian project think about the deprecation of setup.py install? That caused me to make #3313, which uses pip to install instead of the typical but apparently now deprecated python setup.py install. So, I could make some CMake option that uses the old style of installation, or, I could perhaps add some pip options that avoid the network entirely: --no-index --no-deps -f ./. What do you think on this one?

Next, on to the PYTHONPATH issue. A bit of history: @wdconinc the code you found and patched (which nukes PYTHONPATH and replaces it with CMAKE_PYTHON_PATH) is necessary because setuptools and pip will refuse to install a package if the destination location is not in PYTHONPATH. So, I think you are right that a good solution is appending the destination location to PYTHONPATH instead of totally overwriting it. Do you want to open a PR with that patch? If not I can do it. It looks fine to me as-is.

Another note: we once had a situation where installing the Python bindings to CMAKE_INSTALL_PREFIX was incorrect (#2797), which is similar to your situation. So, the PYTHON_INSTALL_PREFIX CMake option was added, so that you can directly install to that location. I suspect that you could also solve your issue by simply specifying -DPYTHON_INSTALL_PREFIX="$PYTHONPATH".

[rcurtin]

Thanks! I also opened #3505 to handle the network issue. Would you like to try that patch in your environment to make sure that it doesn't break anything? I know that it works on Fedora, but not sure on Debian or spack---would be great to know first before merging it. :)

[barak]

First things first: @barak, what does the Debian project think about the deprecation of setup.py install? That caused me to make https://github.com/mlpack/mlpack/pull/3313, which uses pip to install instead of the typical but apparently now deprecated python setup.py install. So, I could make some CMake option that uses the old style of installation, or, I could perhaps add some pip options that avoid the network entirely: --no-index --no-deps -f ./. What do you think on this one?

Unfortunately, I treasure my ignorance of this Python packaging stuff.