cedarai / rules_pyvenv

Bazel rules for creating Python virtual envs.

Bazel rules for creating Python virtual environments.

See example/ for an example.

Installation

Add the following to your WORKSPACE.

(Note: see the releases page for release-specific WORKSPACE config)

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_pyvenv",
    strip_prefix = "rules_pyvenv-main",
    url = "https://github.com/cedarai/rules_pyvenv/archive/main.tar.gz",
)

These rules require a recent version of Python 3.6+ and rules_python. The environment is built using the venv library that ships with the Python standard library. If using the system-provided Python on Debian/Ubuntu, you may need to run

apt install python3.8-venv

On Windows, you need to enable symlink support.

Example

$ cd example
$ bazel run //:venv env
$ . env/bin/activate

rules_pyvenv's Issues

Missing data dependencies

Issue

The resulting virtual environment are missing data dependencies of the package.

When running the command pyre, a script within the pyre_check package invokes a binary that sits inside the data folder. However, this data folder is not included by the virtual environment created with this Bazel rules.

For example, @pip_pyre_check//:data/bin/pyre.bin is needed in order for pyre command to successfully run.

I think the fix here would be to include the data dependencies into the generated virtual environment.

➜ pyre
ƛ Invalid configuration: Cannot locate a Pyre binary to run.

Release 1.1 seems to have the incorrect SHA assigned

Steps to Reproduce:

$ cat WORKSPACE 
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_pyvenv",
    sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    strip_prefix = "rules_pyvenv-1.1",
    url = "https://github.com/cedarai/rules_pyvenv/archive/refs/tags/1.1.tar.gz",
)

$ bazel sync
INFO: Repository rules_pyvenv instantiated at:
  /tmp/s/WORKSPACE:3:13: in <toplevel>
Repository rule http_archive defined at:
  /home/trironkk/.cache/bazel/_bazel_trironkk/3c4ec0d0df958c43c4a5751556a64a69/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
WARNING: Download from https://github.com/cedarai/rules_pyvenv/archive/refs/tags/1.1.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Checksum was 74aa1cda
e0ba34daa00791d4a73ad80d5117854c057bdb535e3e84f36b6b91aa but wanted e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
ERROR: An error occurred during the fetch of repository 'rules_pyvenv':
   Traceback (most recent call last):
        File "/home/trironkk/.cache/bazel/_bazel_trironkk/3c4ec0d0df958c43c4a5751556a64a69/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/cedarai/rules_pyvenv/archive/refs/tags/1.1.tar.gz] to /home/trironkk/.cache/bazel/_bazel_trironkk/3c4ec0d0df958c43c4a57515
56a64a69/external/rules_pyvenv/temp3900851868545547333/1.1.tar.gz: Checksum was 74aa1cdae0ba34daa00791d4a73ad80d5117854c057bdb535e3e84f36b6b91aa but wanted e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b
7852b855
ERROR: /tmp/s/WORKSPACE:3:13: fetching http_archive rule //external:rules_pyvenv: Traceback (most recent call last):
        File "/home/trironkk/.cache/bazel/_bazel_trironkk/3c4ec0d0df958c43c4a5751556a64a69/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/cedarai/rules_pyvenv/archive/refs/tags/1.1.tar.gz] to /home/trironkk/.cache/bazel/_bazel_trironkk/3c4ec0d0df958c43c4a57515
56a64a69/external/rules_pyvenv/temp3900851868545547333/1.1.tar.gz: Checksum was 74aa1cdae0ba34daa00791d4a73ad80d5117854c057bdb535e3e84f36b6b91aa but wanted e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b
7852b855
ERROR: java.io.IOException: Error downloading [https://github.com/cedarai/rules_pyvenv/archive/refs/tags/1.1.tar.gz] to /home/trironkk/.cache/bazel/_bazel_trironkk/3c4ec0d0df958c43c4a5751556a64a69/external/rules_
pyvenv/temp3900851868545547333/1.1.tar.gz: Checksum was 74aa1cdae0ba34daa00791d4a73ad80d5117854c057bdb535e3e84f36b6b91aa but wanted e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
 checking cached actions

References:

• https://github.com/cedarai/rules_pyvenv/releases/tag/1.1

setuptools requirement broken in venv

When py_venv attempts to install a setuptools dependency into the virtual environment, that setuptools package is broken. I believe this is due to the fact that the venv stdlib module will automatically install a version of setuptools into the fresh virtual environment. Then when the py_venv rule begins creating symlinks to the setuptools package files, it only symlinks to files that do not already exist, resulting in a mangled combination of the pre-installed setuptools package and the desired setuptools package.

I am currently able to work around this using the following extra_pip_commands:

py_venv(
    ...
    extra_pip_commands = [
        "install --ignore-installed setuptools"
    ]
)

which fixes the corrupted setuptools package but is not ideal since I do not believe it's using the "locked" version of setuptools which I am expecting.

rules_python runfiles.py is not added to the correct location in the venv

When creating a venv with the following config:

py_venv(
    name = "venv",
    always_link = True,
    deps = all_requirements + [
        "@rules_python//python/runfiles",
    }
)

I get the following warning: Warning: [../rules_python/python/runfiles/runfiles.py] didn't match any imports. Including as [python/runfiles/runfiles.py]

However, python/runfiles/runfiles.py isn't actually the correct location for that file. It should be rules_python/python/runfiles/runfiles.py.

I was able to get around this issue locally by swapping the return of get_site_packages_path from include_path to wspath, but I don't know if that's a correct universal fix. I did notice that rules_python is missing from imports in the generated _deps.json, so maybe adding rules_python/python/runfiles there would also fix things.

Allow installing python deps using pth files

Currently, rules_pyvenv installs all the python deps by symlinking them in site-packages. However, in bazel, the python deps are actually added to the PYTHONPATH, which means that for some deps, just symlinking into site-packages doesn't work.

However, venv supports this via pth files. Would it be feasible to add a flag to py_venv to also create a .pth file to append these deps to the PYTHONPATH as well?

Support for handling py_library dependencies

This is great, thanks for sharing. One thing that appears to be missing is hooking up any py_library dependencies as part of the generated venv

Executables in venv/bin are only generated for console scripts

It seems like rules_pyvenv will only create executables in the venv bin for console scripts.

For example: tox works as intended, generating venv/bin/tox. However nothing is created in the bin for py-spy or pyhcl.

py-spy is actually a rust binary that gets built by maturin, but pyhcl is just regular python.

As far as I can tell, the commonality between them is that when you unzip a wheel of those packages, you get a <package name>.data/scripts folder that contains files that should be copied into the venv bin. There's also no entry_points.txt in their .dist-info folders.

I'm not 100% if this is a bug exclusively with rules_pyvenv. It seems like repositories generated by pip_parse in rules_python all have a bin folder with the mostly correct files in them (the python files have a #!/dev/null shebang), but only the tox repository has an additional py_binary definition in its BUILD.bazel. Edit: It looks like there's a related issue already on rules_python for this: bazelbuild/rules_python#801

This was how I got to the pip_parse generated directories, assuming pip_parse is named pip:
$(bazel info output_base)/external/pip_tox
$(bazel info output_base)/external/pip_pyhcl
$(bazel info output_base)/external/pip_py_spy

I'd be happy to help work on a fix for this after discussing the best approach.