aspect-build / rules_py Goto Github PK

What happened?

When creating a venv with the py_venv rule, none of the libraries installed with pip are exported.

Version

Development (host) and target OS/architectures: Ubuntu x86_64

Output of bazel --version: bazel 7.0.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: aspect_rules_py = 0.5.0 or HEAD, rules_python = 0.29.0

Language(s) and/or frameworks involved:
Python

How to reproduce

Minimal example:

MODULE.bazel:

bazel_dep(name = "aspect_rules_py", version = "0.5.0")

bazel_dep(name = "rules_python", version = "0.29.0")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")

python.toolchain(
    python_version = "3.10",
    is_default = True,
)

use_repo(python, "python_versions")

register_toolchains("@python_versions//:all")

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")

pip.parse(
    hub_name = "pip",
    python_version = "3.10",
    requirements_lock = "//:requirements_lock.txt",
)

use_repo(pip, "pip")

requirements_lock.txt:

pyserial==3.5 \
    --hash=sha256:3c77e014170dfffbd816e6ffc205e9842efb10be9f58ec16d3e8675b4925cddb \
    --hash=sha256:c4451db6ba391ca6ca299fb3ec7bae67a5c55dde170964c7a14ceefec02f2cf0
    # via -r requirements.txt

BUILD:

load("@aspect_rules_py//py:defs.bzl", "py_venv")
load("@pip//:requirements.bzl", "all_requirements")

py_venv(
    name = "venv",
    deps = all_requirements,
)

Commands:

• bazel run //:venv
• .venv/bin/python
  • import serial (or inspect .venv and see that pyserial is missing)

Any other information?

No response

Although these rules behave differently, we need to be able to plug in the native / rules_python rules when interacting with external dependencies.

Currently the resolve_toolchain method assumes that the toolchain is setting interpreter, but it could set interpreter_path and therefore won't have a file associated. This method and downstream consumers should handle this.

Windows support in general will require funding the development.

If you or your company require Windows support, please consider donating to the project either in time or via Aspect's Open Collective https://opencollective.com/aspect-build

What happened?

When we use py_binary to build docker images with rules_oci, we couldn't build the tarball for ubuntu images. The error log:

ERROR: /Users/lyao/bazel/bazel-examples/oci_python_image/hello_world/BUILD.bazel:18:10: Creating virtual environment for //hello_world:hello_world failed: (Exit 126): sandbox-exec failed: error executing command
(cd /private/var/tmp/_bazel_lyao/1e813cdbb954646ce252ec3692bb82d4/sandbox/darwin-sandbox/10/execroot/_main &&
exec env -
TMPDIR=/var/folders/gm/r7n8598x7zq6rwkd1_r75_3h0000gp/T/
/usr/bin/sandbox-exec -f /private/var/tmp/_bazel_lyao/1e813cdbb954646ce252ec3692bb82d4/sandbox/darwin-sandbox/10/sandbox.sb /var/tmp/_bazel_lyao/install/7d4a2659eff664c24e2f7d717eef5139/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/private/var/tmp/_bazel_lyao/1e813cdbb954646ce252ec3692bb82d4/sandbox/darwin-sandbox/10/stats.out' /bin/bash -c bazel-out/darwin_arm64-fastbuild-ST-baeacbe55d95/bin/hello_world/hello_world.venv_venv.sh)
bazel-out/darwin_arm64-fastbuild-ST-baeacbe55d95/bin/hello_world/hello_world.venv_venv.sh: line 38: external/rules_python0.21.0python~python3_9_aarch64-unknown-linux-gnu/bin/python3: cannot execute binary file
INFO: Elapsed time: 0.763s, Critical Path: 0.37s
INFO: 3 processes: 3 internal.
FAILED: Build did NOT complete successfully

I'm using https://github.com/aspect-build/bazel-examples/tree/main/oci_python_image to reproduce the error.

The issue seems relate to the wrong python interpreter path and is not platform agnostic when calling venv.

I tried the following command with Mac as a workaround:
❯ bazel build //... --sandbox_debug --platform=//hello_world:x86_64_linux
it ran into the same issue (different error) and seems the current implementation of py_binary breaks the cross-platform build.

Version

Development (host) and target OS/architectures:
host: Mac with M1
target OS/arch: linux

Output of bazel --version:
bazel 6.2.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
the same as the example: https://github.com/aspect-build/bazel-examples

Language(s) and/or frameworks involved:

How to reproduce

build the example "oci_python_image" with Mac

Any other information?

No response

What is the current behavior?

rules_py doesn't seem to have an easy way to use multiple python toolchains in the same repository.

Describe the feature

When using python_register_multi_toolchains from bazelbuild/rules_python, it creates py_test and py_binary macros for each version of python that are loaded from @python//<version>:defs.bzl. It would be nice to have a rules_py analog to python_register_multi_toolchains, or some easy way to have rules_py's py_* rules use multiple python toolchains in a single workspace.

Currently we assume there's a pip installed in the same bin directory as python, and we assume the version is compatible with the flags passed. If using the hermetic toolchains from rules_python this is generally okay, however if using a different toolchain or that of the host, then we can not guarantee the presence of pip where we expect it.

An alternative would be to use https://github.com/pypa/installer directly and not have pip involved.

Another alternative would be to not to use either for actions, and have bazel layout the venv using the runfiles symlinks.

What happened?

If the .pth file points to something like a foo_pb2.py generated file, and that file changes (say a new proto field was added), what's required for PyCharm to pick up the new field? Does it notice the file change by setting a watch on it? Do we have to restart the editor, or make some no-op change to the .pth file to force it to get the change?

Requested by @jvolkman

Version

HEAD

How to reproduce

No response

Any other information?

No response

What happened?

First off, thank you for all the effort, this tree looks very promising.

After pulling together a few small tweaks to get the virtual environments set up, I ran into an issue that's likely to take longer than I have time for. What I'm seeing is that any package I try to install that requires setuptools (at least I assume) appears to break, not sure why. As an example here is me trying to install pyright into the venv so I can set up code completion in my editor (same issue for python-lsp-server). Perhaps this is related to the other wheel bugs reported against the venv rules?

Note that after adding --strip-extras to pip-compile, and --no-deps to the pip install command I was able to progress further into installing my repos requirements into the virtual environment, but am currently blocked by this.

% pip -v install pyright 
Using pip 22.3.1 from /home/USER/src/REDACTED/.server_bin.venv/lib/python3.9/site-packages/pip (python 3.9)
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/REDACTED/simple/
Collecting pyright
  Downloading https://us-python.pkg.dev/REDACTED/pyright/pyright-1.1.290-py3-none-any.whl (16 kB)
Collecting nodeenv>=1.6.0
  Downloading https://us-python.pkg.dev/REDACTED/nodeenv/nodeenv-1.7.0-py2.py3-none-any.whl (21 kB)
Collecting setuptools
  Downloading https://us-python.pkg.dev/REDACTED/setuptools/setuptools-66.1.1-py3-none-any.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 5.5 MB/s eta 0:00:00
Installing collected packages: setuptools, nodeenv, pyright
ERROR: Could not install packages due to an OSError.
Check the permissions.
Traceback (most recent call last):
  File "/home/USER/src/REDACTED/.server_bin.venv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 494, in run
    installed = install_given_reqs(
  File "/home/USER/src/REDACTED/.server_bin.venv/lib/python3.9/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "/home/USER/src/REDACTED/.server_bin.venv/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 792, in install
    install_wheel(
  File "/home/USER/src/REDACTED/.server_bin.venv/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 729, in install_wheel
    _install_wheel(
  File "/home/USER/src/REDACTED/.server_bin.venv/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 589, in _install_wheel
    file.save()
  File "/home/USER/src/REDACTED/.server_bin.venv/lib/python3.9/site-packages/pip/_internal/operations/install/wheel.py", line 382, in save
    os.unlink(self.dest_path)
PermissionError: [Errno 13] Permission denied: '/home/USER/src/REDACTED/.server_bin.venv/lib/python3.9/site-packages/_distutils_hack/__init__.py'

Version

Development (host) and target OS/architectures:

Output of bazel --version: bazel 5.4.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

Local copy of a personal fork, specifically this branch: https://github.com/tellett/rules_py/tree/fix_venv_python_path

Also a forked version of rules_python (up to date w/ the latest release) to install GCP artifact registry connectivity dependencies, specifically I installed or updated the following dependencies:

requirement("importlib_metadata"),
requirement("zipp"),
requirement("more_itertools"),
requirement("cachetools"),
requirement("certifi"),
requirement("cffi"),
requirement("charset_normalizer"),
requirement("cryptography"),
requirement("google_auth"),
requirement("idna"),
requirement("jaraco.classes"),
requirement("jeepney"),
requirement("keyring"),
requirement("keyrings.google_artifactregistry_auth"),
requirement("pluggy"),
requirement("pyasn1"),
requirement("pyasn1_modules"),
requirement("pycparser"),
requirement("requests"),
requirement("rsa"),
requirement("SecretStorage"),
requirement("six"),
requirement("urllib3"),

Language(s) and/or frameworks involved:

Python, FastAPI, PyTorch, GCP, etc.

How to reproduce

I believe this is above, if not please let me know and I'll provide more details or a sample repo.

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty

If a targets dependencies include the setuptools wheel, then this causes permission denied errors when attempting to install, as site-packages contains readonly symlinks to both setuptools and _distutils_hack.

We should be able to detect this state and not link in these.

 bazel-out/darwin_arm64-fastbuild/bin/ipython.venv.source/bin/python3 -B -s -I -m pip install --quiet --no-compile --require-virtualenv --no-input --no-cache-dir --disable-pip-version-check --no-python-version-warning --only-binary=:all: --no-dependencies --no-index -r bazel-out/darwin_arm64-fastbuild/bin/ipython.venv.requirements.txt
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/private/var/tmp/_bazel_matt/142bc527510d77fe74ad445c581c00a2/sandbox/darwin-sandbox/10/execroot/rules_py_example/bazel-out/darwin_arm64-fastbuild/bin/ipython.venv.source/lib/python3.8/site-packages/setuptools/__init__.py'
Check the permissions.

What happened?

I'm migrating a large codebase from rules_python to rules_py. One problem that I've had to address multiple times is that directory names and filenames are now shadowing third-party packages.

• I had to rename the langchain directory not to conflict with https://pypi.org/project/langchain/. import langchain.llms found our langchain directory first in sys.path.
• Similarly, for a pyspark directory.
• The built-in copy module ended up importing from our org.py:

When using rules_py, a number of tests fail with a callstack like this:

Traceback (most recent call last):
  File "/mnt/tmpfs/BAZEL_OUTPUT/sandbox/processwrapper-sandbox/33663/execroot/blah_blah/bazel-out/k8-fastbuild/bin/src/python/pytask/foo/bar/quux.runfiles/blah_blah/tools/build_rules/py/private/pytest_loader.py", line 6, in <module>
    import pytest
...
  File "/mnt/tmpfs/BAZEL_OUTPUT/sandbox/processwrapper-sandbox/33663/execroot/blah_blah/bazel-out/k8-fastbuild/bin/src/python/pytask/foo/bar/quux.runfiles/py_deps_main/attrs/attrs/attr/_make.py", line 5, in <module>
    import copy
  File "<frozen zipimport>", line 259, in load_module
  File "/mnt/tmpfs/BAZEL_OUTPUT/execroot/blah_blah/external/python_interpreter/lib/python38.zip/copy.py", line 60, in <module>
  File "//mnt/tmpfs/BAZEL_OUTPUT/sandbox/processwrapper-sandbox/33663/execroot/blah_blah/bazel-out/k8-fastbuild/bin/src/python/pytask/foo/bar/quux.runfiles/blah_blah/tasks/tasks/org.py", line 3, in <module>

If you look at the copy.py source, it's actually trying to do from org.python.core import PyStringMap. This is Jython support; a handled ImportError is raised in CPython. I worked around this by renaming org.py.

sys.path is substantially longer in rules_py. There are 1193 entries for the above test, versus 484 for rules_python. As far as I can tell, the primary difference is that every directory in our source tree that we import from seems to be in sys.path when using rules_py.

Version

Development (host) and target OS/architectures:
Linux x86_64, Ubuntu 20.02
Output of bazel --version:
bazel 6.2.0
Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
0.3.0
Language(s) and/or frameworks involved:
Python, Scala, Java, ...

How to reproduce

No response

Any other information?

No response

Wheels should go in deps, rather than wheels

What happened?

When using a standalone Python 3.11 toolchain, the environment setup in the entrypoint template is not right.

If we set up the Python interpreter to be version 3.11 as in the reproduction section below, we get an error from the Python interpreter at launch.

% bazel test //examples/pytest:pytest_test
DEBUG: Rule 'python_toolchain_aarch64-apple-darwin' indicated that a canonical reproducible form can be obtained by modifying arguments url = ["https://github.com/indygreg/python-build-standalone/releases/download/20230116/cpython-3.11.1+20230116-aarch64-apple-darwin-install_only.tar.gz"] and dropping ["urls"]
DEBUG: Repository python_toolchain_aarch64-apple-darwin instantiated at:
  /Users/sidb/Workspaces/rules_py/WORKSPACE:19:27: in <toplevel>
  /private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/external/rules_python/python/repositories.bzl:533:26: in python_register_toolchains
Repository rule python_repository defined at:
  /private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/external/rules_python/python/repositories.bzl:366:36: in <toplevel>
INFO: Analyzed target //examples/pytest:pytest_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //examples/pytest:pytest_test (see /private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/testlogs/examples/pytest/pytest_test/test.log)
INFO: From Testing //examples/pytest:pytest_test:
==================== Test output for //examples/pytest:pytest_test:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/sandbox/darwin-sandbox/28/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/bin/examples/pytest/pytest_test.runfiles/pytest_test.venv/bin/python'
  isolated = 1
  environment = 0
  user site = 0
  safe_path = 1
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.11'
  sys._base_executable = '/private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/execroot/aspect_rules_py/external/python_toolchain_aarch64-apple-darwin/bin/python3.11'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/private/var/tmp/_bazel_sidb/a21ec54da3d04a4699759ed6246d5320/sandbox/darwin-sandbox/28/execroot/aspect_rules_py/bazel-out/darwin_arm64-fastbuild/bin/examples/pytest/pytest_test.runfiles/pytest_test.venv/bin/python'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python311.zip',
    '/install/lib/python3.11',
    '/install/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001e27f5b40 (most recent call first):
  <no Python frame>
================================================================================

Version

Development (host) and target OS/architectures:

Output of bazel --version: bazel 5.4.0 (but same error with latest version of Bazel)

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: HEAD of this repo -- commit 913676de2f89b8272cdd14bc53f719534e58c31f

Language(s) and/or frameworks involved: Python

How to reproduce

To use a Python 3.11 toolchain with an example, we can modify this repo itself and test one of the examples here.

diff --git a/WORKSPACE b/WORKSPACE
index fc4261e..a5028a3 100644
--- a/WORKSPACE
+++ b/WORKSPACE
@@ -18,7 +18,7 @@ load("@rules_python//python:repositories.bzl", "python_register_toolchains")
 
 python_register_toolchains(
     name = "python_toolchain",
-    python_version = "3.9",
+    python_version = "3.11",
 )
 
 ############################################
diff --git a/examples/pytest/BUILD.bazel b/examples/pytest/BUILD.bazel
index 9398436..3decfc4 100644
--- a/examples/pytest/BUILD.bazel
+++ b/examples/pytest/BUILD.bazel
@@ -1,5 +1,4 @@
-load("@rules_python//python:defs.bzl", "py_test")
-load("@//py:defs.bzl", "py_pytest_main")
+load("@//py:defs.bzl", "py_pytest_main", "py_test")
 
 py_pytest_main(name = "__test__")
 
diff --git a/internal_deps.bzl b/internal_deps.bzl
index f0b12e8..ca2b071 100644
--- a/internal_deps.bzl
+++ b/internal_deps.bzl
@@ -12,6 +12,13 @@ def http_archive(name, **kwargs):
 
 def rules_py_internal_deps():
     "Fetch deps needed for local development"
+    http_archive(
+        name = "rules_python",
+        sha256 = "94750828b18044533e98a129003b6a68001204038dc4749f40b195b24c38f49f",
+        strip_prefix = "rules_python-0.21.0",
+        url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.21.0.tar.gz",
+    )
+
     http_archive(
         name = "build_bazel_integration_testing",
         urls = [

Any other information?

This issue may be relevant here -- https://discuss.python.org/t/python3-11-seems-to-break-venv-created-from-non-system-python-with-zipped-stdlib/20874.

What happened?

C:\dev\test>bazel run :script

ERROR: C:/dev/test/BUILD.bazel:5:10: Creating virtual environment for //:script failed: (Exit -1073741515): bash.exe failed: error executing CreateVenv command (from target //:script)

bazel-out/x64_windows-fastbuild/bin/script.venv_venv.sh: line 72: bazel-out/x64_windows-fastbuild/bin/script.venv.source/bin/activate: No such file or directory

Problem details:
python -m venv venv places the activation scripts in venv\Scripts\ on windows rather than venv\bin\ and so are not found by the .sh script.

Version

Development (host) and target OS/architectures:
win10/x64

Output of bazel --version:
bazel 7.0.0 with msys2

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
aspect_rules_py, version: 0.5.0
rules_python, version: 0.22.0

How to reproduce

# MODULE.bazel
module(
    name = "test",
    version = "0.0.0"
)

bazel_dep(name = "aspect_rules_py", version = "0.5.0")
bazel_dep(name = "rules_python", version = "0.22.0", dev_dependency = True)

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    name = "python3",
    configure_coverage_tool = True,
    python_version = "3.11",
)
use_repo(python, "python3_toolchains")

register_toolchains(
    "@python3_toolchains//:all",
)

# BUILD
load("@aspect_rules_py//py:defs.bzl", "py_binary")

py_binary(
    name = "script",
    srcs = ["script.py"],
    main = "script.py",
)

Run with: bazel build //:script

### Any other information?

_No response_

What happened?

Aspect suggests that flag for performance.

$ bazel run //engineering/projects/aws_lambda_hello_world:aws_lambda_hello_world_bin.venv
Starting local Bazel server and connecting to it...
INFO: Analyzed target //engineering/projects/aws_lambda_hello_world:aws_lambda_hello_world_bin.venv (48 packages loaded, 2615 targets configured).
INFO: Found 1 target...
INFO: Deleting stale sandbox base /shared/cache/bazel/user_base/31d1af6682b925f850635e639ebdf8e8/sandbox
Target //engineering/projects/aws_lambda_hello_world:aws_lambda_hello_world_bin.venv up-to-date:
  bazel-bin/engineering/projects/aws_lambda_hello_world/aws_lambda_hello_world_bin.venv_create_venv.sh
INFO: Elapsed time: 2.522s, Critical Path: 0.11s
INFO: 3 processes: 3 internal.
INFO: Running command line: bazel-bin/engineering/projects/aws_lambda_hello_world/aws_lambda_hello_world_bin.venv_create_venv.sh
INFO: Build Event Protocol files produced successfully.
INFO: Build completed successfully, 3 total actions
/shared/cache/bazel/user_base/31d1af6682b925f850635e639ebdf8e8/execroot/bazel-out/k8-fastbuild/bin/engineering/projects/aws_lambda_hello_world/aws_lambda_hello_world_bin.venv_create_venv.sh: line 49: external/python3_8_x86_64-unknown-linux-gnu/bin/python3: No such file or directory

Version

Development (host) and target OS/architectures:

Output of bazel --version:
6.0.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
0.1.0

Language(s) and/or frameworks involved:

How to reproduce

No response

Any other information?

No response

Attributes like size etc that are for tests get passed though to py_venv in the py_test macro, causing bazel to error.

Each time the terminal py_binary or py_test is run a venv is created, which takes a bunch of time. We can likely speed this up by creating this in a different action and passing it to the rule.

What happened?

When building a py_binary target, the output contains the whl-files.
The same behavior can be observed when building a container image using rules_oci.

Version

Development (host) and target OS/architectures:

Output of bazel --version:
6.3.2

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

Language(s) and/or frameworks involved:

How to reproduce

see this output for the symptom:

$ pwd
/home/betaboon/src/bazel/rules_py

$ bazel build //examples/django:django
INFO: Analyzed target //examples/django:django (1 packages loaded, 2 targets configured).
INFO: Found 1 target...
Target //examples/django:django up-to-date:
  bazel-bin/examples/django/django
INFO: Elapsed time: 0.412s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

$ find bazel-bin/examples/django/ -iname "*.whl"
bazel-bin/examples/django/django.runfiles/aspect_rules_py/external/pypi_asgiref/asgiref-3.7.2-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/aspect_rules_py/external/pypi_django/Django-4.2.5-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/aspect_rules_py/external/pypi_sqlparse/sqlparse-0.4.4-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/aspect_rules_py/external/pypi_typing_extensions/typing_extensions-4.7.1-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/aspect_rules_py/external/python_toolchain_x86_64-unknown-linux-gnu/lib/python3.9/ensurepip/_bundled/pip-23.0.1-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/aspect_rules_py/external/python_toolchain_x86_64-unknown-linux-gnu/lib/python3.9/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/pypi_asgiref/asgiref-3.7.2-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/pypi_django/Django-4.2.5-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/pypi_sqlparse/sqlparse-0.4.4-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/pypi_typing_extensions/typing_extensions-4.7.1-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/python_toolchain_x86_64-unknown-linux-gnu/lib/python3.9/ensurepip/_bundled/pip-23.0.1-py3-none-any.whl
bazel-bin/examples/django/django.runfiles/python_toolchain_x86_64-unknown-linux-gnu/lib/python3.9/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl

$ find bazel-bin/examples/django/ -iname "django"
bazel-bin/examples/django/
bazel-bin/examples/django/django
bazel-bin/examples/django/django.runfiles/aspect_rules_py/examples/django
bazel-bin/examples/django/django.runfiles/aspect_rules_py/examples/django/django
bazel-bin/examples/django/django.venv.source/lib/python3.9/site-packages/django
bazel-bin/examples/django/django.venv.source/lib/python3.9/site-packages/django/forms/jinja2/django
bazel-bin/examples/django/django.venv.source/lib/python3.9/site-packages/django/forms/templates/django

$ ls bazel-bin/examples/django/django.venv.source/lib/python3.9/site-packages/django
 apps   conf   contrib   core   db   dispatch   forms   http   middleware   template   templatetags   test   urls   utils   views   __init__.py   __main__.py   shortcuts.py

Any other information?

When building a container-image using the example in aspect-build/bazel-examples does not exhibit the same behavior.
This seems to be due to the usage of pip.parse and thus effectively rules_python's py_libary.

When using the builtin autodetecting toolchain, it fails to correctly find the users python binary as it calls which python at runtime, at which point the venv has been activated and the wrapper script from the toolchain finds itself as the result of which python, therefore getting stuck in a loop.

We'd need either need to stipulate that this ruleset is incompatible with the builtin autodetecting toolchain or provide a better autodetecting toolchain that does the lookup at analysis time in a repo rule and therefore can be marked as configurable.

I've seen other folks write tests for things and it seems to pay off in the long run. I think we should do that here too.

Placeholder issue to add tests to rules_py as currently there is only one (although it hits quite a few spots).

What is the current behavior?

No response

Describe the feature

ipython is a good example of a debugging facility that developers likely want in the venv, but it's a pain to include it in the requirements, and isn't needed for running tests or binaries so it's not strictly a dependency.

Currently venv creation is tied to the terminal py_binary or py_test.

Under bazel this is fine and what we want, but for IDEs the user likely wants to create a venv for all the targets in their project, ie, all the test and binary targets in a given scope, or at the very least all the test targets so the IDE can run them.

This will lead to potential differences between testing under bazel and testing via the IDE as tests may have access to dependencies in the IDE that they wouldn't have under bazel.

What happened?

When building a py_binary with --build_python_zip specified, expected .zip artifact is not generated.

Version

Development (host) and target OS/architectures:
Ubuntu 20.04

Output of bazel --version:
bazel 7.0.0 (bazelisk)

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
aspect_rules_py, version: 0.5.0
rules_python, version: 0.22.0

Language(s) and/or frameworks involved:
Python

How to reproduce

# MODULE.bazel

bazel_dep(name = "aspect_rules_py", version = "0.5.0")

bazel_dep(name = "rules_python", version = "0.22.0", dev_dependency = True)

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    name = "python3",
    configure_coverage_tool = True,
    python_version = "3.11",
)
use_repo(python, "python3_toolchains")

register_toolchains(
    "@python3_toolchains//:all",
)

# BUILD

load("@aspect_rules_py//py:defs.bzl", "py_binary")

py_binary(
    name = "script",
    srcs = ["script.py"],
    main = "script.py",
)

Running with: bazel build //:script --build_python_zip

### Any other information?

_No response_

What happened?

The venv.temp.sh script calls python -m venv which generates the venv template (source code).

The outputs have absolute path embedded in activate.sh, activate.fish etc, which would cause nondeterministic outputs.

Version

Development (host) and target OS/architectures:

Output of bazel --version:
bazel 6.3.2

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:
0.5.0

Language(s) and/or frameworks involved:
n/a

How to reproduce

Using the master branch of rules_py

gzh@desk-0181:~/rules_py$ bazel build //examples/django:django
INFO: Invocation ID: 37ecb87f-e0a2-49ba-bfa9-2d0611dcf806
INFO: Analyzed target //examples/django:django (46 packages loaded, 2518 targets configured).
INFO: Found 1 target...
Target //examples/django:django up-to-date:
  bazel-bin/examples/django/django
INFO: Elapsed time: 4.720s, Critical Path: 2.53s
INFO: 8 processes: 1 remote cache hit, 7 internal.
INFO: Build completed successfully, 8 total actions
tusen@desk-0181:~/rules_py$ cat bazel-out/k8-fastbuild/bin/examples/django/django.venv.source/bin/activate | grep VIRTUAL_ENV
    unset VIRTUAL_ENV
VIRTUAL_ENV="/home/tusen/.cache/bazel/_bazel_tusen/e441b6fe7baafb2a17e187ee01b3e54e/sandbox/linux-sandbox/204/execroot/aspect_rules_py/bazel-out/k8-fastbuild/bin/examples/django/django.venv.source"

Any other information?

No response

What is the current behavior?

Users have to paste some starlark into their projects.

Describe the feature

I think we can get to a public API similar to js_image_layer that we're able to commit to, and add to this repository.

What is the current behavior?

No response

Describe the feature

This is useful to get a deployment target which matches the python expectation to resolve at runtime

Fund our work

• Sponsor our open source work by donating a feature bounty

land bazelbuild/rules_python#723

What happened?

It looks like the generated venv isn't complete. I cannot import repo-local packages with repo-relative import mechanism.

Version

Development (host) and target OS/architectures: Ubuntu 20.04

Output of bazel --version: 5.3.2

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

rules_py: c89d948

Language(s) and/or frameworks involved:

Python and Bazel

How to reproduce

I created a simple workspace in a fork: https://github.com/oqton/rules_py/tree/feature/import_path_bug. To reproduce just do:

cd examples/import_path_bug
bazelisk run //libs/bar

On my machine I get:

bazelisk run //libs/bar
INFO: Analyzed target //libs/bar:bar (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //libs/bar:bar up-to-date:
  bazel-bin/libs/bar/bar
INFO: Elapsed time: 0.443s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
[
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python38.zip",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/external/oqton_python_x86_64-unknown-linux-gnu/lib/python3.8/lib-dynload",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/bar.venv/lib/python3.8/site-packages",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/foo",
  "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/bar"
]
Traceback (most recent call last):
  File "/home/mvukov/.cache/bazel/_bazel_mvukov/56518e08ebcaaca6f37ddc3196658013/execroot/import_path_bug/bazel-out/k8-fastbuild/bin/libs/bar/bar.runfiles/import_path_bug/libs/bar/bar_app.py", line 6, in <module>
    from libs.foo import foo_lib
ModuleNotFoundError: No module named 'libs'

With regular Python rules running the target works as expected.

numpy is the gold standard of a package that's hard to build.

What is the current behavior?

Currently our best-effort to make a layered image for Python is three layers: interpreter, site-packages, app:
https://github.com/aspect-build/bazel-examples/blob/main/oci_python_image/py_layer.bzl

However, when a package appears under multiple applications, the site-packages layer for each app will have a duplicate of that package, making the build outputs larger. When pytorch is one such package (it is many gigabytes for the GPU variant) then this problem leads to builds taking minutes just to fetch build inputs from a remote cache.

Describe the feature

We can probably do something with aspects to visit each third-party package and produce a tar file containing its site-packages entry as a distinct layer. Then when we build a container image for an application these tar files should just get re-used rather than being re-built.

If there is a dependency on an external py_library that isn't consumed via the py_wheel rule, eg it's a py_library generated by rules_python for a pip install, and referenced via the :pkg label, then when a venv is created for the test or binary target for IDE consumption, the dependency is missing from the venv, as the .pth file points to the path in the runfiles tree.

What is the current behavior?

No response

Describe the feature

We build a custom Python interpreter into an artifact, which is downloaded at build time. To get rules_py to work, I had to update our interpreter build script to run bin/python -m ensurepip so that the pip package was installed at lib/python3.8/site-packages in the tarball. Otherwise, the rules_py virtualenv stuff fails miserably.

Please document any assumptions about the Python interpreter.

What is the current behavior?

No response

Describe the feature

Similar to what we have for rules_oci

What happened?

bazel run :test.venv
....
INFO: Running command line: bazel-bin/test.venv_create_venv.sh
WARNING: Requirement 'external/deps_attrs/attrs-21.4.0-py2.py3-none-any.whl' looks like a filename, but the file does not exist
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '/bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/execroot/demo/bazel-out/k8-fastbuild/bin/test.venv_create_venv.sh.runfiles/demo/external/deps_attrs/attrs-21.4.0-py2.py3-none-any.whl'

Version

Development (host) and target OS/architectures: linux, linux

Output of bazel --version: 6.3.2

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: 43beacc

Language(s) and/or frameworks involved: python

How to reproduce

Workspace file:

load("@rules_python//python:repositories.bzl", "py_repositories", "python_register_toolchains")

py_repositories()

python_register_toolchains(
    name = "python39",
    python_version = "3.9",
)

load("@rules_python//python:pip.bzl", "package_annotation", "pip_parse")

PY_WHEEL_RULE_CONTENT = """\
load("@aspect_rules_py//py:defs.bzl", "py_wheel")
py_wheel(
    name = "wheel",
    src = ":whl",
)
"""

PACKAGES = [
    "attrs",
]

ANNOTATIONS = {
    pkg: package_annotation(additive_build_content = PY_WHEEL_RULE_CONTENT)
    for pkg in PACKAGES
}

local_repository(
    name = "aspect_rules_py",
    path = "/workspaces/demo/rules_py",
)

pip_parse(
    name = "deps",
    annotations = ANNOTATIONS,
    python_interpreter_target = "@python39_host//:python",
    requirements_lock = "//:requirements.txt",
)

load("@deps//:requirements.bzl", "install_deps")

install_deps()

load("@aspect_rules_py//py:repositories.bzl", "rules_py_dependencies")

rules_py_dependencies()

Build file:

load("@aspect_rules_py//py:defs.bzl", "py_binary")

py_binary(
    name = "test",
    srcs = ["bla.py"],
    deps = [
        "@deps_attrs//:wheel",
    ],
)

Any other information?

WHL_REQUIREMENTS_FILE contains the following:

external/deps_attrs/attrs-21.4.0-py2.py3-none-any.whl

Obviously, if pip will search this path, it will not find it because it didn't go through runfiles/rlocation. PIP_FIND_LINKS will not change anything because it's the exact same list.

The following patch seems to resolve the issue. We just pass the fully rlocationed paths via requirements.txt:

diff --git a/py/private/venv/venv.tmpl.sh b/py/private/venv/venv.tmpl.sh
index 64c2725..f0f4df0 100644
--- a/py/private/venv/venv.tmpl.sh
+++ b/py/private/venv/venv.tmpl.sh
@@ -89,8 +89,21 @@ if [ "$INSTALL_WHEELS" = true ]; then
   # Call to pip to "install" our dependencies. The `find-links` section in the config points to the external downloaded wheels,
   # while `--no-index` ensures we don't reach out to PyPi
   # We may hit command line length limits if passing a large number of find-links flags, so set them on the PIP_FIND_LINKS env var
-  PIP_FIND_LINKS=$(tr '\n' ' ' < "${WHL_REQUIREMENTS_FILE}")
-  export PIP_FIND_LINKS
+
+  RLOCATION_REQUIREMENTS_FILE="rlocation_requirements.txt"
+  if [ -f $RLOCATION_REQUIREMENTS_FILE ] ; then
+    rm $RLOCATION_REQUIREMENTS_FILE
+  fi
+  touch $RLOCATION_REQUIREMENTS_FILE
+
+  for WHL_PATH in $(cat "${WHL_REQUIREMENTS_FILE}")
+  do
+    WHL_PATH_FOR_RLOCATION="${WHL_PATH#external/}"
+
+    WHL_RLOCATION=$(rlocation $WHL_PATH_FOR_RLOCATION)
+
+    echo $WHL_RLOCATION >> $RLOCATION_REQUIREMENTS_FILE
+  done
 
   PIP_FLAGS=(
     "--quiet"
@@ -105,7 +118,7 @@ if [ "$INSTALL_WHEELS" = true ]; then
     "--no-index"
   )
 
-  ${VPIP} install "${PIP_FLAGS[@]}" -r "${WHL_REQUIREMENTS_FILE}"
+  ${VPIP} install "${PIP_FLAGS[@]}" -r "$RLOCATION_REQUIREMENTS_FILE"
 
   unset PIP_FIND_LINKS
 fi

What happened?

Support for py_test.stamp, https://bazel.build/reference/be/python#py_test.stamp

$ bazel test //examples/py:py_test 
ERROR: /home/jamison/code/rules_py/examples/py/BUILD.bazel:3:8: //examples/py:py_test: no such attribute 'stamp' in 'py_test' rule
ERROR: error loading package 'examples/py': Package 'examples/py' contains errors
INFO: Elapsed time: 0.095s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

Version

Development (host) and target OS/architectures:

Output of bazel --version:

bazel 5.4.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

local rules_py workspace, HEAD

How to reproduce

https://github.com/jl-applied/rules_py/tree/jamison/161,

bazel test //examples/py:py_test

Any other information?

No response

What is the current behavior?

It's difficult to work in a Python monorepo with a single set of requirements, because not all teams are using aligned dependency versions today. However, multiple requirements are difficult because a transitive dependency of an application can use a conflicting version from a direct dependency.

Describe the feature

References:

• background reading: https://blog.aspect.dev/multiple-deps
• design doc from Aspect: https://docs.google.com/document/d/1A1ZGSLn7WTbWdBVxNQtB9c3fZTFITprdwhru89_O28A/edit#heading=h.i23ncs9b913x
• specification from Verkada: https://docs.google.com/document/d/1yLNMt3rRbvK80evKOIpqxmwzuMP7THh371L85QkwMx4/edit
  A PR with one implementation: #87

What is the current behavior?

kwargs in py_pytest_main macro is only applied to the underlying _py_pytest_main rule. Thus, testonly does not get applied to the py_library rule which makes up the pytest entry point.

Describe the feature

It would be useful to be able to pass in testonly flag to the underlying py_library rule defined by the py_pytest_main macro.

Specifically here

What is the current behavior?

Running a .venv target will create the virtual environment in the root of the repo, with the name .<target>.venv. The virtual environment is not inherently relocatable due to the relative paths inside of first_party.pth.

Describe the feature

Allow for the virtual environments to be created in the same directory as the rule is defined. The current default naming behavior is fine, but adding support for:

bazel run :target.venv env

to name the resulting virtual environment would be great!

Fund our work

• Sponsor our open source work by donating a feature bounty

Ensure deps are passed correctly via the providers, and the terminal py_binary or py_test can correctly pick up the wheels or add sources to the pth files.

What is the current behavior?

No response

Describe the feature

Probably using rules_oci.

It would be nice to show it working with AWS lambda which has a particular layout required in the image.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

• chore(deps): update dependency rules_rust to v0.48.0

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update dependency boto3 to v1.34.150
• chore(deps): update dependency botocore to v1.34.150
• chore(deps): update dependency exceptiongroup to v1.2.2
• chore(deps): update dependency neptune to v1.10.4
• chore(deps): update dependency org_chromium_sysroot_linux_arm64 to v0.0.20
• chore(deps): update dependency org_chromium_sysroot_linux_x86_64 to v0.0.20
• chore(deps): update dependency referencing to v0.35.1
• chore(deps): update dependency s3transfer to v0.10.2
• chore(deps): update dependency swagger-spec-validator to v3.0.4
• chore(deps): update dependency bazel to v7.2.1
• chore(deps): update dependency bazel_skylib to v1.7.1
• chore(deps): update dependency bazel_skylib_gazelle_plugin to v1.7.1
• chore(deps): update dependency io_bazel_rules_go to v0.49.0
• chore(deps): update dependency io_bazel_stardoc to v0.7.0
• chore(deps): update dependency jsonschema to v4.23.0
• chore(deps): update dependency packaging to v24.1
• chore(deps): update dependency pillow to v10.4.0
• chore(deps): update dependency pluggy to v1.5.0
• chore(deps): update dependency pytest to v8.3.2
• chore(deps): update dependency rpds-py to v0.19.1
• chore(deps): update dependency rules_python to v0.34.0
• chore(deps): update dependency rules_python_gazelle_plugin to v0.34.0
• chore(deps): update dependency jsonpointer to v3
• chore(deps): update dependency numpy to v2
• chore(deps): update dependency psutil to v6
• chore(deps): update dependency webcolors to v24
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency django to v4.2.14 [security]
• chore(deps): update dependency urllib3 to v1.26.19 [security]
• chore(deps): update dependency certifi to v2024.7.4 [security]
• chore(deps): update dependency requests to v2.32.2 [security]
• chore(deps): update dependency sqlparse to v0.5.0 [security]
• chore(deps): update dependency aspect_bazel_lib to v1.42.3
• chore(deps): update dependency bazel_gazelle to v0.37.0
• chore(deps): update dependency typing-extensions to v4.12.2
• fix(deps): update rust crate clap to 4.5.11
• chore(deps): update dependency toolchains_llvm to v1
• chore(deps): update softprops/action-gh-release action to v2
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• chore(deps): update dependency aspect_bazel_lib to v2

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

PyLance seems happy with the pth files in site-packages, but it does only ask the venv interpreter once what's on the path. Currently these paths root seem incorrect as absolute paths work as expected.

We may want to consider generating the configuration JSON for PyLance instead as this allows setting search paths and can be cheaply regenerated.

What is the current behavior?

Currently a py_binary target will attempt to symlink its venv on execution

Describe the feature

This prevents deploying py_binary on a read-only filesystem. The error is also not immediately clear since the mkdir will fail silently, resulting in a "directory does not exist" error.

Fund our work

• Sponsor our open source work by donating a feature bounty

What happened?

Calling a py_binary in a subprocess fails if called with extra env vars. I prepared a demonstration in this branch: https://github.com/oqton/rules_py/tree/feature/subprocess_env_bug.

Here is the output I get:

bazelisk test //py/tests/binary_calls_binary/second
INFO: Analyzed target //py/tests/binary_calls_binary/second:second (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //py/tests/binary_calls_binary/second:second (see /home/mvukov/.cache/bazel/_bazel_mvukov/b62d0fb48c016b2423b665eb9db4b100/execroot/aspect_rules_py/bazel-out/k8-fastbuild/testlogs/py/tests/binary_calls_binary/second/second/test.log)
INFO: From Testing //py/tests/binary_calls_binary/second:second:
==================== Test output for //py/tests/binary_calls_binary/second:second:
ERROR: cannot find bazel_tools/tools/bash/runfiles/runfiles.bash
Traceback (most recent call last):
  File "/home/mvukov/.cache/bazel/_bazel_mvukov/b62d0fb48c016b2423b665eb9db4b100/sandbox/linux-sandbox/27/execroot/aspect_rules_py/bazel-out/k8-fastbuild/bin/py/tests/binary_calls_binary/second/second.runfiles/aspect_rules_py/py/tests/binary_calls_binary/second/second_main.py", line 3, in <module>
    subprocess.check_call('py/tests/binary_calls_binary/first/first', env={'FOO': 'bar'})
  File "/home/mvukov/.cache/bazel/_bazel_mvukov/b62d0fb48c016b2423b665eb9db4b100/execroot/aspect_rules_py/external/python_toolchain_x86_64-unknown-linux-gnu/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'py/tests/binary_calls_binary/first/first' returned non-zero exit status 1.
================================================================================
Target //py/tests/binary_calls_binary/second:second up-to-date:
  bazel-bin/py/tests/binary_calls_binary/second/second
INFO: Elapsed time: 1.692s, Critical Path: 1.27s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//py/tests/binary_calls_binary/second:second                             FAILED in 0.6s
  /home/mvukov/.cache/bazel/_bazel_mvukov/b62d0fb48c016b2423b665eb9db4b100/execroot/aspect_rules_py/bazel-out/k8-fastbuild/testlogs/py/tests/binary_calls_binary/second/second/test.log

INFO: Build completed, 1 test FAILED, 2 total actions

Without env flags in the subprocess the test succeeds.

Version

Development (host) and target OS/architectures:

Output of bazel --version: 5.3.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

rules_py: based on c89d948.

Language(s) and/or frameworks involved:

Python

How to reproduce

# Call this when the aforementioned branch is checked out.
bazelisk test //py/tests/binary_calls_binary/second

Any other information?

No response

Fund our work

• Sponsor our open source work by donating a bug bounty