soniaai / rules_poetry

Bazel rules that use Poetry for Python package management

Poetry Rules

Poetry is a package manager for Python. Bazel is a build system. Together they form a powerful, (mostly) hermetic build system for Python.

Installation

Prerequisites

• Poetry 1.0 or higher
• Bazel 1.0 or higher

The rules depend on Poetry's current lock file format. Versions 0.12.x or lower have a different format that is no longer supported.

Workspace Modifications

Add these lines to your project's WORKSPACE file:

# Poetry rules for managing Python dependencies

http_archive(
    name = "com_sonia_rules_poetry",
    sha256 = "8a7a6a5d2ef859ba4309929f3b4d61031f2a4bfed6f450f04ab09443246a4b5c",
    strip_prefix = "rules_poetry-ecd0d9c66b89403667304b11da3bd99764797a63",
    urls = ["https://github.com/soniaai/rules_poetry/archive/ecd0d9c66b89403667304b11da3bd99764797a63.tar.gz"],
)

load("@com_sonia_rules_poetry//rules_poetry:defs.bzl", "poetry_deps")

poetry_deps()

load("@com_sonia_rules_poetry//rules_poetry:poetry.bzl", "poetry")

poetry(
    name = "poetry",
    lockfile = "//:poetry.lock",
    pyproject = "//:pyproject.toml",
    # optional, if you would like to pull from pip instead of a Bazel cache
    tags = ["no-remote-cache"],
)

and use it to transitively pull in package dependencies using the generated dependency macro:

load("@poetry//:dependencies.bzl", "dependency")
load("@rules_python//python:defs.bzl", "py_library")

py_library(
    name = "some_package",
    srcs = ["example.py"],
    deps = [
        dependency("pytorch"),
        dependency("tensorflow"),
    ],
)

Usage

Usage is nearly identical to Poetry without Bazel. To add a dependency just run poetry add <package name>. To remove a package run poetry remove <package name>.

Caveats

• May not work in network isolated sandboxes (untested)
• Cache support is best-effort
• Source wheels are built on demand and may not always produce deterministic results

Rules Comparison

As you're probably aware, there are a number of Python rules for Bazel due to rules_python being insufficient for many real world applications. While there is significant overlap between these rules, they're not identical in implementation nor in capability. A feature summary is listed below:

Feature	rules_poetry	rules_python
Backend	poetry	pip
Incremental	yes	no
Transitive	yes	no
Isolation	yes	no
Hash Validation	yes	depends
Deterministic	yes	yes
Crosstool	no	no

---

• Backend: which program is used for package management.
• Incremental: adding, upgrading or removing packages only affects those that are modified. Unchanged packages are not reinstalled or downloaded.
• Transitive: depending on one package will include all other packages it indirectly depends on.
• Isolation: packages in the system or user site-packages should not be present in Python's path.
• Hash Validation: wheel and source hashes are validated before installation
• Deterministic: builds should be repeatable and cacheable
• Crosstool: cross compilation (target != execution platform)

TODOs

• Investiage using http_file instead of pip download to fetch dependencies
• Expose the Poetry binary as a Bazel py_binary
• Generate wrappers named python3 python3.7 etc
• Add the wrappers to the PATH so the interpreter entrypoint is consistent
• Unpack wheels directly into container layers to improve cacheability
• Improve documentation
• Write more tests

Contributions Welcome!

rules_poetry's Issues

Update the instructions to the latest release

It appears the current instructions on how to actually use this package uses an older version of the code. Creating another release would be beneficial since some bugs have been fixed in the meantime.

Support for package.source

I found out that rules_poetry does not support package.source.

How hard would it be to add this? I only need support for private repositories (one that implements https://www.python.org/dev/peps/pep-0503/), so not git or directories as mentioned in
https://github.com/soniaai/rules_poetry/blob/master/rules_poetry/poetry.bzl#L87

Edit: I added args.add("-i") args.add("...my_repo_url...") here https://github.com/soniaai/rules_poetry/blob/master/rules_poetry/defs.bzl#L138 and that worked, so this doesn't seem too hard.

//test:test_sys_marker is failing

I noticed while working on #9 that //test:test_sys_marker is failing on a clean checkout. Don't know if it's something about my environment or a legitimate regression. I can try to bisect and find when it may have started.

As an aside, that might be a good reason to set up Github actions to run tests on push/PRs

poetry virtualenv python is not used

It's likely what is referred to in the TODOs, but how can I use this to execute the poetry virtualenv python instead of the platform python?

Does rules_poetry understand target platforms / cross-compilation?

I was wondering how rules_poetry might handle this scenario:

1. Host machine is macOS
2. A py_binary called //:app with dependencies from rules_poetry
3. And a Docker image called //:app_image constructed using container_image (from rules_docker) that is based on Linux AMD64 that contains the py_binary

Now, when doing bazel run //:app, Poetry should install for the host machine (in this case macOS)

But when doing bazel build //:app_image , Poetry should install for Linux AMD64, since this is the base of the Docker image

Is there a way to configure rules_poetry for this?

Perhaps with two calls in the WORKSPACE?

poetry(
  name = "poetry_host",
  lockfile = "//:poetry.lock",
  pyproject = "//:pyproject.toml",
)

poetry(
  name = "poetry_linux_amd64",
  lockfile = "//:poetry.lock",
  pyproject = "//:pyproject.toml",
  target = [ "linux_x64" ],
)

Is this possible or even required?

Support poetry 2.0 lock format

Poetry now uses a new 2.0 lock format since v1.3.0. Should update rules_poetry to support it.

Proposal: Deprecate this package

The approach this package takes to getting poetry dependencies predates a lot of major improvements which have been made to rules_python. I've made a sort of reference implementation here of what poetry support with the latest rules_python could look like here: https://github.com/AndrewGuenther/rules_python_poetry

In total, it's under 30 lines of code and it works quite smoothly. Compared to the relatively significant size of this implementation, I just don't see a compelling reason for this project to continue. To be clear, this isn't a shot at the maintainers here. I used this package for a long time and have made some contributions of my own, but the Bazel Python ecosystem has evolved past the need for much of the complexity here.

Bazel 4.0 and above now supports JSON parsing natively

https://docs.bazel.build/versions/main/skylark/lib/json.html

Would be nice to be able to ditch json_parser.bzl...

Mechanism to fetch a `dev` dependency or from any other group?

Is it possible to fetch a dev dependency or from any other group?

target 'library_pytest' not declared in package ''

What I am doing:
I copied the WORKSPACE setup verbatim and added a poetry dependency to one of my targets. This configuration worked, then I worked on some other Bazel functionality for my project. When I tried running tests after I was done with my other functionality, the rules_poetry seem to no longer work.

That other functionality has nothing to do with poetry or pip, but it does involve building wheels from my source code, it uses the standard python_rules py_package and py_wheel for that.

What is the problem:
When I run bazel test ..., I get this error:

bazel test ...
ERROR: /Users/me/Work/dialog_platform/_experimental_lib_for_bazel/BUILD.bazel:16:8: no such target '@poetry//:library_fastapi': target 'library_fastapi' not declared in package '' defined by /private/var/tmp/_bazel_me/69e3e7ed8e1fbf19ba79afe448a27e50/external/poetry/BUILD and referenced by '//_experimental_lib_for_bazel:test_experimental_lib_for_bazel'
ERROR: /Users/me/Work/dialog_platform/_experimental_lib_for_bazel/BUILD.bazel:16:8: no such target '@poetry//:library_pytest': target 'library_pytest' not declared in package '' defined by /private/var/tmp/_bazel_me/69e3e7ed8e1fbf19ba79afe448a27e50/external/poetry/BUILD and referenced by '//_experimental_lib_for_bazel:test_experimental_lib_for_bazel'
ERROR: Analysis of target '//_experimental_lib_for_bazel:test_experimental_lib_for_bazel' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.432s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

This is the complete output of the command.

I tried reproducing this problem on another computer, but couldn't.

I tried running bazel clean and cleaning my poetry caches as well. It did not help