tubular / rules_pygen Goto Github PK

pip_install

Ran this within our Bazel repo and it failed at the following place:

The error message was saying Failed to build wheel X for a number of packages listed in our requirements.txt (eg. pyyaml, httppretty, jsondiff, ...). I looked into the source and ran the command this tool was running like so:

python3.6 -m pip wheel --disable-pip-version-check --requirement third_party/python/requirements.txt --wheel-dir /tmp/blah

This gave more detailed error messages, one of which being this:

  Running setup.py clean for stringcase
  Building wheel for wrapt (setup.py) ... error
  Complete output from command /nix/store/f698bw0zrv79vzihz6xwcyd84sncrqp8-python3-3.6.8/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/var/folders/0b/p1vldpsd4f99nc50bv4t4csh0000gn/T/pip-wheel-qejpjlkq/wrapt/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /private/var/folders/0b/p1vldpsd4f99nc50bv4t4csh0000gn/T/pip-wheel-mwqsnd8h:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

This typically indicates a dependency on wheel (thus needing to do pip install wheel). I will investigate this further, but wanted to throw this issue up now in case you recognise immediately what the problem is.

Btw, thanks for open-sourcing this! It's much better than the hacky alternative I wrote at our company. 👍

Sometimes, a package is not packaged correctly upstream. Then, when we install it under the bazel environment, the package cannot be found when we run our code. We want the ability to specify patches we want to add, and rules_pygen will generate something like patch_commands = ["mv package_location/ correct_package_location/"] automatically in the .bzl file.

We do an airgapped build where the CI server has no internet access. We check in the whl files into SCM.

When specifying a requirements.txt with local file references the rules generator logs warnings and then fails with FileNotFound:

# requirements.txt
yarl-1.4.2-cp36-cp36m-manylinux1_x86_64.whl

WARNING: Requirement 'yarl-1.4.2-cp36-cp36m-manylinux1_x86_64.whl' looks like a filename, but the file does not exist

I think that the missing feature is a parameter for :generate to provide a directory (and possibly specify it multiple times) of whl files that can be resolved against when the requirements.txt file has a filename specification.

Description

I can see for this line in the source that you're aware of the problem of handling packages that provide only a .tar.gz.

I just ran into this problem as our requirements.txt file includes avro-python3, which does not provide a .whl.

For posterity, the full DEBUG error lok from rules_pygen was:

Processing wheelinfo for /var/folders/0b/p1vldpsd4f99nc50bv4t4csh0000gn/T/tmpf01qd6hu/avro_python3-1.9.0-py3-none-any.whl
DEBUG:rules_pygen.rules_generator:Wheel name is: avro-python3
DEBUG:rules_pygen.rules_generator:Will find additional wheels for other platforms using prefix: avro_python3-1.9.0
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_jbelotti/a8a1f81a1422a72c79c6c81641a8c816/execroot/data_science/bazel-out/darwin-fastbuild/bin/external/rules_pygen/generator.runfiles/rules_pygen/src/rules_pygen/__main__.py", line 41, in <module>
    gen.run()
  File "/private/var/tmp/_bazel_jbelotti/a8a1f81a1422a72c79c6c81641a8c816/execroot/data_science/bazel-out/darwin-fastbuild/bin/external/rules_pygen/generator.runfiles/rules_pygen/src/rules_pygen/rules_generator.py", line 289, in run
    deps = self._parse_wheel_dependencies(wheel_links)
  File "/private/var/tmp/_bazel_jbelotti/a8a1f81a1422a72c79c6c81641a8c816/execroot/data_science/bazel-out/darwin-fastbuild/bin/external/rules_pygen/generator.runfiles/rules_pygen/src/rules_pygen/rules_generator.py", line 424, in _parse_wheel_dependencies
    raise PyBazelRuleGeneratorException("Dependency {} is missing wheels!".format(dependency))
rules_pygen.rules_generator.PyBazelRuleGeneratorException: Dependency avro_python3 is missing wheels!

Would you be interested in a PR, or are you already in-progress on this problem?

Thanks for making and sharing these rules ❤️

I've tried to follow the directions, but I'm having trouble with numpy, and I wonder if I missed something in the setup. I'm running on MacOS, and trying to run a py_binary and a docker py3_image. With both bazel run :demo and bazel run :ddemo, I get an error when importing numpy about ImportError: cannot import name 'WinDLL'.

I've created an example repo here that reproduces the problem: https://github.com/kindlyops/bazel-python-demo

Do I need to do something different with the config_setting calls? I'm not certain I correctly followed the instructions about them being hardcoded at //tool_bazel

On account of this recent change to bazel rules_pygen produces the following problem on 0.29.1:

ERROR: /private/var/tmp/_bazel_jonathonbelotti/1d31e26348def3cbe892454dd398945f/external/pypi__click_7_0/BUILD.bazel:3:1: name 'native' is not defined
ERROR: /private/var/tmp/_bazel_jonathonbelotti/1d31e26348def3cbe892454dd398945f/external/pypi__click_7_0/BUILD.bazel:3:1: name 'native' is not defined