bazel-contrib / bazel-mypy-integration Goto Github PK
View Code? Open in Web Editor NEWππΏπ Integrate MyPy type-checking into your Python Bazel builds
License: MIT License
ππΏπ Integrate MyPy type-checking into your Python Bazel builds
License: MIT License
Hello, I've met a problem while intergrating the new mypy-intergration. The problem might be that I have relied on many rules package like
rules_bison rules_cc rules_flex rules_foreign_cc rules_foreign_cc_bazel_version rules_java rules_m4 rules_proto
rules_python and etc
The error was
'struct' value has no field or method 'to_list'
Available attributes: difference, disjoint, intersection, is_equal, is_subset, union
I think it's the problem with bazel_skylib version conflits. While in mypy, we use version 1.0.2. It might have been used a lower version in the other packages. But in mypy, we have
def repositories():
excludes = native.existing_rules().keys()
rules_python_version = "0.1.0"
if "bazel_skylib" not in excludes:
http_archive(
name = "bazel_skylib",
urls = [
"https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.0.2/bazel-skylib-1.0.2.tar.gz",
"https://github.com/bazelbuild/bazel-skylib/releases/download/1.0.2/bazel-skylib-1.0.2.tar.gz",
],
sha256 = "97e70364e9249702246c0e9444bccdc4b847bed1eb03c5a3ece4f83dfe6abc44",
)
If bazel_skylib is already installed with a old version by the other packages, the new version 1.0.2 will not be installed. This will cause the problem I met. So, how could I do with this problem. Thanks you so much !
Cool project π
It looks pretty low effort to bind this to a rule https://docs.bazel.build/versions/2.0.0/skylark/aspects.html#invoking-the-aspect-from-a-rule
Are there challenges that are you stopping you from doing this? Are there reasons to prefer strictly an aspect approach?
I'd like to run mypy when testing and not when doing a normal build. Having a mypy_test
rule seems like a natural extension.
Happy to give this a stab if you're low on bandwidth.
Currently all code in examples/
is Python. This is not good for integration testing as a real-world use-case will involve ignoring non-Python code that does exist in the WORKSPACE.
examples/
This module https://registry.bazel.build/modules/rules_mypy is new and claims to resolve some of the issues in this repo.
It also has enthusiastic maintainers and not so much legacy baggage.
@adzenith I think we should compare this repo with that one to decide if it's clearly a superset of what we are doing.
Had this show up when I tried to integrate this tool with the work monorepo. The --cache-map
list is enormous, starting with:
--cache-map 'production/backend/deploy/buildkite/deploy_service_pipeline.py'
'production/backend/deploy/buildkite/deploy_service_pipeline.py.meta.json'
'production/backend/deploy/buildkite/deploy_service_pipeline.py.data.json'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/__init__.py'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/__init__.py.meta.json'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/__init__.py.data.json'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_compat.py'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_compat.py.meta.json'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_compat.py.data.json'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_constants.py'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_constants.py.meta.json'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_constants.py.data.json'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_native.py'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_native.py.meta.json'
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_native.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/__init__.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/__init__.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/__init__.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/_compat.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/_compat.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/_compat.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/_identifier.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/_identifier.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/_identifier.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncfilters.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncfilters.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncfilters.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncsupport.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncsupport.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncsupport.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/bccache.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/bccache.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/bccache.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/compiler.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/compiler.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/compiler.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/constants.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/constants.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/constants.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/debug.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/debug.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/debug.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/defaults.py'
'external/canva_pip_deploy/pypi__jinja2/jinja2/defaults.py.meta.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/defaults.py.data.json'
'external/canva_pip_deploy/pypi__jinja2/jinja2/environment.py' 'external/canva
I think I didn't discover this because I don't have an example in this project that uses an external Python package.
Hi, considering there was a case like
a.py is import by b.py and c.py, and there is 2 target b and c in BUILD file.
When running bazel-mypy on b and c, will a.py be checked 2 times, once in each target ?
At time of writing, the README
offers Mypy version 0.750 in it's demo code/config. It can be found that this version is not compatible with Python 3.9, producing a great number of "syntax error in type comment"
errors when attempting to run a type-checking aspect or rule. See this issue: python/mypy#9916.
The investigation of this issue was done in the following reproduction repo: github.com/bogdan-lytvynov/bazel-mypy-integration-failed
If you clone that repo and attempt to run the integration using Python 3.9, with --test_output=all
you will see the following kind of error lines:
/usr/local/lib/python3.9/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:41: error: syntax error in type comment
/usr/local/lib/python3.9/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:42: error: syntax error in type comment
/usr/local/lib/python3.9/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:43: error: syntax error in type comment
/usr/local/lib/python3.9/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:46: error: syntax error in type comment
...
To see that the error is not coming from the Bazel integration, I did python3 -m pip install mypy==0.750
and ran mypy print/print.py
and got the same errors.
Action:
This is not a bug in the integration but I'll add a note in the README about it.
I'm filing this bug so that people can subscribe to it to find out when this exists in the bazel central registry.
I've tried several different ways of referencing packages like types-python-dateutil
from PyPI and I can't get any of them to actually provide the type stubs to mypy using bazel-mypy-integration
.
Here's a repo where I've demonstrated the approaches I've tried.
requirement()
a dependency of the py_binary
target passed to mypy_test
.mypy_stubs
target manually pointing at the .pyi
files inside the pypi target.mypy_stubs
target pointing at the pypi target as returned by requirement()
. This one causes bazel errors because the py_library
rule internal to the pip_install
workspace rule does not include .pyi
files in its srcs
(only in data
), as I understand it.When running bazel test --test_output=all //:uses_deps_mypy
, I see the following output:
INFO: From Testing //:uses_deps_mypy:
==================== Test output for //:uses_deps_mypy:
INFO: Analyzed target //:uses_deps_mypy (33 packages loaded, 1623 targets configured).
INFO: Found 1 test target...
FAIL: //:uses_deps_mypy
uses-deps.py:1: error: Library stubs not installed for "dateutil" (or incompatible with Python 3.8)
uses-deps.py:1: note: Hint: "python3 -m pip install types-python-dateutil"
uses-deps.py:1: note: (or run "mypy --install-types" to install all missing stub packages)
uses-deps.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)
================================================================================
Target //:uses_deps_mypy up-to-date:
bazel-bin/uses_deps_mypy
INFO: Elapsed time: 3.973s, Critical Path: 3.21s
INFO: 7 processes: 5 internal, 2 processwrapper-sandbox.
INFO: Build completed, 1 test FAILED, 7 total actions
//:uses_deps_mypy FAILED in 2.6s
Interestingly, if I run mypy manually using the stubs as downloaded by bazel from pypi, I get the same error:
$ MYPYPATH=~/.cache/bazel/_bazel_fwingerter/13dabd0983aec3a797d4285d705dd267/external/mypy_stubs/pypi__types_python_dateutil/ mypy uses-deps.py
uses-deps.py:1: error: Library stubs not installed for "dateutil" (or incompatible with Python 3.8)
uses-deps.py:1: note: Hint: "python3 -m pip install types-python-dateutil"
uses-deps.py:1: note: (or run "mypy --install-types" to install all missing stub packages)
uses-deps.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
uses-deps.py:2: error: Library stubs not installed for "dateutil.parser" (or incompatible with Python 3.8)
Found 2 errors in 1 file (checked 1 source file)
But if I rename the directory to dateutil
, it works (obviously this is not sane as one shouldn't muck around in the bazel cache, but it helps demonstrate that mypy and the bazel plumbing do not seem to agree on directory structure):
$ cd ~/.cache/bazel/_bazel_fwingerter/13dabd0983aec3a797d4285d705dd267/external/mypy_stubs/pypi__types_python_dateutil
$ cp -ar dateutil-stubs dateutil
$ cd -
$ MYPYPATH=~/.cache/bazel/_bazel_fwingerter/13dabd0983aec3a797d4285d705dd267/external/mypy_stubs/pypi__types_python_dateutil/ mypy uses-deps.py
Success: no issues found in 1 source file
What is the intended way to use stub libraries from PyPI with bazel-mypy-integration
?
Hi, when I upgraded this repo from 0.2.1 to 0.4.0 I started getting the following build error:
INFO: Invocation ID: 3ac6f011-c75e-44e9-be22-5c78d2ade8f6
ERROR: /home/dummy/BUILD:27:
12: every rule of type mypy_test implicitly depends upon the target
'@mypy_integration//:mypy_config', but this target could not be foun
d because of: error loading package '@mypy_integration//': Unable to
find package for @buildifier_prebuilt//:rules.bzl: The repository '
@buildifier_prebuilt' could not be resolved: Repository '@buildifier
_prebuilt' is not defined.
ERROR: Analysis of target '//dummy:test_mypy' fa
iled; build aborted: Analysis failed
I can fix the errors by patching this repo as follows. Obviously these lines are there for a reason and I am not saying you should comment them out, just hoping this might help pinpoint the error.
diff --git a/BUILD b/BUILD
index c25a0c6..b426456 100644
--- a/BUILD
+++ b/BUILD
@@ -1,18 +1,18 @@
-load("@buildifier_prebuilt//:rules.bzl", "buildifier")
+# load("@buildifier_prebuilt//:rules.bzl", "buildifier")
package(default_visibility = ["//visibility:private"])
-buildifier(
- name = "buildifier.check",
- lint_mode = "warn",
- mode = "diff",
-)
+# buildifier(
+# name = "buildifier.check",
+# lint_mode = "warn",
+# mode = "diff",
+# )
-buildifier(
- name = "buildifier",
- lint_mode = "fix",
- mode = "fix",
-)
+# buildifier(
+# name = "buildifier",
+# lint_mode = "fix",
+# mode = "fix",
+# )
licenses(["notice"]) # MIT
I see a 0.5.0
tag, and some comments on other bugs that look like it was intended to be a release. The recent changes to publish a release artefact look to have failed, or at least didn't work for 0.5.0; there are no 'notes' anywhere; and github thinks the most recent release is 0.4.0.
Last version of mypy/main.py that allowed positional arguments:
https://github.com/python/mypy/blob/v0.971/mypy/main.py
Keyword arguments are required from 0.981 onwards (circle September 8, 2022):
https://github.com/python/mypy/blob/v0.981/mypy/main.py
The fix is straightforward and will requires a simple edit to this line:
https://github.com/bazel-contrib/bazel-mypy-integration/blob/main/mypy/main.py#L7
Upgrading to rules_python-0.13.0
results in
external/mypy_integration/mypy/BUILD:3:10: no such package '@mypy_integration_pip_deps_mypy//': The repository '@mypy_integration_pip_deps_mypy' could not be resolved: Repository '@mypy_integration_pip_deps_mypy' is not defined and referenced by '@mypy_integration//mypy:mypy'
which is likely due to bazelbuild/rules_python#807 which suggest to change to pip_parse
instead of pip_install
.
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
examples/WORKSPACE
rules_python %7Bversion%7D
bazel_skylib 1.7.1
repositories/repositories.bzl
rules_python 0.27.1
bazel_skylib 1.7.1
MODULE.bazel
examples/MODULE.bazel
rules_java 7.6.5
rules_python 0.27.1
.bazelversion
bazel 6.5.0
.github/workflows/continuous-integration.yml
actions/checkout v4
p0deje/setup-bazel 0.8.5
actions/checkout v4
.github/workflows/release.yml
actions/checkout v4
actions/cache v4
softprops/action-gh-release v2
examples/third_party/requirements.txt
parse ==1.12.1
examples/tools/typing/mypy_requirements.txt
mypy ==0.750
mypy-extensions ==0.4.4
typed-ast ==1.4.3
typing-extensions ==4.12.2
third_party/requirements.txt
mypy ==0.790
mypy-extensions ==0.4.4
typed-ast ==1.4.3
typing-extensions ==4.12.2
Will the bazel-mypy-integration
be superseded by the bazel-linting-system
?
Thanks, again, for your efforts on both fronts.
This seems like it could be similar to #15 but I'm opening a new issue because it has nothing to do with protobuf. I have a plain Python application with no external dependencies that only uses rules_python
and Bzlmod.
I am consistently getting the error
main_mypy.runfiles is not a valid Python package name
when running bazel test
.
My source file is
main.py
def sayHello():
return f"Hello, world!"
if __name__ == "__main__":
print(sayHello())
and my BUILD.bazel
contains:
py_binary(
name = "main",
srcs = ["main.py"],
visibility = ["//:__subpackages__"],
)
mypy_test(
name = "main_mypy",
deps = [
":main",
],
)
Lines from relevant files:
MODULE.bazel
bazel_dep(
name = "rules_python",
version = "0.31.0",
)
bazel_dep(
name = "rules_python_gazelle_plugin",
version = "0.31.0",
)
python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
configure_coverage_tool = True,
python_version = "3.12",
)
bazel_dep(
name = "mypy_integration",
version = "0.0.0",
)
git_override(
module_name = "mypy_integration",
remote = "https://github.com/bazel-contrib/bazel-mypy-integration",
commit = "c289c0cd30e53b5cc467b601295834dbc400f4b2",
)
pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
hub_name = "mypy_pip",
python_version = "3.12",
requirements_lock = "//third_party/python/mypy:requirements_lock.txt",
)
use_repo(pip, "mypy_pip")
.bazelrc
build --@mypy_integration//:mypy=//third_party/python/mypy:mypy
third_party/python/mypy/BUILD.bazel
load("@mypy_pip//:requirements.bzl", "requirement")
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
compile_pip_requirements(
name = "requirements",
src = "requirements.txt",
requirements_txt = "requirements_lock.txt",
)
alias(
name = "mypy",
actual = requirement("mypy"),
visibility = ["//visibility:public"],
)
Hi, I work on a large monorepo that has multiple projects owned by different people. I'm migrating these projects to bazel and currently they each have their own mypy.ini
file.
From reading the setup instructions it seems that they are meant to have a single configuration file for the whole repo, is there a way around this?
This bazel plugin works well with mypy 0.770 but after upgrading to 0.780 it complains with
"source file found twice under different module names error". mypy 0.780 works fine outside of bazel.
There is a related issue on mypy: python/mypy#8944. In my case, directories on mypy_path
do not contain __init__.py
files.
mypy_path
within mypy.ini
. Even if I disable that, the problem persists.$(pwd)
to the front of MYPYPATH. I tried playing around with that but still hitting the same issue.So, I am not very sure where the problem comes from.
EDIT: usually I am setting PYTHONPATH in .bazelrc and mypy_path in mypy.ini but even if I add imports = ["<path e.g. ..>"]
in all BUILD files the above steps still apply.
This new feature in bazel might make build --output_groups=+mypy
unnecessary: https://docs.bazel.build/versions/main/skylark/rules.html#validation-actions.
Trying to integrate bazel-mypy-integration
on a project using Python 3.8.6, I get the error "The typed_ast package is not installed". As this is a transitive dependency, I considered that I may have a Python binary mismatch of some kind, but I can't find any indication of that.
This is using bazel-mypy-integration-0.1.0
and rules-python-0.1.0
.
I'm excited to try bazel-mypy-integration
out - thank you for all the effort! Here's the build error:
INFO: Invocation ID: a0d1c3a2-24ca-4731-aaeb-d0fe41c60b45
DEBUG: /private/var/tmp/_bazel_dan/cebcfff554bef27d2846af867a1d4809/external/rules_python/python/legacy_pip_import/pip.bzl:143:10: DEPRECATED: the pip_import rule has been replaced with pip_install, please see rules_python 0.1 release notes
INFO: Analyzed target //processors/context/cli:cli (1 packages loaded, 2 targets configured).
INFO: Found 1 target...
ERROR: /Users/dan/src/10x/data-fixer/datafixer/processors/context/cli/BUILD.bazel:4:10: Type-checking //processors/context/cli:cli failed (Exit 1): cli_mypy_exe failed: error executing command bazel-out/darwin-fastbuild/bin/processors/context/cli/cli_mypy_exe
Use --sandbox_debug to see verbose messages from the sandbox cli_mypy_exe failed: error executing command bazel-out/darwin-fastbuild/bin/processors/context/cli/cli_mypy_exe
Use --sandbox_debug to see verbose messages from the sandbox
The typed_ast package is not installed.
You can install it with `python3 -m pip install typed-ast`.
Aspect @mypy_integration//:mypy.bzl%mypy_aspect of //processors/context/cli:cli failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.823s, Critical Path: 0.69s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
Finally found some open-sourced MyPy integration code from Dropbox: https://github.com/dropbox/dbx_build_tools/blob/master/build_tools/py/mypy.bzl
They seem to do things in a pretty Dropbox-specific way, but I'm sure there's plenty to be learned from their code.
Given there was no activity here for almost a year and my attempt of contribution received no feedback I am wondering if this project is still alive.
I am creating this issue in the hope this has more visibility than creating a PR.
external/mypy_integration/mypy/BUILD:3:10: no such package '@mypy_integration_pip_deps_mypy//': The repository '@mypy_integration_pip_deps_mypy' could not be resolved: Repository '@mypy_integration_pip_deps_mypy' is not defined and referenced by '@mypy_integration//mypy:mypy'
I've correctly copied the bazel WORKSPACE snipped, but can't fix this issue when invoking Python builds.
This is my Python config
#### Build Rules for Python
http_archive(
name = "rules_python",
sha256 = "a644da969b6824cc87f8fe7b18101a8a6c57da5db39caa6566ec6109f37d2141",
strip_prefix = "rules_python-0.20.0",
url = "https://github.com/bazelbuild/rules_python/releases/download/0.20.0/rules_python-0.20.0.tar.gz",
)
load("@rules_python//python:repositories.bzl", "py_repositories")
py_repositories()
load("@rules_python//python:repositories.bzl", "python_register_toolchains")
python_register_toolchains(
name = "python3_10",
# Available versions are listed in @rules_python//python:versions.bzl.
# We recommend using the same version your team is already standardized on.
python_version = "3.10",
)
load("@python3_10//:defs.bzl", "interpreter")
load("@rules_python//python:pip.bzl", "install")
pip_parse(
name = "pip",
python_interpreter_target = interpreter,
requirements_lock = "//third_party/python:requirements.txt",
)
load("@pip//:requirements.bzl", "install_deps")
install_deps()
#######
I'm also using the aspect integration in .bazelrc:
build --aspects @mypy_integration//:mypy.bzl%mypy_aspect
build --output_groups=+mypy
Hi, I recently meet a problem when using bazel-mypy-integration and I wonder if you have a way to solve it.
I use rule_python to control pip dependencies, but when running mypy check like
bazel build --aspects @mypy_integration//:mypy.bzl%mypy_aspect --output_groups=mypy //test:test_mypy_with_numpy
with the BUILD file
load("@py_deps//:requirements.bzl", "requirement")
py_library(
name = 'test_mypy_with_numpy',
srcs = [test_mypy_with_numpy.py'],
deps = [
requirement('numpy'),
]
I found numpy related code wasn't checked. But when I use pip install numpy, it will be checked.
So is there any way to let mypy_integration use requirement from rule_python ?
An aspect can require a toolchain to be available during execution, so this aspect should depend on the Python toolchain.
A decent example of how this is done is in the rules_rust
clippy aspect
I hit the problem because I'm using https://github.com/pubref/rules_protobuf
In general, if generated Python code is referenced, it will cause an error. In the sandbox, the files has paths like bazel-out/k8-fastbuild/bin/...
.
It's making the entire tool unusable, one option could be skipping files in bazel-out dir for now.
Even after trying:
[mypy]
exclude = .* # exclude everything
I cannot get the aspect to exclude files when building. I'm certain my mypy.ini
is being picked up correctly as other settings do work
First, thanks for working on this integration. I'm excited to incorporate it into our Python 3 code base.
With some of my initial tests I'm seeing numerous errors of the type Cannot find implementation or library stub for module named ...
. I suspect the issue is our use of the imports
attribute on a shared py_library
rule.
Our code is structured such that we have a src/python3
directory in our repository which our imports are relative to.
The majority of our packages live under a subdirectory called uc
so most of our imports start with uc.
. We have a py_library rule that everything depends on directly or transitively that has imports = [".."]
. This works well for us with Bazel so far but I'm wondering if it's what is causing the issue.
I noticed that you had #8 which looks at imports for py_test
and py_binary
so I'm wondering if it's the combination of py_library
and ..
that isn't handled.
Any ideas? Thanks!
Hi Jono,
I have reproducible repo to illustrate the issue that I am having, the mypy
throws error when I am using combination of:
build --build_python_zip
and mypy_test
.
git clone https://github.com/kororo/proto-bazel
cd proto-bazel
β proto-bazel git:(master) bazel test //examples/foobar/src:foobar_mypy_test
INFO: Analyzed target //examples/foobar/src:foobar_mypy_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //examples/foobar/src:foobar_mypy_test (see /private/var/tmp/_bazel_wizz/7d4571e5262c107c98e149dc3e1a3e95/execroot/proto-bazel/bazel-out/darwin-fastbuild/testlogs/examples/foobar/src/foobar_mypy_test/test.log)
INFO: From Testing //examples/foobar/src:foobar_mypy_test:
==================== Test output for //examples/foobar/src:foobar_mypy_test:
usage: mypy [-h] [-v] [-V] [more options; see below]
[-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: Cannot find config file 'external/mypy_integration_config/mypy.ini'
================================================================================
Target //examples/foobar/src:foobar_mypy_test up-to-date:
bazel-bin/examples/foobar/src/foobar_mypy_test
INFO: Elapsed time: 4.760s, Critical Path: 4.60s
INFO: 2 processes: 2 darwin-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//examples/foobar/src:foobar_mypy_test FAILED in 4.2s
/private/var/tmp/_bazel_wizz/7d4571e5262c107c98e149dc3e1a3e95/execroot/proto-bazel/bazel-out/darwin-fastbuild/testlogs/examples/foobar/src/foobar_mypy_test/test.log
INFO: Build completed, 1 test FAILED, 2 total actions
If I commented out the build --build_python_zip
in .bazelrc
, it works well. Worth to mention, mypy integration with aspect is working as expected. I suspect there is something missing in the link with the export_files
.
Hi, just reporting this as FYI. It's not really causing me any trouble, I'm happy on mypy version 0.812, just thought I'd start a thread in case this is impacting others.
If I change my mypy_version.txt
file to contain:
mypy==0.900
Then I get this error when I try to build a mypy_test
ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/mypy_integration/mypy/BUILD:3:10: no such package '@mypy_integration_pip_deps//pypi__typed_ast': BUILD file not found in directory 'pypi__typed_ast' of external repository @mypy_integration_pip_deps. Add a BUILD file to a directory to mark it as a package. and referenced by '@mypy_integration//mypy:mypy'
I wonder if this is something to do with the release notes of mypy 0.900: http://mypy-lang.org/news.html
The next mypy release will no longer bundle stubs for third-party libraries. Read this blog post if you are interested in what this means to mypy users and why we are doing this.
I just implemented this integration and for the most part things went smooth. But now I stumbled into a case that I cannot seem to solve, see the example below.
Running below in a virtual env where pytest is installed works, however running it from the base env (without pytest) fails. All other pip depencies do get loaded correctly via bazel, only a few pytest decorators give errors. Running mypy test_foo.py
also passes in the virutalenv (using the same mypy.ini
file).
I'm relatively new to bazel and mypy, so not sure where to start digging. Any thoughts?
WORKSPACE
workspace(name = "myproject")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
#############################################################################
# Python Rules
#############################################################################
http_archive(
name = "rules_python",
sha256 = "b6d46438523a3ec0f3cead544190ee13223a52f6a6765a29eae7b7cc24cc83a0",
url = "https://github.com/bazelbuild/rules_python/releases/download/0.1.0/rules_python-0.1.0.tar.gz",
)
load("@rules_python//python:repositories.bzl", "py_repositories")
py_repositories()
load("@rules_python//python:pip.bzl", "pip_install")
# Create a central repo that knows about the dependencies needed for
# requirements.txt.
pip_install(
name = "eng_pip_deps",
requirements = "//:requirements.txt",
)
#############################################################################
# Mypy aspect plugin
#############################################################################
mypy_integration_version = "0.1.0" # latest @ November 15th 2020
http_archive(
name = "mypy_integration",
sha256 = "511ca642294129b7abebf6afd48aa63e7d91de3ec5fa0689d60d1dc6a94a7d1a",
strip_prefix = "bazel-mypy-integration-{version}".format(version = mypy_integration_version),
url = "https://github.com/thundergolfer/bazel-mypy-integration/archive/{version}.tar.gz".format(
version = mypy_integration_version,
),
)
load(
"@mypy_integration//repositories:repositories.bzl",
mypy_integration_repositories = "repositories",
)
mypy_integration_repositories()
load("@mypy_integration//:config.bzl", "mypy_configuration")
mypy_configuration(mypy_config_file = "//:mypy.ini")
load("@mypy_integration//repositories:deps.bzl", mypy_integration_deps = "deps")
mypy_integration_deps(mypy_requirements_file = "//:mypy_version.txt")
BUILD
load("@eng_pip_deps//:requirements.bzl", "requirement")
load("@rules_python//python:defs.bzl", "py_test")
py_test(
name = "test_foo",
srcs = ["test_foo.py"],
python_version = "PY3",
deps = [
requirement("pytest"),
requirement("pytest-asyncio"),
],
)
test_foo.py
import pytest
@pytest.mark.asyncio
async def test_foo_noop() -> None:
pass
result
$ bazel build //:test_foo [16:49:26]
(16:49:33) INFO: Current date is 2021-01-19
(16:49:33) INFO: Analyzed target //:test_foo (0 packages loaded, 0 targets configured).
(16:49:33) INFO: Found 1 target...
(16:49:34) ERROR: /home/myproject/BUILD:4:8: Couldn't build file test_foo_dummy_out: Type-checking //:test_foo failed (Exit 1): test_foo_mypy_exe failed: error executing command bazel-out/k8-fastbuild/bin/test_foo_mypy_exe
Use --sandbox_debug to see verbose messages from the sandbox test_foo_mypy_exe failed: error executing command bazel-out/k8-fastbuild/bin/test_foo_mypy_exe
Use --sandbox_debug to see verbose messages from the sandbox
test_foo.py:3: error: Untyped decorator makes function "test_foo_noop" untyped
@pytest.mark.asyncio
^
Found 1 error in 1 file (checked 1 source file)
Aspect @mypy_integration//:mypy.bzl%mypy_aspect of //:test_foo failed to build
Use --verbose_failures to see the command lines of failed build steps.
(16:49:34) INFO: Elapsed time: 1.803s, Critical Path: 1.61s
(16:49:34) INFO: 2 processes: 2 internal.
(16:49:34) FAILED: Build did NOT complete successfully
FAIL
Currently the MyPy py_binary
this integration ships gets its dependencies using rules_python
's pip_import
: https://github.com/thundergolfer/bazel-mypy-integration/blob/master/repositories/py_repositories.bzl#L22
It's configured to currently just look for python_interpreter = "python3"
, and now I'm not sure this would work when the underlying interpreter version changes, for example by going from 3.6
to 3.7
.
Looking at the resolved typed_ast
depending inside my Bazel WORKSPACE's outputBase I see:
cat bazel-data-science/external/mypy_integration_pip_deps_pypi__typed_ast_1_4_1/typed_ast-1.4.1.dist-info/WHEEL
Wheel-Version: 1.0
Generator: bdist_wheel (0.33.6)
Root-Is-Purelib: false
Tag: cp36-cp36m-macosx_10_9_x86_64
Which is clearly 3.6 versioned. If the python3
interpreter is upgraded to 3.7 what happens? I think the repository_rule that downloaded the dependencies wouldn't invalidate (because python_interpreter =
or anything else hasn't changed), and the dependencies would be out-of-date.
Bazel docs for repository_rules mention the configure
attr which indicates that a repository_rule "inspects the system for configuration purpose". Should this be set on the repository_rule that does deps downloading?
My current idea is that the python_interpreter
attr of pip_import
should be configurable for the user so that they can specify themselves which interpreter is being used, and have it match how they install dependencies for their WORKSPACE internal code.
Using mypy plugin doesn't seem possible.
cat mypy.ini
[mypy]
ignore_missing_imports = True
plugins = pydantic.mypy
external/mypy_integration_config/mypy.ini:4: error: Error importing plugin 'pydantic.mypy': No module named 'pydantic'
Found 1 error in 1 file (checked 1 source file)
Make _mypy_cli
configurable so users can customise accordingly.
In MyPy if you have a .py
and .pyi
file in the same folder with the same name then MyPy says that the .pyi
should take precedence and the .py
is ignored completely during type-checking.
In a branch, https://github.com/thundergolfer/bazel-mypy-integration/tree/jonathon/supporting-pyi-files, I was passing in both to the --cache-map
and it was throwing out the error:
mypy_stubs/parse_name.pyi: error: Duplicate module named 'mypy_stubs.parse_name' (also at 'mypy_stubs/parse_name.py')
If could in my Starlark code check for this case and then ignore the .py
, but for now I think I just won't support those kinds of .pyi
files.
Open issue about this: python/mypy#3505
This is one approach to #19.
We have a custom py_binary
wrapping mypy that bundles some plugins. Is the project open to allowing users to pass their own mypy? The simple way would be making _mypy_cli
public:
https://github.com/thundergolfer/bazel-mypy-integration/blob/master/mypy.bzl#L23
The more proper way would probably be making a mypy toolchain.
Willing to PR this if it's acceptable.
Not too confident this will be easily fixed on the typeshed
side -> python/typeshed#2521
I have a py_console_script_binary
in my project and the MyPy aspect does not know how to handle it. The build failure is:
usage: mypy [-h] [-v] [-V] [more options; see below]
[-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: Missing target module, package, files, or command.
The following code in _mypy_aspect_impl
does not include the _transition_py_test
and _transition_py_binary
rules from rules_python (code). As a result, mypy did not run for targets that use py_test
from @python//<version>:def.bzl
bazel-mypy-integration/mypy.bzl
Lines 191 to 194 in 863fde2
As a workaround, I created applied a patch in my WORKSPACE and it seem to work:
def _mypy_aspect_impl(_, ctx):
if (ctx.rule.kind not in ["py_binary", "py_library", "py_test", "mypy_test", "_transition_py_test", "_transition_py_binary"] or
ctx.label.workspace_root.startswith("external")):
return []
bazel 6.1.1
0.18.1
c82a2d522d9e8e96ec30c4ab14a8066a09f7483e
(> 0.4.0)Demo workspace (clone of rules_python examples)
Run:
bazel test //tests:\* --aspects @mypy_integration//:mypy.bzl%mypy_aspect --output_groups=mypy
It should produce an error:
tests/version.py:18: error: Incompatible return value type (got "str", expected "int")
The error goes away once patch
line is removed in WORKSPACE
As I understand it, the config.bzl
assumes provided files are mypy.ini
(https://github.com/thundergolfer/bazel-mypy-integration/blob/9e5cafcebfebd310981a02fc0fdf9fbd48350cf9/config.bzl). It would be nice to support pyproject.toml
as well.
Hopefully Aspects are allowed to return DefaultInfo
, and I can do away with having the mypy
output group which must be requested explicitly with --output_groups=+mypy
.
I think if a dependency isn't properly included in a target's deps
list, then the MyPy aspect shouldn't circumvent Bazel and try and find that dependency in order to include it in type-checking; it should instead not look at the dependency and consider it missing.
Using
mypy==0.981
along with
bazel-mypy-integration-0.4.0
results in an (seemingly incorrect) error
Traceback (most recent call last):
File "/home/XXX/.cache/bazel/_bazel_janickm/c58fb2127d9f20dca76a2dc0b0361e0c/sandbox/linux-sandbox/47/execroot/av-depth/bazel-out/host/bin/external/mypy_integration/mypy/mypy.runfiles/mypy_integration/mypy/main.py", line 7, in <module>
main(None, sys.stdout, sys.stderr)
TypeError: main() takes no positional arguments
From what I can tell the called main()
takes three positional arguments. Errors don't show up for lower mypy versions (e.g., mypy==0.971
).
How to integrate this tool into other projects changed much:
It would be great to depend on all those updates through an official release.
Is there a way to configure the output directory? Currently, they get generated next to the sources but I would prefer if they got generated in a hidden directory so that I don't have to see them when I'm navigating through my codebase.
I am trying to stand up a monorepo that runs mypy as part of the build phase. I have been installing third-party libraries via pip_parse
and requirement
in rules_python
. However, when I tried out this integration, I get:
error: Cannot find implementation or library stub for module named "numpy"
for all of the third-party dependencies that I depend on via requirement
. This seems like the most basic use case, so I figure I must be doing something wrong, but I am not seeing anything in the README about third party imports. Is there a way to make this integration aware of my requirements
-based imports?
Thanks,
Phil
Reproduction in https://github.com/thundergolfer/bazel-python-mypy-protobuf/tree/master/bazel_mypy_integration_issue_36
I'm getting:
bazel test //... --test_output=all --sandbox_debug
INFO: Analyzed 5 targets (0 packages loaded, 0 targets configured).
INFO: Found 4 targets and 1 test target...
FAIL: //tools/thing:mypy_test_foo (see /private/var/tmp/_bazel_jonathon/fafac219b6b0ff0ed453a7d553f169f1/execroot/bazel_mypy_integration_issue_36/bazel-out/darwin-fastbuild/testlogs/tools/thing/mypy_test_foo/test.log)
INFO: From Testing //tools/thing:mypy_test_foo:
==================== Test output for //tools/thing:mypy_test_foo:
tools/thing/proto/bling_api_pb2.pyi:26: error: syntax error in type comment [syntax]
In the internal code that hit this issue, the error is:
Use --sandbox_debug to see verbose messages from the sandbox
tools/foo/proto/foo_api_pb2.py: error: Duplicate module named 'tools.foo.proto.foo_api_pb2' (also at 'tools/foo/proto/foo_api_pb2.pyi')
Found 1 error in 1 file (errors prevented further checking)
INFO: Elapsed time: 2.294s, Critical Path: 1.98s
INFO: 5 processes: 5 internal.
FAILED: Build did NOT complete successfully
In that internal error, the integration should be letting the .pyi
file 'win', as is stated in the documentation: https://mypy.readthedocs.io/en/stable/stubs.html#:~:text=Mypy%20uses%20stub%20files%20stored,to%20type%20check%20your%20code.
This is a great project. Thanks for putting this together.
I believe that the sha256 value for 0.0.7 which is listed in the README.md is incorrect. I believe that the actual value is bf7ecd386740328f96c343dca095a63b93df7f86f8d3e1e2e6ff46e400880077
. The corrected
http_archive(
name = "mypy_integration",
sha256 = "bf7ecd386740328f96c343dca095a63b93df7f86f8d3e1e2e6ff46e400880077", # for 0.0.7
strip_prefix = "bazel-mypy-integration-{version}".format(version = mypy_integration_version),
url = "https://github.com/thundergolfer/bazel-mypy-integration/archive/{version}.zip".format(
version = mypy_integration_version,
),
)
The integration attempts to write out cache files using mypy's undocumented Bazel integration (--bazel
, --package-root
, and --cache-map
) but these files are never written (due to sandboxing?). This can be reproduced by via the tests in this repo:
$ ./test.sh
$ find $(bazel info output_base) -name '*.data.json'
# returns nothing
If you temporarily disable sandboxing via --spawn-strategy=standalone
, i.e.:
diff --git a/test/shell/test_mypy.sh b/test/shell/test_mypy.sh
index db1dcc0..a3bbb6c 100644
--- a/test/shell/test_mypy.sh
+++ b/test/shell/test_mypy.sh
@@ -5,7 +5,7 @@ dir=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
runner=$(get_test_runner "${1:-local}")
test_ok_on_valid_imported_mypy_typings() {
- action_should_succeed build --verbose_failures --aspects //:mypy.bzl%mypy_aspect --output_groups=mypy //test:correct_imported_mypy_typings
+ action_should_succeed build --spawn_strategy=standalone --verbose_failures --aspects //:mypy.bzl%mypy_aspect --output_groups=mypy //test:correct_imported_mypy_typings
}
test_ok_on_valid_imported_mypy_test() {
and then run the tests again:
$ bazel clean; ./test.sh
$ find $(bazel info output_base) -name '*.data.json'
/home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/execroot/bazel_mypy_integration/bazel-out/k8-fastbuild/bin/test/correct_generated.py.data.json
$ find $(bazel info output_base) -name '*.meta.json'
/home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/execroot/bazel_mypy_integration/bazel-out/k8-fastbuild/bin/test/correct_generated.py.meta.json
then our the cache files for our transitive dependencies are written to the output directory. However, it also writes our the cache files for the root dependency directory to the workspace:
$ git status
On branch master
Your branch is up to date with 'origin/master'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
test/__init__.data.json
test/__init__.meta.json
test/correct_imported_typings.py.data.json
test/correct_imported_typings.py.meta.json
If you enable verbose mypy logging (via setting the DEBUG
variable in mypy.bzl
to True
and changing the tests so output to stdout and stderr), then you can see mypy's logs:
OG: Cached module os.path has changed interface
LOG: Writing sys /home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/sandbox/processwrapper-sandbox/66/execroot/bazel_mypy_integration/bazel-out/host/bin/mypy/mypy.r
unfiles/mypy_integration_pip_deps/pypi__mypy/mypy/typeshed/stdlib/3/sys.pyi sys.meta.json sys.data.json
LOG: Cached module sys has changed interface
LOG: Writing builtins /home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/sandbox/processwrapper-sandbox/66/execroot/bazel_mypy_integration/bazel-out/host/bin/mypy/m
ypy.runfiles/mypy_integration_pip_deps/pypi__mypy/mypy/typeshed/stdlib/2and3/builtins.pyi builtins.meta.json builtins.data.json
LOG: Cached module builtins has changed interface
LOG: Processing SCC singleton (importlib) as inherently stale with stale deps (builtins importlib.abc types typing)
LOG: Writing importlib /home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/sandbox/processwrapper-sandbox/66/execroot/bazel_mypy_integration/bazel-out/host/bin/mypy/
mypy.runfiles/mypy_integration_pip_deps/pypi__mypy/mypy/typeshed/stdlib/3/importlib/__init__.pyi importlib/__init__.meta.json importlib/__init__.data.json
LOG: Cached module importlib has changed interface
LOG: Processing SCC singleton (test) as inherently stale with stale deps (builtins)
LOG: Writing test ./test/__init__.py test/__init__.meta.json test/__init__.data.json
LOG: Cached module test has changed interface
LOG: Processing SCC singleton (test.correct_generated) as inherently stale with stale deps (builtins typing)
LOG: Writing test.correct_generated bazel-out/k8-fastbuild/bin/test/correct_generated.py bazel-out/k8-fastbuild/bin/test/correct_generated.py.meta.json bazel-out/k8-fastbuild/bin/test/co
rrect_generated.py.data.json
LOG: Cached module test.correct_generated has changed interface
LOG: Processing SCC singleton (test.correct_imported_typings) as inherently stale with stale deps (builtins test.correct_generated)
LOG: Writing test.correct_imported_typings test/correct_imported_typings.py test/correct_imported_typings.py.meta.json test/correct_imported_typings.py.data.json
LOG: Cached module test.correct_imported_typings has changed interface
LOG: No fresh SCCs left in queue
(You'll note in the logs above that cache files appear to be written to 3 separate locations (sandbox/processwrapper-sandbox/66/execroot
, bazel-out/k8-fastbuild/bin/test/
, and the workspace).
I would like to use the changes added in this pull request so that I can use a newer version of mypy: #45
Could you please add a new release?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.