Coder Social home page Coder Social logo

bazel-contrib / bazel-mypy-integration Goto Github PK

View Code? Open in Web Editor NEW
115.0 6.0 44.0 198 KB

πŸπŸŒΏπŸ’š Integrate MyPy type-checking into your Python Bazel builds

License: MIT License

Python 7.78% Shell 33.60% Starlark 58.62%
mypy bazel python type-checking

bazel-mypy-integration's People

Contributors

adzenith avatar alexeagle avatar caseyduquettesc avatar dmadisetti avatar harrysarson avatar josh-newman avatar jsharpe avatar lamcw avatar martis42 avatar mhansen avatar purkhusid avatar radixseven avatar renovate[bot] avatar robbrit avatar thundergolfer avatar thundergolfer-two avatar uebelandre avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

bazel-mypy-integration's Issues

bazel_skylib version problem

Hello, I've met a problem while intergrating the new mypy-intergration. The problem might be that I have relied on many rules package like

rules_bison rules_cc rules_flex rules_foreign_cc rules_foreign_cc_bazel_version rules_java rules_m4 rules_proto
rules_python and etc

The error was

'struct' value has no field or method 'to_list'
Available attributes: difference, disjoint, intersection, is_equal, is_subset, union

I think it's the problem with bazel_skylib version conflits. While in mypy, we use version 1.0.2. It might have been used a lower version in the other packages. But in mypy, we have

def repositories():
    excludes = native.existing_rules().keys()

    rules_python_version = "0.1.0"

    if "bazel_skylib" not in excludes:
        http_archive(
            name = "bazel_skylib",
            urls = [
                "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.0.2/bazel-skylib-1.0.2.tar.gz",
                "https://github.com/bazelbuild/bazel-skylib/releases/download/1.0.2/bazel-skylib-1.0.2.tar.gz",
            ],
            sha256 = "97e70364e9249702246c0e9444bccdc4b847bed1eb03c5a3ece4f83dfe6abc44",
        )

If bazel_skylib is already installed with a old version by the other packages, the new version 1.0.2 will not be installed. This will cause the problem I met. So, how could I do with this problem. Thanks you so much !

Why aspect over rule? [Question/Feature Request]

Cool project πŸŽ‰

It looks pretty low effort to bind this to a rule https://docs.bazel.build/versions/2.0.0/skylark/aspects.html#invoking-the-aspect-from-a-rule

Are there challenges that are you stopping you from doing this? Are there reasons to prefer strictly an aspect approach?
I'd like to run mypy when testing and not when doing a normal build. Having a mypy_test rule seems like a natural extension.

Happy to give this a stab if you're low on bandwidth.

Don't execute on non-python code

Currently all code in examples/ is Python. This is not good for integration testing as a real-world use-case will involve ignoring non-Python code that does exist in the WORKSPACE.

  1. Add some non-python code into examples/
  2. Fix aspect so it skips non Python targets

`external` Python code gets input to --cache-map, create "Argument list too long" error

Description

Had this show up when I tried to integrate this tool with the work monorepo. The --cache-map list is enormous, starting with:

--cache-map 'production/backend/deploy/buildkite/deploy_service_pipeline.py' 
'production/backend/deploy/buildkite/deploy_service_pipeline.py.meta.json' 
'production/backend/deploy/buildkite/deploy_service_pipeline.py.data.json' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/__init__.py' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/__init__.py.meta.json' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/__init__.py.data.json' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_compat.py' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_compat.py.meta.json' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_compat.py.data.json' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_constants.py' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_constants.py.meta.json' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_constants.py.data.json' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_native.py' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_native.py.meta.json' 
'external/canva_pip_deploy/pypi__markupsafe/markupsafe/_native.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/__init__.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/__init__.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/__init__.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/_compat.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/_compat.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/_compat.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/_identifier.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/_identifier.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/_identifier.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncfilters.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncfilters.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncfilters.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncsupport.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncsupport.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/asyncsupport.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/bccache.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/bccache.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/bccache.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/compiler.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/compiler.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/compiler.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/constants.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/constants.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/constants.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/debug.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/debug.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/debug.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/defaults.py' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/defaults.py.meta.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/defaults.py.data.json' 
'external/canva_pip_deploy/pypi__jinja2/jinja2/environment.py' 'external/canva

I think I didn't discover this because I don't have an example in this project that uses an external Python package.

mypy check python target dependency multi times

Hi, considering there was a case like
a.py is import by b.py and c.py, and there is 2 target b and c in BUILD file.
When running bazel-mypy on b and c, will a.py be checked 2 times, once in each target ?

Mypy 0.750 not compatible with Python 3.9

At time of writing, the README offers Mypy version 0.750 in it's demo code/config. It can be found that this version is not compatible with Python 3.9, producing a great number of "syntax error in type comment" errors when attempting to run a type-checking aspect or rule. See this issue: python/mypy#9916.

The investigation of this issue was done in the following reproduction repo: github.com/bogdan-lytvynov/bazel-mypy-integration-failed

If you clone that repo and attempt to run the integration using Python 3.9, with --test_output=all you will see the following kind of error lines:

/usr/local/lib/python3.9/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:41: error: syntax error in type comment
/usr/local/lib/python3.9/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:42: error: syntax error in type comment
/usr/local/lib/python3.9/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:43: error: syntax error in type comment
/usr/local/lib/python3.9/site-packages/mypy/typeshed/stdlib/2and3/builtins.pyi:46: error: syntax error in type comment
...

To see that the error is not coming from the Bazel integration, I did python3 -m pip install mypy==0.750 and ran mypy print/print.py and got the same errors.


Action:

This is not a bug in the integration but I'll add a note in the README about it.

Stubs from pypi are not found by mypy

I've tried several different ways of referencing packages like types-python-dateutil from PyPI and I can't get any of them to actually provide the type stubs to mypy using bazel-mypy-integration.

Here's a repo where I've demonstrated the approaches I've tried.

  1. Make the pypi target as returned by requirement() a dependency of the py_binary target passed to mypy_test.
  2. Define a mypy_stubs target manually pointing at the .pyi files inside the pypi target.
  3. Define a mypy_stubs target pointing at the pypi target as returned by requirement(). This one causes bazel errors because the py_library rule internal to the pip_install workspace rule does not include .pyi files in its srcs (only in data), as I understand it.

When running bazel test --test_output=all //:uses_deps_mypy, I see the following output:

INFO: From Testing //:uses_deps_mypy:
==================== Test output for //:uses_deps_mypy:
INFO: Analyzed target //:uses_deps_mypy (33 packages loaded, 1623 targets configured).
INFO: Found 1 test target...
FAIL: //:uses_deps_mypy
uses-deps.py:1: error: Library stubs not installed for "dateutil" (or incompatible with Python 3.8)
uses-deps.py:1: note: Hint: "python3 -m pip install types-python-dateutil"
uses-deps.py:1: note: (or run "mypy --install-types" to install all missing stub packages)
uses-deps.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)
================================================================================
Target //:uses_deps_mypy up-to-date:
  bazel-bin/uses_deps_mypy
INFO: Elapsed time: 3.973s, Critical Path: 3.21s
INFO: 7 processes: 5 internal, 2 processwrapper-sandbox.
INFO: Build completed, 1 test FAILED, 7 total actions
//:uses_deps_mypy                                                        FAILED in 2.6s

Interestingly, if I run mypy manually using the stubs as downloaded by bazel from pypi, I get the same error:

$ MYPYPATH=~/.cache/bazel/_bazel_fwingerter/13dabd0983aec3a797d4285d705dd267/external/mypy_stubs/pypi__types_python_dateutil/ mypy uses-deps.py
uses-deps.py:1: error: Library stubs not installed for "dateutil" (or incompatible with Python 3.8)
uses-deps.py:1: note: Hint: "python3 -m pip install types-python-dateutil"
uses-deps.py:1: note: (or run "mypy --install-types" to install all missing stub packages)
uses-deps.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
uses-deps.py:2: error: Library stubs not installed for "dateutil.parser" (or incompatible with Python 3.8)
Found 2 errors in 1 file (checked 1 source file)

But if I rename the directory to dateutil, it works (obviously this is not sane as one shouldn't muck around in the bazel cache, but it helps demonstrate that mypy and the bazel plumbing do not seem to agree on directory structure):

$ cd ~/.cache/bazel/_bazel_fwingerter/13dabd0983aec3a797d4285d705dd267/external/mypy_stubs/pypi__types_python_dateutil
$ cp -ar dateutil-stubs dateutil
$ cd -
$ MYPYPATH=~/.cache/bazel/_bazel_fwingerter/13dabd0983aec3a797d4285d705dd267/external/mypy_stubs/pypi__types_python_dateutil/ mypy uses-deps.py
Success: no issues found in 1 source file

What is the intended way to use stub libraries from PyPI with bazel-mypy-integration?

Error: Repository '@buildifier _prebuilt' is not defined

Hi, when I upgraded this repo from 0.2.1 to 0.4.0 I started getting the following build error:

INFO: Invocation ID: 3ac6f011-c75e-44e9-be22-5c78d2ade8f6
ERROR: /home/dummy/BUILD:27:
12: every rule of type mypy_test implicitly depends upon the target 
'@mypy_integration//:mypy_config', but this target could not be foun
d because of: error loading package '@mypy_integration//': Unable to
 find package for @buildifier_prebuilt//:rules.bzl: The repository '
@buildifier_prebuilt' could not be resolved: Repository '@buildifier
_prebuilt' is not defined.
ERROR: Analysis of target '//dummy:test_mypy' fa
iled; build aborted: Analysis failed

I can fix the errors by patching this repo as follows. Obviously these lines are there for a reason and I am not saying you should comment them out, just hoping this might help pinpoint the error.

diff --git a/BUILD b/BUILD
index c25a0c6..b426456 100644
--- a/BUILD
+++ b/BUILD
@@ -1,18 +1,18 @@
-load("@buildifier_prebuilt//:rules.bzl", "buildifier")
+# load("@buildifier_prebuilt//:rules.bzl", "buildifier")
 
 package(default_visibility = ["//visibility:private"])
 
-buildifier(
-    name = "buildifier.check",
-    lint_mode = "warn",
-    mode = "diff",
-)
+# buildifier(
+#     name = "buildifier.check",
+#     lint_mode = "warn",
+#     mode = "diff",
+# )
 
-buildifier(
-    name = "buildifier",
-    lint_mode = "fix",
-    mode = "fix",
-)
+# buildifier(
+#     name = "buildifier",
+#     lint_mode = "fix",
+#     mode = "fix",
+# )
 
 licenses(["notice"])  # MIT
 

0.5.0 release failed

I see a 0.5.0 tag, and some comments on other bugs that look like it was intended to be a release. The recent changes to publish a release artefact look to have failed, or at least didn't work for 0.5.0; there are no 'notes' anywhere; and github thinks the most recent release is 0.4.0.

Error with rules_python-0.13.0

Upgrading to rules_python-0.13.0 results in

external/mypy_integration/mypy/BUILD:3:10: no such package '@mypy_integration_pip_deps_mypy//': The repository '@mypy_integration_pip_deps_mypy' could not be resolved: Repository '@mypy_integration_pip_deps_mypy' is not defined and referenced by '@mypy_integration//mypy:mypy'

which is likely due to bazelbuild/rules_python#807 which suggest to change to pip_parse instead of pip_install.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

bazel
examples/WORKSPACE
  • rules_python %7Bversion%7D
  • bazel_skylib 1.7.1
repositories/repositories.bzl
  • rules_python 0.27.1
  • bazel_skylib 1.7.1
bazel-module
MODULE.bazel
examples/MODULE.bazel
  • rules_java 7.6.5
  • rules_python 0.27.1
bazelisk
.bazelversion
  • bazel 6.5.0
github-actions
.github/workflows/continuous-integration.yml
  • actions/checkout v4
  • p0deje/setup-bazel 0.8.5
  • actions/checkout v4
.github/workflows/release.yml
  • actions/checkout v4
  • actions/cache v4
  • softprops/action-gh-release v2
pip_requirements
examples/third_party/requirements.txt
  • parse ==1.12.1
examples/tools/typing/mypy_requirements.txt
  • mypy ==0.750
  • mypy-extensions ==0.4.4
  • typed-ast ==1.4.3
  • typing-extensions ==4.12.2
third_party/requirements.txt
  • mypy ==0.790
  • mypy-extensions ==0.4.4
  • typed-ast ==1.4.3
  • typing-extensions ==4.12.2

  • Check this box to trigger a request for Renovate to run again on this repository

main_mypy.runfiles is not a valid Python package name

This seems like it could be similar to #15 but I'm opening a new issue because it has nothing to do with protobuf. I have a plain Python application with no external dependencies that only uses rules_python and Bzlmod.

I am consistently getting the error

main_mypy.runfiles is not a valid Python package name

when running bazel test.

My source file is

main.py

def sayHello():
    return f"Hello, world!"

if __name__ == "__main__":
    print(sayHello())

and my BUILD.bazel contains:

py_binary(
    name = "main",
    srcs = ["main.py"],
    visibility = ["//:__subpackages__"],
)

mypy_test(
    name = "main_mypy",
    deps = [
        ":main",
    ],
)

Lines from relevant files:

MODULE.bazel

bazel_dep(
    name = "rules_python",
    version = "0.31.0",
)
bazel_dep(
    name = "rules_python_gazelle_plugin",
    version = "0.31.0",
)

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    configure_coverage_tool = True,
    python_version = "3.12",
)

bazel_dep(
    name = "mypy_integration",
    version = "0.0.0",
)

git_override(
    module_name = "mypy_integration",
    remote = "https://github.com/bazel-contrib/bazel-mypy-integration",
    commit = "c289c0cd30e53b5cc467b601295834dbc400f4b2",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "mypy_pip",
    python_version = "3.12",
    requirements_lock = "//third_party/python/mypy:requirements_lock.txt",
)
use_repo(pip, "mypy_pip")

.bazelrc

build --@mypy_integration//:mypy=//third_party/python/mypy:mypy

third_party/python/mypy/BUILD.bazel

load("@mypy_pip//:requirements.bzl", "requirement")
load("@rules_python//python:pip.bzl", "compile_pip_requirements")

compile_pip_requirements(
    name = "requirements",
    src = "requirements.txt",
    requirements_txt = "requirements_lock.txt",
)

alias(
    name = "mypy",
    actual = requirement("mypy"),
    visibility = ["//visibility:public"],
)

Support for multiple mypy.ini files

Hi, I work on a large monorepo that has multiple projects owned by different people. I'm migrating these projects to bazel and currently they each have their own mypy.ini file.

From reading the setup instructions it seems that they are meant to have a single configuration file for the whole repo, is there a way around this?

mypy 0.780 source file found twice under different module names error

This bazel plugin works well with mypy 0.770 but after upgrading to 0.780 it complains with
"source file found twice under different module names error". mypy 0.780 works fine outside of bazel.

There is a related issue on mypy: python/mypy#8944. In my case, directories on mypy_path do not contain __init__.py files.

  • I am setting the mypy_path within mypy.ini. Even if I disable that, the problem persists.
  • Bazel tests pass if I disable the bazel mypy checks (so I think the bazel setup is - mostly - correct).
  • The template adds $(pwd) to the front of MYPYPATH. I tried playing around with that but still hitting the same issue.

So, I am not very sure where the problem comes from.

EDIT: usually I am setting PYTHONPATH in .bazelrc and mypy_path in mypy.ini but even if I add imports = ["<path e.g. ..>"] in all BUILD files the above steps still apply.

Error: "The typed_ast package is not installed"

Trying to integrate bazel-mypy-integration on a project using Python 3.8.6, I get the error "The typed_ast package is not installed". As this is a transitive dependency, I considered that I may have a Python binary mismatch of some kind, but I can't find any indication of that.

This is using bazel-mypy-integration-0.1.0 and rules-python-0.1.0.

I'm excited to try bazel-mypy-integration out - thank you for all the effort! Here's the build error:

INFO: Invocation ID: a0d1c3a2-24ca-4731-aaeb-d0fe41c60b45
DEBUG: /private/var/tmp/_bazel_dan/cebcfff554bef27d2846af867a1d4809/external/rules_python/python/legacy_pip_import/pip.bzl:143:10: DEPRECATED: the pip_import rule has been replaced with pip_install, please see rules_python 0.1 release notes
INFO: Analyzed target //processors/context/cli:cli (1 packages loaded, 2 targets configured).
INFO: Found 1 target...
ERROR: /Users/dan/src/10x/data-fixer/datafixer/processors/context/cli/BUILD.bazel:4:10: Type-checking //processors/context/cli:cli failed (Exit 1): cli_mypy_exe failed: error executing command bazel-out/darwin-fastbuild/bin/processors/context/cli/cli_mypy_exe

Use --sandbox_debug to see verbose messages from the sandbox cli_mypy_exe failed: error executing command bazel-out/darwin-fastbuild/bin/processors/context/cli/cli_mypy_exe

Use --sandbox_debug to see verbose messages from the sandbox
The typed_ast package is not installed.
You can install it with `python3 -m pip install typed-ast`.
Aspect @mypy_integration//:mypy.bzl%mypy_aspect of //processors/context/cli:cli failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.823s, Critical Path: 0.69s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Is this project still alive?

Given there was no activity here for almost a year and my attempt of contribution received no feedback I am wondering if this project is still alive.

I am creating this issue in the hope this has more visibility than creating a PR.

@mypy_integration_pip_deps

external/mypy_integration/mypy/BUILD:3:10: no such package '@mypy_integration_pip_deps_mypy//': The repository '@mypy_integration_pip_deps_mypy' could not be resolved: Repository '@mypy_integration_pip_deps_mypy' is not defined and referenced by '@mypy_integration//mypy:mypy'

I've correctly copied the bazel WORKSPACE snipped, but can't fix this issue when invoking Python builds.

This is my Python config

#### Build Rules for Python
http_archive(
    name = "rules_python",
    sha256 = "a644da969b6824cc87f8fe7b18101a8a6c57da5db39caa6566ec6109f37d2141",
    strip_prefix = "rules_python-0.20.0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.20.0/rules_python-0.20.0.tar.gz",
)


load("@rules_python//python:repositories.bzl", "py_repositories")

py_repositories()

load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
    name = "python3_10",
    # Available versions are listed in @rules_python//python:versions.bzl.
    # We recommend using the same version your team is already standardized on.
    python_version = "3.10",
)

load("@python3_10//:defs.bzl", "interpreter")


load("@rules_python//python:pip.bzl", "install")

pip_parse(
    name = "pip",
    python_interpreter_target = interpreter,
    requirements_lock = "//third_party/python:requirements.txt",
)

load("@pip//:requirements.bzl", "install_deps")

install_deps()
####### 

I'm also using the aspect integration in .bazelrc:

build --aspects @mypy_integration//:mypy.bzl%mypy_aspect
build --output_groups=+mypy

mypy can't find rule_python requirements

Hi, I recently meet a problem when using bazel-mypy-integration and I wonder if you have a way to solve it.
I use rule_python to control pip dependencies, but when running mypy check like

bazel build  --aspects @mypy_integration//:mypy.bzl%mypy_aspect --output_groups=mypy //test:test_mypy_with_numpy

with the BUILD file

load("@py_deps//:requirements.bzl", "requirement")

py_library(
    name = 'test_mypy_with_numpy',
    srcs = [test_mypy_with_numpy.py'],
    deps = [
        requirement('numpy'),
    ]

I found numpy related code wasn't checked. But when I use pip install numpy, it will be checked.
So is there any way to let mypy_integration use requirement from rule_python ?

exclude setting in mypy.ini does not work

Even after trying:

[mypy]
exclude = .* # exclude everything

I cannot get the aspect to exclude files when building. I'm certain my mypy.ini is being picked up correctly as other settings do work

Failing to resolve source code imports

First, thanks for working on this integration. I'm excited to incorporate it into our Python 3 code base.

With some of my initial tests I'm seeing numerous errors of the type Cannot find implementation or library stub for module named .... I suspect the issue is our use of the imports attribute on a shared py_library rule.

Our code is structured such that we have a src/python3 directory in our repository which our imports are relative to.

The majority of our packages live under a subdirectory called uc so most of our imports start with uc.. We have a py_library rule that everything depends on directly or transitively that has imports = [".."]. This works well for us with Bazel so far but I'm wondering if it's what is causing the issue.

I noticed that you had #8 which looks at imports for py_test and py_binary so I'm wondering if it's the combination of py_library and .. that isn't handled.

Any ideas? Thanks!

File visibility failures in MyPy if "--build_python_zip" is active while running 'bazel test' on 'mypy_test' target

Hi Jono,

I have reproducible repo to illustrate the issue that I am having, the mypy throws error when I am using combination of:
build --build_python_zip and mypy_test.

git clone https://github.com/kororo/proto-bazel
cd proto-bazel

➜  proto-bazel git:(master) bazel test //examples/foobar/src:foobar_mypy_test
INFO: Analyzed target //examples/foobar/src:foobar_mypy_test (0 packages loaded, 0 targets configured).
INFO: Found 1 test target...
FAIL: //examples/foobar/src:foobar_mypy_test (see /private/var/tmp/_bazel_wizz/7d4571e5262c107c98e149dc3e1a3e95/execroot/proto-bazel/bazel-out/darwin-fastbuild/testlogs/examples/foobar/src/foobar_mypy_test/test.log)
INFO: From Testing //examples/foobar/src:foobar_mypy_test:
==================== Test output for //examples/foobar/src:foobar_mypy_test:
usage: mypy [-h] [-v] [-V] [more options; see below]
            [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: Cannot find config file 'external/mypy_integration_config/mypy.ini'
================================================================================
Target //examples/foobar/src:foobar_mypy_test up-to-date:
  bazel-bin/examples/foobar/src/foobar_mypy_test
INFO: Elapsed time: 4.760s, Critical Path: 4.60s
INFO: 2 processes: 2 darwin-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//examples/foobar/src:foobar_mypy_test                                   FAILED in 4.2s
  /private/var/tmp/_bazel_wizz/7d4571e5262c107c98e149dc3e1a3e95/execroot/proto-bazel/bazel-out/darwin-fastbuild/testlogs/examples/foobar/src/foobar_mypy_test/test.log

INFO: Build completed, 1 test FAILED, 2 total actions

If I commented out the build --build_python_zip in .bazelrc, it works well. Worth to mention, mypy integration with aspect is working as expected. I suspect there is something missing in the link with the export_files.

no such package '@mypy_integration_pip_deps//pypi__typed_ast' when using mypy_version >= 0.900

Hi, just reporting this as FYI. It's not really causing me any trouble, I'm happy on mypy version 0.812, just thought I'd start a thread in case this is impacting others.

If I change my mypy_version.txt file to contain:

mypy==0.900

Then I get this error when I try to build a mypy_test

ERROR: /private/var/tmp/_bazel_mark/c85e0696457b3a2f404f0721275052b4/external/mypy_integration/mypy/BUILD:3:10: no such package '@mypy_integration_pip_deps//pypi__typed_ast': BUILD file not found in directory 'pypi__typed_ast' of external repository @mypy_integration_pip_deps. Add a BUILD file to a directory to mark it as a package. and referenced by '@mypy_integration//mypy:mypy'

I wonder if this is something to do with the release notes of mypy 0.900: http://mypy-lang.org/news.html

The next mypy release will no longer bundle stubs for third-party libraries. Read this blog post if you are interested in what this means to mypy users and why we are doing this.

Target using pytest fail with "Untyped decorator makes function untyped"

I just implemented this integration and for the most part things went smooth. But now I stumbled into a case that I cannot seem to solve, see the example below.

Running below in a virtual env where pytest is installed works, however running it from the base env (without pytest) fails. All other pip depencies do get loaded correctly via bazel, only a few pytest decorators give errors. Running mypy test_foo.py also passes in the virutalenv (using the same mypy.ini file).

I'm relatively new to bazel and mypy, so not sure where to start digging. Any thoughts?

WORKSPACE

workspace(name = "myproject")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

#############################################################################
# Python Rules
#############################################################################
http_archive(
    name = "rules_python",
    sha256 = "b6d46438523a3ec0f3cead544190ee13223a52f6a6765a29eae7b7cc24cc83a0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.1.0/rules_python-0.1.0.tar.gz",
)
load("@rules_python//python:repositories.bzl", "py_repositories")
py_repositories()
load("@rules_python//python:pip.bzl", "pip_install")

# Create a central repo that knows about the dependencies needed for
# requirements.txt.
pip_install(
    name = "eng_pip_deps",
    requirements = "//:requirements.txt",
)

#############################################################################
# Mypy aspect plugin
#############################################################################
mypy_integration_version = "0.1.0"  # latest @ November 15th 2020
http_archive(
    name = "mypy_integration",
    sha256 = "511ca642294129b7abebf6afd48aa63e7d91de3ec5fa0689d60d1dc6a94a7d1a",
    strip_prefix = "bazel-mypy-integration-{version}".format(version = mypy_integration_version),
    url = "https://github.com/thundergolfer/bazel-mypy-integration/archive/{version}.tar.gz".format(
        version = mypy_integration_version,
    ),
)
load(
    "@mypy_integration//repositories:repositories.bzl",
    mypy_integration_repositories = "repositories",
)
mypy_integration_repositories()
load("@mypy_integration//:config.bzl", "mypy_configuration")
mypy_configuration(mypy_config_file = "//:mypy.ini")
load("@mypy_integration//repositories:deps.bzl", mypy_integration_deps = "deps")
mypy_integration_deps(mypy_requirements_file = "//:mypy_version.txt")

BUILD

load("@eng_pip_deps//:requirements.bzl", "requirement")
load("@rules_python//python:defs.bzl", "py_test")

py_test(
    name = "test_foo",
    srcs = ["test_foo.py"],
    python_version = "PY3",
    deps = [
        requirement("pytest"),
        requirement("pytest-asyncio"),
    ],
)

test_foo.py

import pytest

@pytest.mark.asyncio
async def test_foo_noop() -> None:
    pass

result

$ bazel build //:test_foo                                                                                                                                                                       [16:49:26]
(16:49:33) INFO: Current date is 2021-01-19
(16:49:33) INFO: Analyzed target //:test_foo (0 packages loaded, 0 targets configured).
(16:49:33) INFO: Found 1 target...
(16:49:34) ERROR: /home/myproject/BUILD:4:8: Couldn't build file test_foo_dummy_out: Type-checking //:test_foo failed (Exit 1): test_foo_mypy_exe failed: error executing command bazel-out/k8-fastbuild/bin/test_foo_mypy_exe

Use --sandbox_debug to see verbose messages from the sandbox test_foo_mypy_exe failed: error executing command bazel-out/k8-fastbuild/bin/test_foo_mypy_exe

Use --sandbox_debug to see verbose messages from the sandbox
test_foo.py:3: error: Untyped decorator makes function "test_foo_noop" untyped
    @pytest.mark.asyncio
     ^
Found 1 error in 1 file (checked 1 source file)
Aspect @mypy_integration//:mypy.bzl%mypy_aspect of //:test_foo failed to build
Use --verbose_failures to see the command lines of failed build steps.
(16:49:34) INFO: Elapsed time: 1.803s, Critical Path: 1.61s
(16:49:34) INFO: 2 processes: 2 internal.
(16:49:34) FAILED: Build did NOT complete successfully
FAIL

MyPy dependency version issues when changing `python3` interpreter version

Currently the MyPy py_binary this integration ships gets its dependencies using rules_python's pip_import: https://github.com/thundergolfer/bazel-mypy-integration/blob/master/repositories/py_repositories.bzl#L22

It's configured to currently just look for python_interpreter = "python3", and now I'm not sure this would work when the underlying interpreter version changes, for example by going from 3.6 to 3.7.

Looking at the resolved typed_ast depending inside my Bazel WORKSPACE's outputBase I see:

cat bazel-data-science/external/mypy_integration_pip_deps_pypi__typed_ast_1_4_1/typed_ast-1.4.1.dist-info/WHEEL
Wheel-Version: 1.0
Generator: bdist_wheel (0.33.6)
Root-Is-Purelib: false
Tag: cp36-cp36m-macosx_10_9_x86_64

Which is clearly 3.6 versioned. If the python3 interpreter is upgraded to 3.7 what happens? I think the repository_rule that downloaded the dependencies wouldn't invalidate (because python_interpreter = or anything else hasn't changed), and the dependencies would be out-of-date.

Bazel docs for repository_rules mention the configure attr which indicates that a repository_rule "inspects the system for configuration purpose". Should this be set on the repository_rule that does deps downloading?

My current idea is that the python_interpreter attr of pip_import should be configurable for the user so that they can specify themselves which interpreter is being used, and have it match how they install dependencies for their WORKSPACE internal code.

MyPy Plugins not supported

Description.

Using mypy plugin doesn't seem possible.

cat mypy.ini
[mypy]
ignore_missing_imports = True
plugins = pydantic.mypy
external/mypy_integration_config/mypy.ini:4: error: Error importing plugin 'pydantic.mypy': No module named 'pydantic'                            
Found 1 error in 1 file (checked 1 source file)    

Solution.

Make _mypy_cli configurable so users can customise accordingly.

Support including .pyi alongside .py files (eg. foo/bar.py and foo/bar.pyi)

In MyPy if you have a .py and .pyi file in the same folder with the same name then MyPy says that the .pyi should take precedence and the .py is ignored completely during type-checking.

In a branch, https://github.com/thundergolfer/bazel-mypy-integration/tree/jonathon/supporting-pyi-files, I was passing in both to the --cache-map and it was throwing out the error:

mypy_stubs/parse_name.pyi: error: Duplicate module named 'mypy_stubs.parse_name' (also at 'mypy_stubs/parse_name.py')

If could in my Starlark code check for this case and then ignore the .py, but for now I think I just won't support those kinds of .pyi files.


Open issue about this: python/mypy#3505

Transition rules from rules_python does not run mypy

🐞 bug report

The following code in _mypy_aspect_impl does not include the _transition_py_test and _transition_py_binary rules from rules_python (code). As a result, mypy did not run for targets that use py_test from @python//<version>:def.bzl

def _mypy_aspect_impl(_, ctx):
if (ctx.rule.kind not in ["py_binary", "py_library", "py_test", "mypy_test"] or
ctx.label.workspace_root.startswith("external")):
return []

As a workaround, I created applied a patch in my WORKSPACE and it seem to work:

 def _mypy_aspect_impl(_, ctx): 
     if (ctx.rule.kind not in ["py_binary", "py_library", "py_test", "mypy_test", "_transition_py_test", "_transition_py_binary"] or 
         ctx.label.workspace_root.startswith("external")): 
         return []

Versions

How to reproduce

Demo workspace (clone of rules_python examples)

Run:

bazel test //tests:\*    --aspects @mypy_integration//:mypy.bzl%mypy_aspect --output_groups=mypy

It should produce an error:

tests/version.py:18: error: Incompatible return value type (got "str", expected "int")

The error goes away once patch line is removed in WORKSPACE

Remove need for --output_groups=+mypy

Hopefully Aspects are allowed to return DefaultInfo, and I can do away with having the mypy output group which must be requested explicitly with --output_groups=+mypy.

mypy-0.981 errors

Using

mypy==0.981

along with

bazel-mypy-integration-0.4.0

results in an (seemingly incorrect) error

Traceback (most recent call last):
  File "/home/XXX/.cache/bazel/_bazel_janickm/c58fb2127d9f20dca76a2dc0b0361e0c/sandbox/linux-sandbox/47/execroot/av-depth/bazel-out/host/bin/external/mypy_integration/mypy/mypy.runfiles/mypy_integration/mypy/main.py", line 7, in <module>
    main(None, sys.stdout, sys.stderr)
TypeError: main() takes no positional arguments

From what I can tell the called main() takes three positional arguments. Errors don't show up for lower mypy versions (e.g., mypy==0.971).

Requesting a new release

How to integrate this tool into other projects changed much:

  • Compatible to modern rules_python
  • Users inject mypy
  • Basic support for bzlmod

It would be great to depend on all those updates through an official release.

Cannot find third-party libraries used via `requirement`

I am trying to stand up a monorepo that runs mypy as part of the build phase. I have been installing third-party libraries via pip_parse and requirement in rules_python. However, when I tried out this integration, I get:

error: Cannot find implementation or library stub for module named "numpy"

for all of the third-party dependencies that I depend on via requirement. This seems like the most basic use case, so I figure I must be doing something wrong, but I am not seeing anything in the README about third party imports. Is there a way to make this integration aware of my requirements-based imports?

Thanks,
Phil

'mypy_stubs' rule not working. Reproduction within.

Reproduction in https://github.com/thundergolfer/bazel-python-mypy-protobuf/tree/master/bazel_mypy_integration_issue_36

I'm getting:

bazel test //... --test_output=all --sandbox_debug
INFO: Analyzed 5 targets (0 packages loaded, 0 targets configured).
INFO: Found 4 targets and 1 test target...
FAIL: //tools/thing:mypy_test_foo (see /private/var/tmp/_bazel_jonathon/fafac219b6b0ff0ed453a7d553f169f1/execroot/bazel_mypy_integration_issue_36/bazel-out/darwin-fastbuild/testlogs/tools/thing/mypy_test_foo/test.log)
INFO: From Testing //tools/thing:mypy_test_foo:
==================== Test output for //tools/thing:mypy_test_foo:
tools/thing/proto/bling_api_pb2.pyi:26: error: syntax error in type comment  [syntax]

In the internal code that hit this issue, the error is:

Use --sandbox_debug to see verbose messages from the sandbox
tools/foo/proto/foo_api_pb2.py: error: Duplicate module named 'tools.foo.proto.foo_api_pb2' (also at 'tools/foo/proto/foo_api_pb2.pyi')
Found 1 error in 1 file (errors prevented further checking)
INFO: Elapsed time: 2.294s, Critical Path: 1.98s
INFO: 5 processes: 5 internal.
FAILED: Build did NOT complete successfully

In that internal error, the integration should be letting the .pyi file 'win', as is stated in the documentation: https://mypy.readthedocs.io/en/stable/stubs.html#:~:text=Mypy%20uses%20stub%20files%20stored,to%20type%20check%20your%20code.

Incorrect sha256 in README.md

This is a great project. Thanks for putting this together.

I believe that the sha256 value for 0.0.7 which is listed in the README.md is incorrect. I believe that the actual value is bf7ecd386740328f96c343dca095a63b93df7f86f8d3e1e2e6ff46e400880077. The corrected

http_archive(
    name = "mypy_integration",
    sha256 = "bf7ecd386740328f96c343dca095a63b93df7f86f8d3e1e2e6ff46e400880077",  # for 0.0.7
    strip_prefix = "bazel-mypy-integration-{version}".format(version = mypy_integration_version),
    url = "https://github.com/thundergolfer/bazel-mypy-integration/archive/{version}.zip".format(
        version = mypy_integration_version,
    ),
)

Cache is not being written or used

The integration attempts to write out cache files using mypy's undocumented Bazel integration (--bazel, --package-root, and --cache-map) but these files are never written (due to sandboxing?). This can be reproduced by via the tests in this repo:

$ ./test.sh
$ find $(bazel info output_base) -name '*.data.json'
# returns nothing

If you temporarily disable sandboxing via --spawn-strategy=standalone, i.e.:

 diff --git a/test/shell/test_mypy.sh b/test/shell/test_mypy.sh
 index db1dcc0..a3bbb6c 100644
 --- a/test/shell/test_mypy.sh
 +++ b/test/shell/test_mypy.sh
 @@ -5,7 +5,7 @@ dir=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
  runner=$(get_test_runner "${1:-local}")

  test_ok_on_valid_imported_mypy_typings() {
 -  action_should_succeed build --verbose_failures --aspects //:mypy.bzl%mypy_aspect --output_groups=mypy //test:correct_imported_mypy_typings
 +  action_should_succeed build --spawn_strategy=standalone --verbose_failures --aspects //:mypy.bzl%mypy_aspect --output_groups=mypy //test:correct_imported_mypy_typings
  }

  test_ok_on_valid_imported_mypy_test() {

and then run the tests again:

$ bazel clean; ./test.sh
$ find $(bazel info output_base) -name '*.data.json'
/home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/execroot/bazel_mypy_integration/bazel-out/k8-fastbuild/bin/test/correct_generated.py.data.json
$ find $(bazel info output_base) -name '*.meta.json'
/home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/execroot/bazel_mypy_integration/bazel-out/k8-fastbuild/bin/test/correct_generated.py.meta.json

then our the cache files for our transitive dependencies are written to the output directory. However, it also writes our the cache files for the root dependency directory to the workspace:

$ git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        test/__init__.data.json
        test/__init__.meta.json
        test/correct_imported_typings.py.data.json
        test/correct_imported_typings.py.meta.json

If you enable verbose mypy logging (via setting the DEBUG variable in mypy.bzl to True and changing the tests so output to stdout and stderr), then you can see mypy's logs:

OG:  Cached module os.path has changed interface
LOG:  Writing sys /home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/sandbox/processwrapper-sandbox/66/execroot/bazel_mypy_integration/bazel-out/host/bin/mypy/mypy.r
unfiles/mypy_integration_pip_deps/pypi__mypy/mypy/typeshed/stdlib/3/sys.pyi sys.meta.json sys.data.json
LOG:  Cached module sys has changed interface
LOG:  Writing builtins /home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/sandbox/processwrapper-sandbox/66/execroot/bazel_mypy_integration/bazel-out/host/bin/mypy/m
ypy.runfiles/mypy_integration_pip_deps/pypi__mypy/mypy/typeshed/stdlib/2and3/builtins.pyi builtins.meta.json builtins.data.json
LOG:  Cached module builtins has changed interface
LOG:  Processing SCC singleton (importlib) as inherently stale with stale deps (builtins importlib.abc types typing)
LOG:  Writing importlib /home/vscode/.cache/bazel/_bazel_vscode/8dbf04a8279d05eb224863e2e901551c/sandbox/processwrapper-sandbox/66/execroot/bazel_mypy_integration/bazel-out/host/bin/mypy/
mypy.runfiles/mypy_integration_pip_deps/pypi__mypy/mypy/typeshed/stdlib/3/importlib/__init__.pyi importlib/__init__.meta.json importlib/__init__.data.json
LOG:  Cached module importlib has changed interface
LOG:  Processing SCC singleton (test) as inherently stale with stale deps (builtins)
LOG:  Writing test ./test/__init__.py test/__init__.meta.json test/__init__.data.json
LOG:  Cached module test has changed interface
LOG:  Processing SCC singleton (test.correct_generated) as inherently stale with stale deps (builtins typing)
LOG:  Writing test.correct_generated bazel-out/k8-fastbuild/bin/test/correct_generated.py bazel-out/k8-fastbuild/bin/test/correct_generated.py.meta.json bazel-out/k8-fastbuild/bin/test/co
rrect_generated.py.data.json
LOG:  Cached module test.correct_generated has changed interface
LOG:  Processing SCC singleton (test.correct_imported_typings) as inherently stale with stale deps (builtins test.correct_generated)
LOG:  Writing test.correct_imported_typings test/correct_imported_typings.py test/correct_imported_typings.py.meta.json test/correct_imported_typings.py.data.json
LOG:  Cached module test.correct_imported_typings has changed interface
LOG:  No fresh SCCs left in queue

(You'll note in the logs above that cache files appear to be written to 3 separate locations (sandbox/processwrapper-sandbox/66/execroot, bazel-out/k8-fastbuild/bin/test/, and the workspace).

Add a new release

I would like to use the changes added in this pull request so that I can use a newer version of mypy: #45

Could you please add a new release?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.