serge-sans-paille / beniget Goto Github PK

View Code? Open in Web Editor NEW

69.0 69.0 20.0 110 KB

Extract semantic information about static Python code

License: BSD 3-Clause "New" or "Revised" License

Python 100.00%

beniget's People

Contributors

Stargazers

Watchers

Forkers

paugier pombredanne cfandy axelminn marimeireles reuben sylvaincorlay wzrumich hroncok huangxiaodui tristanlatr arpitjain799 ibm-z-oss-oda alvistack jarvx meta-introspector sysfce2 wolfi-chainguard-demo

beniget's Issues

What about dropping python2 ?

Keeping compat with python is becoming harder and herder... how long do you want to keep supporting python2 ?

Scoping rule not correctly honored for type annotation

There is no ambiguity in the following code:

class Attr:...
class Cls:
    def Attr(self) -> Attr:...

but beniget issues a warning about -> Attr being potentially unbound at runtime.

Silent mode

beniget prints some warning for unbound identifiers, could it be possible to have a silent mode that does not print those warnings?

I can come up with a PR.

Bug in class scope + comprehension

The following code seem to confuse beniget, it's not a regression...

class Cls:
    foo = b'1',
    [_ for _ in foo]

Produces warning:

W: unbound identifier 'foo' at ./test.py:4:16

Class decorator not in users

import gast as ast
import beniget

code = """
def nice(obj):
    return obj

@nice
class My(object):
    pass

@nice
def foo():
    pass

@nice
def bar():
    pass
"""

module = ast.parse(code)
duc = beniget.DefUseChains()
duc.visit(module)

decor = module.body[0]

print(duc.chains[decor])

users = duc.chains[decor].users()
print(users)
assert len(users) == 3

gives

nice -> (nice -> (), nice -> ())
[<gast.gast.Name object at 0x7f9833ea6160> -> (), <gast.gast.Name object at 0x7f9833ea6e10> -> ()]
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
~/tmp/tmp_beniget.py in <module>
     33 print(users)
     34 
---> 35 assert len(users) == 3

Make it work with the standard library

I believe very little effort is needed to make it work on python3 with the ast module.
Of course, this would only work on python3 for python3 ast. But compatibility with python2 would still be possible by installing gast.

What do you think ?

Warlus target should be stored in first non comprehension scope

While looking at https://github.com/gvanrossum/gvanrossum.github.io/blob/main/formal/build.py#L98-L108, I've noticed that beniget does not correctly understands the warlus operator.

Regression in Lambda functions

This new test was passing in version 0.4.1 and now shows nothing

    def test_simple_lambda(self):
        code = "lambda y: True"
        node = ast.parse(code)
        c = beniget.DefUseChains()
        c.visit(node)
        self.assertEqual(c.dump_chains(node), [])
        self.assertEqual(c.dump_chains(node.body[0].value), ['y -> ()'])

0.4.1: please rename tests/.py to tests/tests_.py

Please rename tests/.py to tests/tests_.py because pytest scans for units only tests_*.py files.
Without that pytest is not able to find any units.

Read Code from Python File

Hi,

I am working on a research project, and I am using beniget for part of my research problem. An issue I found was about reading code from a python file. When I am working with several files or apply changes to one file, I should prepare my code such that beniget can process it. I wondered if it is possible to add a feature so beniget can read code from a file.

Simple DefUse Analysis

Hi, I have been trying to perform a simple DefUse analysis on my code. Essentially, I just want to know for a given line of code, which other lines of code should be executed before it.

code = open("my_code.py", "r").read()
module = ast.parse(code)
duc = beniget.DefUseChains()
duc.visit(module)

From the example in the code, I know the duc should contain the required information in the duc.chains. But I am a little confused about the right way to access the information. For example, I know each line of code would be presented as a node in module and I can access it by doing module.body[line #], how can I look it up in the duc.chains? I have tried different ways and unfortunately no luck.

Bug: a name that is being bound cannot use itself, even if it’s declared in enclosing scopes.

This name error is uncaught by beniget.

def outer():
    x = 1
    def middle():
        x = 1
        class mytype(str):
            x = x+1 # <- this triggers NameError: name 'x' is not defined
    return middle
outer()()

Incompatibility extended AST with comment lines (from Transonic)

Usually, there is no problem to use the extended AST with comment lines (defined in Transonic here https://bitbucket.org/fluiddyn/transonic/src/default/transonic/analyses/extast.py) in Beniget.

However, here is one case where there is a problem:

import beniget
from transonic.analyses.extast import parse

code = """
t = (
    "toto",
    # comment
    "titi",
)
"""

mod = parse(code)
duc = beniget.DefUseChains()
duc.visit(mod)

It gives:

Traceback (most recent call last):
  File "bug.py", line 16, in <module>
    duc.visit(mod)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/home/pierre/Dev/beniget/beniget/beniget.py", line 324, in visit_Module
    self.process_body(node.body)
  File "/home/pierre/Dev/beniget/beniget/beniget.py", line 275, in process_body
    self.visit(stmt)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/home/pierre/Dev/beniget/beniget/beniget.py", line 428, in visit_Assign
    dvalue = self.visit(node.value)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/home/pierre/Dev/beniget/beniget/beniget.py", line 844, in visit_List
    self.visit(elt).add_user(dnode)
AttributeError: 'NoneType' object has no attribute 'add_user'

i don't know what would be the correct way to fix that.

i could monkey patch beniget to add a visit for the CommentLine node. But it seems hacky.
The extended AST with comment lines could be defined deeper (with a cleaner implementation) and Beniget could handle the corresponding node.
Or a quick fix in beniget...

Add support for Type Parameter Syntax (python 3.12)

https://peps.python.org/pep-0695/

AnnAssign links the target as beeing the def of dannotation

The little it of code seems strange to https://github.com/serge-sans-paille/beniget/blob/master/beniget/beniget.py#L769-L772

I think the annotation should not be added to the uses of the target of the assignment. It's like saying "the assign target is where it's annotation is defined", which doesn't make any sens to me.

~~That way we could cleanup the extra lambda/None values in the _defered_annotations stack as well.~~ (we need it for #72

Tell me what you think, @serge-sans-paille.

rpmbuild the package failed

hi
i wan't to build the package by rpmbuild. When i write a spec file and run the "rpmbuild -bb python-beniget.spec", the result is failed. The error message is "Dependency tokens must begin with alpha-numeric, '_' or '/': ~= 0.5".

I located the problem and confirmed that the cause was the requirement.txt file. If I change "gast ~= 0.5.0" to "gast>=0.5.0" in requirement.txt, the problem is solved.

Please check whether the modification is reasonable. Thank you

Temporary nodes in assignment lhs are not visited

Let's take this example: NameError().name = 't'

Here, the ast.Call instance uses won't be part of the chains at all, even if we could argue that it's still a user of NameError, the same manner NameError won't be added to the uses of the builtins NameError symbol. This is why I think we should unconditionally visit this part.

Originally posted by @tristanlatr in #68 (comment)

PEP 563 support

Hello,

I’m working on adding support for postponed evaluation of annotations.

I figured I’d create an issue for this feature first.

Using beniget to parse stubs produce errors

Here are the warnings that beniget reports (with PEP563 enabled) when parsing the typeshed builtins module and all its dependencies.

W: unbound identifier '_ScandirIterator' at os:734:74
W: unbound identifier 'TraceFunction' at _typeshed:306:59
W: unbound identifier 'Callable' at typing:159:25
W: unbound identifier '_ClassInfo' at builtins:1391:59
W: unbound identifier '_ClassInfo' at builtins:1393:41
W: unbound identifier 'AbstractContextManager' at contextlib:38:33
W: unbound identifier 'ellipsis' at types:606:19
W: unbound identifier 'CDLL' at ctypes:13:31
W: unbound identifier '_CData' at ctypes:14:27
W: unbound identifier '_FuncPointer' at ctypes:88:49
W: unbound identifier 'Array' at ctypes:125:40
W: unbound identifier 'A' at re:167:12
W: unbound identifier 'I' at re:170:17
W: unbound identifier 'L' at re:172:13
W: unbound identifier 'M' at re:174:16
W: unbound identifier 'S' at re:176:13
W: unbound identifier 'X' at re:178:14
W: unbound identifier 'U' at re:180:14
W: unbound identifier 'T' at re:182:15
W: unbound identifier 'Enum' at enum:37:52
W: unbound identifier 'IntFlag' at enum:201:11
W: unbound identifier 'KEEP' at enum:229:48
W: unbound identifier 'Message' at email.message:15:31

Most of them are because cyclic definitions of type aliases, or using a forward reference in a TypeVar bound argument.

We could introduce a new constructor flag that would signify whether we're inside a stub module and change a few behaviour regarding name resolution for a few specific scenarios.

Tell me what you think,

import missing from DefUseChains?

import gast as ast
import beniget

code = "import numpy as np\nnp"

module = ast.parse(code)
du = beniget.DefUseChains()
du.visit(module)

print(du.chains)

assert module.body[0] in du.chains

gives

{<gast.gast.Name object at 0x7f74d1205208>: <gast.gast.Name object at 0x7f74d1205208> -> ()}
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
~/tmp/tmp_beniget.py in <module>
     11 print(du.chains)
     12 
---> 13 assert module.body[0] in du.chains

Why is there no chains for the first Import node?

AttributeError indexing by Ellipsis

This one is funny:

import gast as ast
import beniget
mod = ast.parse("var[...]")
du = beniget.DefUseChains()
du.visit(mod)

gives


Traceback (most recent call last):
  File "toto.py", line 7, in <module>
    du.visit(mod)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/home/pierre/Dev/beniget/beniget/beniget.py", line 329, in visit_Module
    self.process_body(node.body)
  File "/home/pierre/Dev/beniget/beniget/beniget.py", line 280, in process_body
    self.visit(stmt)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/home/pierre/Dev/beniget/beniget/beniget.py", line 657, in visit_Expr
    self.generic_visit(node)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 272, in generic_visit
    self.visit(value)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/home/pierre/Dev/beniget/beniget/beniget.py", line 790, in visit_Subscript
    self.visit(node.slice).add_user(dnode)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/home/pierre/Dev/beniget/beniget/beniget.py", line 739, in visit_Await
    self.visit(node.value).add_user(dnode)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 262, in visit
    return visitor(node)
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 266, in generic_visit
    for field, value in iter_fields(node):
  File "/home/pierre/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 177, in iter_fields
    for field in node._fields:
AttributeError: 'NoneType' object has no attribute '_fields'

Why visit_Await?

Note that there is no problem with var[Ellipsis]

Variable lenght arguments in lambda are ignored

lambda *args: args

The code above generates the following warning:

W: unbound identifier 'args' at ./test.py:1:14

test-suite failures on macOS

first of all: thanks for maintaining this package!

When updating to the latest version in MacPorts, I am seeing now the following failures for the feature/tests that were recently added.

======================================================================
ERROR: test_named_expr_complex (tests.chains.TestDefUseChains)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/tests/chains.py", line 401, in test_named_expr_complex
    self.checkChains(
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/tests/chains.py", line 35, in checkChains
    c.visit(node)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/beniget/beniget.py", line 341, in visit_Module
    self.process_body(node.body)
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/beniget/beniget.py", line 292, in process_body
    self.visit(stmt)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/beniget/beniget.py", line 599, in visit_If
    self.visit(node.test)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 411, in generic_visit
    for field, value in iter_fields(node):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 249, in iter_fields
    for field in node._fields:
AttributeError: 'NoneType' object has no attribute '_fields'

======================================================================
ERROR: test_named_expr_simple (tests.chains.TestDefUseChains)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/tests/chains.py", line 392, in test_named_expr_simple
    self.checkChains(
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/tests/chains.py", line 35, in checkChains
    c.visit(node)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/beniget/beniget.py", line 341, in visit_Module
    self.process_body(node.body)
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/beniget/beniget.py", line 292, in process_body
    self.visit(stmt)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/beniget/beniget.py", line 599, in visit_If
    self.visit(node.test)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 411, in generic_visit
    for field, value in iter_fields(node):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 249, in iter_fields
    for field in node._fields:
AttributeError: 'NoneType' object has no attribute '_fields'

======================================================================
ERROR: test_named_expr_with_rename (tests.chains.TestDefUseChains)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/tests/chains.py", line 411, in test_named_expr_with_rename
    self.checkChains(
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/tests/chains.py", line 35, in checkChains
    c.visit(node)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/beniget/beniget.py", line 341, in visit_Module
    self.process_body(node.body)
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/beniget/beniget.py", line 292, in process_body
    self.visit(stmt)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1/beniget/beniget.py", line 599, in visit_If
    self.visit(node.test)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 407, in visit
    return visitor(node)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 411, in generic_visit
    for field, value in iter_fields(node):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 249, in iter_fields
    for field in node._fields:
AttributeError: 'NoneType' object has no attribute '_fields'

----------------------------------------------------------------------
Ran 145 tests in 0.200s

FAILED (errors=3)
Test failed: <unittest.runner.TextTestResult run=145 errors=3 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=145 errors=3 failures=0>
Command failed:  cd "/opt/local/var/macports/build/_Users_renee_Software_macports-ports_python_py-beniget/py39-beniget/work/beniget-0.4.1" && /opt/local/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 setup.py --no-user-cfg test

Missing 0.4.1 tag

Can you push, please?

AssertionError in the debug sanity checks when parsing idlelib.config

It can be reproduced as follows, but I don't have a minimal reproducer at the moment.

% curl --silent https://raw.githubusercontent.com/python/cpython/3.12/Lib/idlelib/config.py > test.py
% python3 -m beniget test.py 
W: unbound identifier 'line' at test.py:885:8
W: unbound identifier 'crc' at test.py:886:50
[...]
  File "beniget/beniget.py", line 616, in visit_Module
    assert nb_defs == nb_heads + nb_bltns - nb_overloaded_bltns
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

fdef.returns not in chains

import gast as ast
import beniget

mod = ast.parse("""
T = int
def func() -> T:
    return 1
""")

fdef = mod.body[1]
node = fdef.returns

du = beniget.DefUseChains()
du.visit(mod)

du.chains[node]

ud = beniget.UseDefChains(du)

ud.chains[node]

leads to a KeyError...

Isn't it a bug?

Bug in nonlocal

def f():
    nonlocal f
    f = 2
    return f
f()

This code does not generate any warnings, while failing with SyntaxError: no binding for nonlocal 'f' found.

PyPI release

I @serge-sans-paille, just wondering if you plan to release a new version on pypi soon?

Thanks

Bug in starred assignments

This valid python code produces warnings:

curr, *parts = [1,2,3]
while curr:
    print(curr)
    if parts:
        curr, *parts = parts
    else:
        break

Warnings appear twice

W: unbound identifier 'parts' at ./test.py:4:7
W: unbound identifier 'parts' at ./test.py:5:23
W: unbound identifier 'parts' at ./test.py:4:7
W: unbound identifier 'parts' at ./test.py:5:23

Add support for the match statement (python 3.10)

All the bindings are not represented as Name[Store] in the ast (But by MatchAs node etc), so beniget needs special handling for these. It’s more or less the same situation as the except handler not having a store name for the exception. But it this case gast transforms it to actually have a Store name…

see for the specs: https://peps.python.org/pep-0622/

Bug with global keyword

The following code seems to confuse beniget:

def func():
    global GlobalClass
    class GlobalClass(object):...
func()
i = GlobalClass()

Creates warnings:

W: unbound identifier 'GlobalClass' at ./test.py:5:4
W: 'GlobalClass' is defined but not used at ./test.py:3:4

Something is not right with comprehensions

Hello Serge,

I've noticed a stange bug while running beniget on pydoctor's code, here the command I used to reproduce the bug:

python3 -m beniget.beniget ../pydoctor/pydoctor/astbuilder.py 
<frozen runpy>:128: RuntimeWarning: 'beniget.beniget' found in sys.modules after import of package 'beniget', but prior to execution of 'beniget.beniget'; this may result in unpredictable behaviour
W: unbound identifier 'name' at ../pydoctor/pydoctor/astbuilder.py:388:23
W: unbound identifier 'name' at ../pydoctor/pydoctor/astbuilder.py:385:16
[...]

which points to code like:

            names = [
                name
                for name in chain(mod.contents.keys(),
                                  mod._localNameToFullName_map.keys())
                if not name.startswith('_')
                ]

I cannot reproduce it inside the tests, thought :/

I've made sure that no older version of beniget is installed on my site-packages. I don't really understand the python warning... But it seems it could be linked to the project structure, so it's likely unrelated.

Tell me what you think,

serge-sans-paille / beniget Goto Github PK

beniget's People

Contributors

Stargazers

Watchers

Forkers

beniget's Issues

Recommend Projects

Recommend Topics

Recommend Org