func module might be None about pytypes

[Stewori]

Do you know under what circumstances that happens? Do you have a code snippet to reproduce? I just ask to figure out what behavior should be applied then...

[Stewori]

We'll need to find a workaround. Any idea how to find the module that hosts the function in such a case? What about using pytypes.util.getmodule on function.__code__? Or is __code__ also not available or __code__.co_filename not properly set? Would be helpful if you could try this.

[blueyed]

It is not easily reproducible it seems.

Would need to investigate.

I see it with some of the tests from https://github.com/Vimjas/covimerage, when enabling pytypes with the following patch:

diff --git i/tests/conftest.py w/tests/conftest.py
index 5fcd676..636a3c6 100644
--- i/tests/conftest.py
+++ w/tests/conftest.py
@@ -3,8 +3,41 @@
 
 from click.testing import CliRunner
 import pytest
+import pytypes
 
+import covimerage
+import covimerage.cli
 
+pytypes.enable_checking(False)
+
+pytypes.typelogged(covimerage.__init__)
+pytypes.typelogged(covimerage._compat)
+pytypes.typelogged(covimerage.cli)
+pytypes.typelogged(covimerage.coveragepy)
+pytypes.typelogged(covimerage.utils)

Running with pytest -x -l then gives:

FAIL tests/test_cli.py::test_cli

========================================= FAILURES ==========================================
_________________________________________ test_cli __________________________________________

runner = <click.testing.CliRunner object at 0x7ff771f09ef0>
tmpdir = local('/tmp/pytest-of-user/pytest-76/test_cli0')

    def test_cli(runner, tmpdir):
        with tmpdir.as_cwd() as old_dir:
            with pytest.raises(SystemExit) as excinfo:
                cli.write_coverage([os.path.join(
>                   str(old_dir), 'tests/fixtures/conditional_function.profile')])

excinfo    = <ExceptionInfo KeyError tblen=12>
old_dir    = local('…/src/covimerage')
runner     = <click.testing.CliRunner object at 0x7ff771f09ef0>
tmpdir     = local('/tmp/pytest-of-user/pytest-76/test_cli0')

tests/test_cli.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.venv/lib/python3.6/site-packages/click/core.py:722: in __call__
    return self.main(*args, **kwargs)
.venv/lib/python3.6/site-packages/click/core.py:697: in main
    rv = self.invoke(ctx)
.venv/lib/python3.6/site-packages/click/core.py:895: in invoke
    return ctx.invoke(self.callback, **ctx.params)
.venv/lib/python3.6/site-packages/click/core.py:535: in invoke
    return callback(*args, **kwargs)
covimerage/cli.py:35: in write_coverage
    if not m.write_coveragepy_data(data_file=data_file):
covimerage/__init__.py:185: in write_coveragepy_data
    cov_data = self.get_coveragepy_data()
covimerage/__init__.py:180: in get_coveragepy_data
    self._coveragepy_data = self._get_coveragepy_data()
covimerage/__init__.py:137: in _get_coveragepy_data
    data = CoverageData()
../../Vcs/pytypes/pytypes/typechecker.py:798: in checker_tp
    pytypes.log_type(check_args, res, func, slf, prop_getter, parent_class, specs)
../../Vcs/pytypes/pytypes/typelogger.py:81: in log_type
    md = util.getmodule_for_member(func, prop_getter)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

func = <function __init__ at 0x7ff77249c268>, prop_getter = False

    def getmodule_for_member(func, prop_getter=False):
        if isinstance(func, property):
            md = func.fget.__module__ if prop_getter else func.fset.__module__
            return sys.modules[md]
        else:
            func0 = func
            while hasattr(func0, '__func__'):
                func0 = func0.__func__
            print('getmodule_for_member', func, func0)
>           return sys.modules[func0.__module__]
E           KeyError: None

func       = <function __init__ at 0x7ff77249c268>
func0      = <function __init__ at 0x7ff77249c268>
prop_getter = False

../../Vcs/pytypes/pytypes/util.py:577: KeyError
----------------------------------- Captured stdout call ------------------------------------
getmodule_for_member <function CoverageData.__attrs_post_init__ at 0x7ff772493b70> <function CoverageData.__attrs_post_init__ at 0x7ff772493b70>
getmodule_for_member <function __init__ at 0x7ff77249c268> <function __init__ at 0x7ff77249c268>
!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
============================ 1 failed, 3 passed in 0.55 seconds =============================
mname covimerage.coveragepy
mname None
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "…/Vcs/pytypes/pytypes/typelogger.py", line 357, in _dump_at_exit
    dump_cache()
  File "…/Vcs/pytypes/pytypes/typelogger.py", line 343, in dump_cache
    mnode = _module_node(mname)
  File "…/Vcs/pytypes/pytypes/typelogger.py", line 721, in __init__
    self.module = sys.modules[mname]
KeyError: None

[blueyed]

(Pdb++) getmodule(func.__code__)
(Pdb++) func.__code__
<code object __init__ at 0x7eff0d4d1390, file "<attrs generated init 1be0c750addf894649adf64b0294a1fecc7cb566>", line 1>
(Pdb++) func.__code__.co_filename
'<attrs generated init 1be0c750addf894649adf64b0294a1fecc7cb566>'
(Pdb++) 

[blueyed]

I've tried the following to reproduce it, but there it works.

I've also tried moving Foo to another module.

import pytypes                                                                      
import attr                                                                         
                                                                                    
@attr.s                                                                             
class Foo():                                                                        
    bar = attr.ib()                                                                 
                                                                                    
                                                                                    
with pytypes.TypeLogger():                                                        
    foo = Foo(bar='bar')                                                            
    print(foo.bar)                                                                  

(Pdb++) foo.__init__
<bound method __init__ of Foo(bar='bar')>
(Pdb++) foo.__init__.__code__
<code object __init__ at 0x7fabbe7f7a50, file "<attrs generated init 49980b18611a9f5fa9eb65233aa27f64d808270c>", line 1>

[Stewori]

So, the trick with co_filename seems not to be workable. Still, attrs-generated methods might still allow to identify the module somehow (finally they are hosted in some module, arent't they?). I never used attrs so far, but if you can figure out a way I will gladly include such a special treatment for attrs. What does dir(foo.__init__) look like? Does it yield some reference on the encapsulating class or module? Maybe inspect.stack holds some useful information.

[blueyed]

Would have to look at it again I guess.
IIRC there is some work in progress / ideas being discussed to provide type information.
Hopefully @hynek can help out with some pointer?!

[hynek]

Well, init (and in next release eq/ne) aren't really defined anywhere – they are generated and attached. Dunno if we can somehow attach fake module information…feel free to open an issue and we’ll have a look.

[Stewori]

Is __qualname__ available for generated methods? (this wouldn't solve it for Python 2 unless something similar would be available there too)

[hynek]

so um could y’all check whether python-attrs/attrs#316 fixes this? I’d like to kick 17.4 out before it becomes 18.1. :)

[Stewori]

Sorry, had no time to follow up on this yet. Reviewing your changes I am confident this issue can be solved based on that, but I did not yet actually try it.

[Stewori]

Honestly, the main blocker for me to follow up on this issue is lack of clear steps to reproduce it. Maybe the test code in the corresponding PR in attrs would be a starting point: https://github.com/python-attrs/attrs/pull/316/files#diff-362e0e794ee6f4465b615e60bc885157R1005