Astroid 2.2 and Python 3.8 test failures about asttokens

[dsagal]

I added a commit on master which fixes the issues caused by astroid 2.2

[hroncok]

I get the same failures with Python 3.7. It's the asttokens version where things started to differ. I'll try to find out which.

[hroncok]

The failures start with astroid 2.2.

[vstinner]

I wrote PR #29 to "Fix some Python 3.8 issues".

[keszybz]

With the patch (git master at 21caaaa), tests pass with python3-astroid-2.2.5-1.fc30.noarch.rpm + python3-3.7.3-1.fc30.x86_64.rpm and python3-astroid-2.2.5-2.fc31.noarch + python3-3.7.3-2.fc31.x86_64 but fail with python3-3.8.0~a3-1.fc31.x86_64 + python3-astroid-2.2.5-2.fc31.noarch. Thus, it seems that compat with astroid is done, but compat with python3.8 not yet.

[hroncok]

Here is the output with Python 3.8 on 21caaaa:

$ tox -e py38
GLOB sdist-make: .../asttokens/setup.py
py38 inst-nodeps: .../asttokens/.tox/.tmp/package/1/asttokens-1.1.13.zip
py38 installed: astroid==2.2.0,asttokens==1.1.13,coverage==4.5.3,lazy-object-proxy==1.3.1,nose==1.3.7,six==1.12.0,typed-ast==1.3.5,wrapt==1.11.1
py38 run-test-pre: PYTHONHASHSEED='303088453'
py38 runtests: commands[0] | nosetests
......S....FF..F.FFFF.......F......F.......F.EFFFF..........EE..FFE.....F....F.
======================================================================
ERROR: test_comprehensions (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 369, in test_comprehensions
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_mark_tokens.py", line 21, in create_mark_checker
    return tools.MarkChecker(source, parse=True)
  File ".../asttokens/tests/tools.py", line 105, in __init__
    self.atok = asttokens.ASTTokens(source, parse=parse, tree=tree)
  File ".../asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File ".../asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File ".../asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File ".../asttokens/asttokens/util.py", line 184, in visit_tree
    ret = postvisit(current, par_value, value)
  File ".../asttokens/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File ".../asttokens/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File ".../asttokens/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File ".../asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got ENDMARKER:'' on line 2 col 1

======================================================================
ERROR: test_fixture7 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 144, in test_fixture7
    def test_fixture7(self): self.verify_fixture_file('astroid/format.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_mark_tokens.py", line 21, in create_mark_checker
    return tools.MarkChecker(source, parse=True)
  File ".../asttokens/tests/tools.py", line 105, in __init__
    self.atok = asttokens.ASTTokens(source, parse=parse, tree=tree)
  File ".../asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File ".../asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File ".../asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File ".../asttokens/asttokens/util.py", line 184, in visit_tree
    ret = postvisit(current, par_value, value)
  File ".../asttokens/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File ".../asttokens/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File ".../asttokens/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File ".../asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got OP:'=' on line 7 col 3

======================================================================
ERROR: test_fixture8 (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 145, in test_fixture8
    def test_fixture8(self): self.verify_fixture_file('astroid/module.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 107, in verify_fixture_file
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_mark_tokens.py", line 21, in create_mark_checker
    return tools.MarkChecker(source, parse=True)
  File ".../asttokens/tests/tools.py", line 105, in __init__
    self.atok = asttokens.ASTTokens(source, parse=parse, tree=tree)
  File ".../asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File ".../asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File ".../asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File ".../asttokens/asttokens/util.py", line 184, in visit_tree
    ret = postvisit(current, par_value, value)
  File ".../asttokens/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File ".../asttokens/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File ".../asttokens/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File ".../asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got OP:'=' on line 53 col 19

======================================================================
ERROR: test_mark_tokens_simple (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 47, in test_mark_tokens_simple
    m = self.create_mark_checker(source)
  File ".../asttokens/tests/test_mark_tokens.py", line 21, in create_mark_checker
    return tools.MarkChecker(source, parse=True)
  File ".../asttokens/tests/tools.py", line 105, in __init__
    self.atok = asttokens.ASTTokens(source, parse=parse, tree=tree)
  File ".../asttokens/asttokens/asttokens.py", line 65, in __init__
    self.mark_tokens(self._tree)
  File ".../asttokens/asttokens/asttokens.py", line 76, in mark_tokens
    MarkTokens(self).visit_tree(root_node)
  File ".../asttokens/asttokens/mark_tokens.py", line 47, in visit_tree
    util.visit_tree(node, self._visit_before_children, self._visit_after_children)
  File ".../asttokens/asttokens/util.py", line 184, in visit_tree
    ret = postvisit(current, par_value, value)
  File ".../asttokens/asttokens/mark_tokens.py", line 90, in _visit_after_children
    nfirst, nlast = self._methods.get(self, node.__class__)(node, first, last)
  File ".../asttokens/asttokens/mark_tokens.py", line 175, in visit_listcomp
    return self.handle_comp('[', node, first_token, last_token)
  File ".../asttokens/asttokens/mark_tokens.py", line 171, in handle_comp
    util.expect_token(before, token.OP, open_brace)
  File ".../asttokens/asttokens/util.py", line 56, in expect_token
    raise ValueError("Expected token %s, got %s on line %s col %s" % (
ValueError: Expected token OP:'[', got OP:'=' on line 53 col 19

======================================================================
FAIL: test_fixture12 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 149, in test_fixture12
    def test_fixture12(self): self.verify_fixture_file('astroid/recursion.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 149, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: "' For issue #25 '\n\n\nclass Base(object):\n    pass\n\n\n" != '""" For issue #25 """\n\n\n\nclass Base(object):\n    pass\n\n\n'
- ' For issue #25 '
? ^               ^
+ """ For issue #25 """
? ^^^               ^^^
+ 
  
  
  class Base(object):
      pass
  
  


======================================================================
FAIL: test_fixture13 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 150, in test_fixture13
    def test_fixture13(self): self.verify_fixture_file('astroid/suppliermodule_test.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 149, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: "' file suppliermodule.py '\n\n\nclass No[229 chars]\n\n" != '""" file suppliermodule.py """\n\n\n\ncl[235 chars]\n\n'
- ' file suppliermodule.py '
? ^                        ^
+ """ file suppliermodule.py """
? ^^^                        ^^^
+ 
  
  
  class NotImplemented(Exception):
      pass
  
  
  
  class Interface:
      
      def get_value(self):
          raise NotImplemented()
      
      def set_value(self, value):
          raise NotImplemented()
  
  
  
  class DoNothing:
      pass
  
  


======================================================================
FAIL: test_fixture4 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 141, in test_fixture4
    def test_fixture4(self): self.verify_fixture_file('astroid/clientmodule_test.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 149, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: '\' docstring for file clientmodule.py \'\[819 chars]\n\n' != '""" docstring for file clientmodule.py ""[827 chars]\n\n'
Diff is 1219 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_fixture6 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 143, in test_fixture6
    def test_fixture6(self): self.verify_fixture_file('astroid/email.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 149, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: '"fake email module to test absolute import doesn\'t grab this one"\n\n' != '"""fake email module to test absolute import doesn\'t grab this one"""\n\n\n\n'
- "fake email module to test absolute import doesn't grab this one"
+ """fake email module to test absolute import doesn't grab this one"""
? ++                                                                 ++
  
+ 
+ 


======================================================================
FAIL: test_fixture7 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 144, in test_fixture7
    def test_fixture7(self): self.verify_fixture_file('astroid/format.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 149, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: "'A multiline string\\n'\nfunction('aeozr[290 chars]\n\n" != '"""A multiline string\n"""\n\nfunction(\[297 chars]\n\n'
- 'A multiline string\n'
? ^                  ---
+ """A multiline string
? ^^^
+ """
+ 
  function('aeozrijzearzer', hop)
  x = [i for i in range(5) if i % 4]
  fonction(1, 2, 3, 4)
  
  def definition(a, b, c):
      return a + b + c
  
  
  class debile(dict, object):
      pass
  
  if aaaa:
      pass
  else:
      (aaaa, bbbb) = (1, 2)
      (aaaa, bbbb) = (bbbb, aaaa)
  hop = aaaa
  __revision__.lower()
  


======================================================================
FAIL: test_fixture8 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 145, in test_fixture8
    def test_fixture8(self): self.verify_fixture_file('astroid/module.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 149, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: "'test module for astroid\\n'\n__revision[1773 chars]\n\n" != '"""test module for astroid\n"""\n\n__rev[1808 chars]\n\n'
Diff is 2477 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_fixture9 (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 146, in test_fixture9
    def test_fixture9(self): self.verify_fixture_file('astroid/module2.py')
  File ".../asttokens/tests/test_mark_tokens.py", line 108, in verify_fixture_file
    tested_nodes = m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 149, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: "from data.module import YO, YOUPI\nimpor[1976 chars]\n\n" != 'from data.module import YO, YOUPI\nimpor[2020 chars]\n\n'
Diff is 2689 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_print_function (tests.test_astroid.TestAstroid)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 250, in test_print_function
    m.verify_all_nodes(self)
  File ".../asttokens/tests/tools.py", line 149, in verify_all_nodes
    test_case.assertEqual(left, right)
AssertionError: "try:\n    enumerate = enumerate\nexcept [1093 chars]\n\n" != 'try:\n    enumerate = enumerate\nexcept [1131 chars]\n\n'
Diff is 1797 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_to_source (tests.test_asttokens.TestASTTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_asttokens.py", line 91, in test_to_source
    self.assertEqual(tools.to_source(root).strip(), source)
AssertionError: "def foo():\n    'xxx'\n    None" != 'def foo():\n    """xxx"""\n    None'
  def foo():
-     'xxx'
+     """xxx"""
      None

======================================================================
FAIL: test_adjacent_strings (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 229, in test_adjacent_strings
    self.assertEqual(m.view_nodes_at(2, 6), {
AssertionError: Items in the first set but not the second:
"Constant:'x y z'"
Items in the second set but not the first:
'Str:\'x y z\' \\\n\'\'\'a b c\'\'\' "u v w"'

======================================================================
FAIL: test_conditional_expr (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 342, in test_conditional_expr
    self.assertEqual(m.view_nodes_at(1, 4),
AssertionError: Items in the first set but not the second:
'Constant:True'
Items in the second set but not the first:
'NameConstant:True'

======================================================================
FAIL: test_decorators (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 449, in test_decorators
    self.assertIn('FunctionDef:@deco1\ndef f():\n  pass', m.view_nodes_at(2, 0))
AssertionError: 'FunctionDef:@deco1\ndef f():\n  pass' not found in set()

======================================================================
FAIL: test_deep_recursion (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 183, in test_deep_recursion
    self.assertEqual(m.view_node(m.all_nodes[-1]),
AssertionError: "Constant:'F1akOFFiRIgPHTZksKBAgMCLGTdGNIAAQgKfDAcgZbj0odOnUA8GBAA7'" != "Str:'F1akOFFiRIgPHTZksKBAgMCLGTdGNIAAQgKfDAcgZbj0odOnUA8GBAA7'"
- Constant:'F1akOFFiRIgPHTZksKBAgMCLGTdGNIAAQgKfDAcgZbj0odOnUA8GBAA7'
? ^^^^ ^^^
+ Str:'F1akOFFiRIgPHTZksKBAgMCLGTdGNIAAQgKfDAcgZbj0odOnUA8GBAA7'
? ^ ^


======================================================================
FAIL: test_del_dict (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 391, in test_del_dict
    self.assertEqual(m.view_nodes_at(1, 5), {'Num:4'})
AssertionError: Items in the first set but not the second:
'Constant:4'
Items in the second set but not the first:
'Num:4'

======================================================================
FAIL: test_keyword_arg_only (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 431, in test_keyword_arg_only
    self.assertEqual(m.view_nodes_at(1, 4), {'Num:1'})
AssertionError: Items in the first set but not the second:
'Constant:1'
Items in the second set but not the first:
'Num:1'

======================================================================
FAIL: test_mark_tokens_multiline (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 95, in test_mark_tokens_multiline
    self.assertEqual(all_text, {
AssertionError: Items in the second set but not the first:
'a,      # line2\nb +     # line3\n  c +   # line4\n  d'

======================================================================
FAIL: test_slices (tests.test_mark_tokens.TestMarkTokens)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_mark_tokens.py", line 205, in test_slices
    self.assertEqual(m.view_nodes_at(1, 1),
AssertionError: Items in the second set but not the first:
'Tuple:foo.Area_Code, str(foo.Phone)[:3], str(foo.Phone)[3:], foo[:], bar[::, :]'

======================================================================
FAIL: test_walk_ast (tests.test_util.TestUtil)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".../asttokens/tests/test_util.py", line 43, in test_walk_ast
    self.assertEqual(scan, [
AssertionError: Lists differ: ["Mod[202 chars]2', 'Constant:1', 'Constant:2', "BinOp:'hello'[95 chars]ld'"] != ["Mod[202 chars]2', 'Num:1', 'Num:2', "BinOp:'hello' + ', ' + [70 chars]ld'"]

First differing element 7:
'Constant:1'
'Num:1'

  ["Module:foo(bar(1 + 2), 'hello' + ', ' + 'world')",
   "Expr:foo(bar(1 + 2), 'hello' + ', ' + 'world')",
   "Call:foo(bar(1 + 2), 'hello' + ', ' + 'world')",
   'Name:foo',
   'Call:bar(1 + 2)',
   'Name:bar',
   'BinOp:1 + 2',
-  'Constant:1',
-  'Constant:2',
+  'Num:1',
+  'Num:2',
   "BinOp:'hello' + ', ' + 'world'",
   "BinOp:'hello' + ', '",
-  "Constant:'hello'",
-  "Constant:', '",
-  "Constant:'world'"]
+  "Str:'hello'",
+  "Str:', '",
+  "Str:'world'"]

----------------------------------------------------------------------
Ran 79 tests in 0.884s

FAILED (SKIP=1, errors=4, failures=18)
ERROR: InvocationError for command '.../asttokens/.tox/py38/bin/nosetests' (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py38: commands failed

[hroncok]

This is due to python/cpython#9445

Class ast.Constant is now used for all constants. Old classes ast.Num, ast.Str, ast.Bytes, ast.NameConstant and ast.Ellipsis are still available, but they will be removed in future Python releases.

[dsagal]

#29 addresses it, I think it can be simplified a bit, but it’s close.

…

On Sat, May 4, 2019 at 12:16 PM Miro Hrončok ***@***.***> wrote: This is due to python/cpython#9445 <python/cpython#9445> Class ast.Constant is now used for all constants. Old classes ast.Num, ast.Str, ast.Bytes, ast.NameConstant and ast.Ellipsis are still available, but they will be removed in future Python releases. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#28 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIKMR3CO4HNMB3CZBHELTTPTWZF7ANCNFSM4HEJKL7Q> .

[keszybz]

It's a bit better now, but many failures still:
FAILED (SKIP=1, errors=8, failures=6)

[keszybz]

Anything we can do from the Fedora side to move this forward? I'd be happy to provide a test environment with all the packages installed to make debugging easier.

[alexmojaki]

Python 3.8.0rc1 is out, which is supposed to mean no more code changes. The official release is expected in 2 weeks. Many tests are still failing on 3.8 that are unrelated to astroid. Can we change the title to "Astroid 2.2 and Python 3.8 test failures"? Is there any work currently being done on this? I'm planning to take a crack at it soon, but I haven't actually worked with this codebase before.

[dsagal]

It looks doable, I'll give it a shot this week.

[alexmojaki]

This can be closed now :)