mbdevpl / typed-astunparse

Python 3 AST unparser with type comments support.

typed-astunparse

Unparser for Python 3 abstract syntax trees (ASTs) with type comments.

The typed-astunparse is to typed-ast as astunparse is to ast. In short: unparsing of Python 3 abstract syntax trees (ASTs) with type comments.

Contents

• Why this module was created
• Usage
• Installation
  • Requirements
• Links
  • Extensions of this module
  • Who's using this module and why
  • References

Why this module was created

The built-in ast module can parse Python source code into AST but it can't generate source code from the AST. The astunparse module (using a refactored version of an obscure script found in official Python repository) provides code generation capability for native Python AST.

However, both ast and astunparse modules completely ignore type comments introduced in PEP 484. They treat them like all other comments, so when you parse the code using compile(), your type comments will be lost. There is no place for them in the AST, so obviously they also cannot be unparsed.

The typed-ast module provides an updated AST including type comments defined in PEP 484 and a parser for Python code that contains such comments.

Unfortunately, typed-ast doesn't provide any means to go from AST back to source code with type comments. This is why module typed-astunparse (i.e. this one) was created: to provide unparser for AST defined in typed-ast.

Usage

Example of roundtrip from code through AST to code:

import typed_ast.ast3
import typed_astunparse

code = 'my_string = None  # type: str'
roundtrip = typed_astunparse.unparse(typed_ast.ast3.parse(code))
print(roundtrip)

This will print:

my_string = None  # type: str

for more examples see examples.ipynb notebook.

Installation

For simplest installation use pip:

pip3 install typed-astunparse

You can also build your own version:

git clone https://github.com/mbdevpl/typed-astunparse
cd typed-astunparse
pip3 install -U test_requirements.txt
python3 -m unittest  # make sure the tests pass
python3 setup.py bdist_wheel
pip3 install dist/*.whl

Requirements

Python version 3.5 or later.

Python libraries as specified in requirements.txt.

Building and running tests additionally requires packages listed in test_requirements.txt.

Tested on Linux, OS X and Windows.

Links

Extensions of this module

If you're extending typed-astunparse and you'd like to share why, feel free to submit a pull request introducing your project.

• horast: human-oriented ast

  Built upon both typed-ast and typed-astunparse providing parsing and unparsing of arbitrary comments in addition to type comments.

  https://pypi.org/project/horast

  https://github.com/mbdevpl/horast

Who's using this module and why

If you're using typed-astunparse in your work and you'd like to share why, feel free to submit a pull request introducing your project.

• static-typing: using typed-astunparse directly to provide AST unparsing function

  https://pypi.org/project/static-typing

  https://github.com/mbdevpl/static-typing

References

• ast:

  https://docs.python.org/3/library/ast.html

  https://greentreesnakes.readthedocs.io/

• astunparse:

  https://pypi.org/project/astunparse

  https://github.com/simonpercivall/astunparse

  https://astunparse.readthedocs.io/en/latest/

• PEP 483 - The Theory of Type Hints:

  https://www.python.org/dev/peps/pep-0483/

• PEP 484 - Type Hints:

  https://www.python.org/dev/peps/pep-0484/

• PEP 3107 - Function Annotations:

  https://www.python.org/dev/peps/pep-3107/

• PEP 526 - Syntax for Variable Annotations:

  https://www.python.org/dev/peps/pep-0526/

• typed-ast:

  https://pypi.org/project/typed-ast

  https://github.com/python/typed_ast

typed-astunparse's Issues

Neither newlines nor triple quotes preserved… for docstrings

import typed_ast.ast3
import typed_astunparse

code = '''
class Snake(object):
    """
    Om nom nom
    
    and some more nom nom nom
    """
    
    def bite(me: str) -> bool:
        return me == 'cython'
'''
roundtrip = typed_astunparse.unparse(typed_ast.ast3.parse(code))
print(roundtrip)

Outputs:

class Snake(object):
    '\n    Om nom nom\n    \n    and some more nom nom nom\n    '

    def bite(me: str) -> bool:
        return (me == 'cython')

Use-case: I'm writing a docstring transformer

per-argument type comments in function definitions are unparsed incorrectly

The way those type comments are unparsed goes against rules of PEP484. This leads to errors when unparsed files are parsed again by typed_ast - no errors with ast.

def function_b8(spam=0,  # type: int
                ham='',  # type: str
                eggs=0.0):  # type: float
    """function with args with type comments"""
    spam, ham, eggs = None, None, None

After parsing and unparsing the above, it looks like:

def function_b8(spam  # type: int
        =0, ham  # type: str
        ='', eggs=0.0):
    # type: float
    'function with args with type comments'
    (spam, ham, eggs) = (None, None, None)

Which is valid code and runs, but it definitely is not a correct unparsing! And moreover, it looks ugly.

unparsing of complex and/or nested f-strings fails sometimes

It fails due to the fact that sub-strings and values in the expressions in f-strings are unparsed without knowledge what string delimiters should be used, and therefore running

import typed_ast.ast3
import typed_astunparse
print(typed_astunparse.unparse(typed_ast.ast3.parse(
    """f'length of "hello" is {len("hello")}'""", mode='eval')))

prints

f'length of "hello" is {len(\'hello\')}'

but parsing it again

typed_ast.ast3.parse("""f'length of "hello" is {len(\\'hello\\')}'""", mode='eval')

results in an error.

Traceback (most recent call last):

  File "/home/mateusz/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-12-b854d28c44ec>", line 1, in <module>
    typed_ast.ast3.parse("""f'length of "hello" is {len(\\'hello\\')}'""", mode='eval')

  File "/home/mateusz/.local/lib/python3.6/site-packages/typed_ast/ast3.py", line 58, in parse
    return _ast3._parse(source, filename, mode, feature_version)

  File "<unknown>", line 1
SyntaxError: f-string expression part cannot include a backslash

This would be fixed by implementing an f-string unparser that is aware which string delimiters should be used to generate code for each sub-string and expression within an f-string.

Problem with version-query

On my environment version-query raises ValueError. More details on mbdevpl/version-query#2

I temporally fixed issue on my environment by putting actual version number on '_version.py',

Syntax Error on Python 3.7

I am trying to run this on python 3.7.0b3 and am getting a SyntaxError when I import it. Is async a reserved keyword now?

$ python3.7
Python 3.7.0b3 (v3.7.0b3:4e7efa9c6f, Mar 29 2018, 08:11:44)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import typed_ast.ast3
>>> import typed_astunparse
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/typed_astunparse/__init__.py", line 12, in <module>
    from .unparser import Unparser
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/typed_astunparse/unparser.py", line 248
    def _generic_FunctionDef(self, t, async=False):
                                          ^
SyntaxError: invalid syntax
>>>

release sdist

Carry over from python/typed_ast#25 (comment)

It does not preserve per-argument type comments

This strips per-argument type comments (example taken from PEP 484):

code = '''
def send_email(address,     # type: Union[str, List[str]]
               sender,      # type: str
               cc,          # type: Optional[List[str]]
               bcc,         # type: Optional[List[str]]
               subject='',
               body=None    # type: List[str]
               ):
    # type: (...) -> bool
    """Send an email message.  Return True if successful."""
    pass
'''

import typed_ast.ast3
import typed_astunparse

print(typed_astunparse.unparse(typed_ast.ast3.parse(code)))

Output is:

def send_email(address, sender, cc, bcc, subject='', body=None):
    # type: (...) -> bool
    'Send an email message.  Return True if successful.'
    pass