Can you ignore ParserSyntaxError (for reserved async keyword in 3.6)? about libcst

[zhammer]

Sounds good to me! #37 is a fitting pr number :)

[zhammer]

I'll update the PR title for better autocomplete if other folks search this in the future

[DragonMinded]

This should be supported on master as of a few minutes ago. I don't have any real-world Python 3.5/3.6 that I can run this against easily. However, I've at least verified that the tokenizer works properly and all of our parsing/rendering tests have been updated with 3.6 and below versions as appropriate.

If you end up with time to give this a try and it works to your satisfaction, let me know!

[zhammer]

Not sure if this is supported:

source_tree = cst.parse_module(python_source, config=cst.PartialParserConfig(python_version="3.6"))

Tried running but still got the async error.

[zhammer]

Ah just saw this:

LibCST/libcst/_parser/_types/config.py

Lines 85 to 89 in 68699e2

	# `parse_version_string` will raise a ValueError if the version is invalid.
	#
	# We use object.__setattr__ because the dataclass is frozen. See:
	# https://docs.python.org/3/library/dataclasses.html#frozen-instances
	# This should be safe behavior inside of `__post_init__`.

.

[bgw]

Hey, @zhammer. Thanks for your interest in LibCST!

Unfortunately we only support 3.7's syntax right now. Sorry for the confusing documentation around python_version. It's there because we intend to support multiple versions of Python syntax in the future. We should probably improve that and raise a ValueError if we see anything other than 3.7.x.

This is actually a limitation in Parso's tokenizer.

CPython 3.6's tokenizer contains a (really ugly) hack that tracks whether you're inside an async function or not, and decides which kind of token to emit for the async and await keywords.

Black's tokenizer (which is similar to Parso's) contains a version of this logic. I think we'd need to port that logic from Black to Parso to fix this problem.

I've upstreamed this issue as davidhalter/parso#83.

Out of curiosity: How many of these syntax errors do you have in your codebase? If it's small enough, it might be worth fixing them by hand, especially seeing as this is a requirement for upgrading to Python 3.7+ anyways.

[zhammer]

Ah, interesting. So, I actually had thought it'd be a fun test-drive of LibCST to automate the 3.6 -> 3.7 syntax refactor - specifically the async keyword case as the codemod seemed pretty straightforward. That being said, it's no problem to do the 3.7 refactor manually.

Thanks for tagging me on those issues! Will follow along.

Also, this was a (very ugly) hack I used to get libcst to run on 3.6 code with async keyword, but at least was able to check my visitor output on those files.

diff --git a/libcst/_parser/_base_parser.py b/libcst/_parser/_base_parser.py
index b143a15..70f2afc 100644
--- a/libcst/_parser/_base_parser.py
+++ b/libcst/_parser/_base_parser.py
@@ -95,6 +95,7 @@ class BaseParser(Generic[_TokenT, _TokenTypeT, _NodeT]):
         self.tokens = tokens
         self.lines = lines
         self._pgen_grammar = pgen_grammar
+        pgen_grammar.__dict__['reserved_syntax_strings'].pop('async', None)
         first_dfa = pgen_grammar.nonterminal_to_dfas[start_nonterminal][0]
         self.stack = [StackNode(first_dfa)]
         self.__was_parse_called = False

Let me know if you'd want a PR for the non-3.7 ValueError.

[bgw]

I've submitted #37 to raise a ValueError when we receive a non-3.7 python version. It also should improve the documentation for PartialParserConfig.

@zhammer if it's okay with you, I think I'll close this once #37 lands, since 3.6 support is an upstream issue and it sounds like Parso isn't interested in supporting 3.6's syntax (davidhalter/parso#83).

[bgw]

@zhammer, I know we said that we weren't planning on supporting it, but it turns out this may become a reality with #60 and #63.

[zhammer]

awesome, thanks for keeping me in mind for the update :) i've had a lot of fun experimenting with libcst so far!

[zhammer]

I’m traveling but can test once back! An easy way to test, I imagine, would be a one line file: async = True. This is valid 3.6 syntax as async isn’t a reserved keyword.

[DragonMinded]

I have tests for that in the tokenizer (even better, the single line async is valid python, its a variable lookup which throws a NameError at runtime). I'm worried more about the fact that the tokenizer hacks had to be ported from lib2to3 to parso and might have some issues on complex edge cases. If you run into any, feel free to open an issue and I'll get it looked at.