Comments (12)
Sounds good to me! #37 is a fitting pr number :)
from libcst.
I'll update the PR title for better autocomplete if other folks search this in the future
from libcst.
This should be supported on master as of a few minutes ago. I don't have any real-world Python 3.5/3.6 that I can run this against easily. However, I've at least verified that the tokenizer works properly and all of our parsing/rendering tests have been updated with 3.6 and below versions as appropriate.
If you end up with time to give this a try and it works to your satisfaction, let me know!
from libcst.
Not sure if this is supported:
source_tree = cst.parse_module(python_source, config=cst.PartialParserConfig(python_version="3.6"))
Tried running but still got the async
error.
from libcst.
Ah just saw this:
LibCST/libcst/_parser/_types/config.py
Lines 85 to 89 in 68699e2
from libcst.
Hey, @zhammer. Thanks for your interest in LibCST!
Unfortunately we only support 3.7's syntax right now. Sorry for the confusing documentation around python_version
. It's there because we intend to support multiple versions of Python syntax in the future. We should probably improve that and raise a ValueError
if we see anything other than 3.7.x
.
This is actually a limitation in Parso's tokenizer.
CPython 3.6's tokenizer contains a (really ugly) hack that tracks whether you're inside an async function or not, and decides which kind of token to emit for the async
and await
keywords.
Black's tokenizer (which is similar to Parso's) contains a version of this logic. I think we'd need to port that logic from Black to Parso to fix this problem.
I've upstreamed this issue as davidhalter/parso#83.
Out of curiosity: How many of these syntax errors do you have in your codebase? If it's small enough, it might be worth fixing them by hand, especially seeing as this is a requirement for upgrading to Python 3.7+ anyways.
from libcst.
Ah, interesting. So, I actually had thought it'd be a fun test-drive of LibCST to automate the 3.6 -> 3.7 syntax refactor - specifically the async keyword case as the codemod seemed pretty straightforward. That being said, it's no problem to do the 3.7 refactor manually.
Thanks for tagging me on those issues! Will follow along.
Also, this was a (very ugly) hack I used to get libcst to run on 3.6 code with async keyword, but at least was able to check my visitor output on those files.
diff --git a/libcst/_parser/_base_parser.py b/libcst/_parser/_base_parser.py
index b143a15..70f2afc 100644
--- a/libcst/_parser/_base_parser.py
+++ b/libcst/_parser/_base_parser.py
@@ -95,6 +95,7 @@ class BaseParser(Generic[_TokenT, _TokenTypeT, _NodeT]):
self.tokens = tokens
self.lines = lines
self._pgen_grammar = pgen_grammar
+ pgen_grammar.__dict__['reserved_syntax_strings'].pop('async', None)
first_dfa = pgen_grammar.nonterminal_to_dfas[start_nonterminal][0]
self.stack = [StackNode(first_dfa)]
self.__was_parse_called = False
Let me know if you'd want a PR for the non-3.7 ValueError
.
from libcst.
I've submitted #37 to raise a ValueError
when we receive a non-3.7 python version. It also should improve the documentation for PartialParserConfig
.
@zhammer if it's okay with you, I think I'll close this once #37 lands, since 3.6 support is an upstream issue and it sounds like Parso isn't interested in supporting 3.6's syntax (davidhalter/parso#83).
from libcst.
@zhammer, I know we said that we weren't planning on supporting it, but it turns out this may become a reality with #60 and #63.
from libcst.
awesome, thanks for keeping me in mind for the update :) i've had a lot of fun experimenting with libcst so far!
from libcst.
I’m traveling but can test once back! An easy way to test, I imagine, would be a one line file: async = True
. This is valid 3.6 syntax as async
isn’t a reserved keyword.
from libcst.
I have tests for that in the tokenizer (even better, the single line async
is valid python, its a variable lookup which throws a NameError
at runtime). I'm worried more about the fact that the tokenizer hacks had to be ported from lib2to3 to parso and might have some issues on complex edge cases. If you run into any, feel free to open an issue and I'll get it looked at.
from libcst.
Related Issues (20)
- ParserSyntaxError for "unmatched ']'" faults wrong location
- AttributeError: module 'libcst' has no attribute '_InverseOf'
- Issue with `whitespace_after_*` not fixable HOT 2
- MaybeSentinel not dataclass
- Implement PEP 696 (defaults for type params) HOT 3
- pyright + `__or__` in generic type annotations gives unknown type HOT 1
- Python 3.13.0b1: Segmentation fault when running tests HOT 2
- Indentation wart when removing an Arg from a Call - is whitespace being represented in the right place? HOT 3
- `ApplyTypeAnnotationsVisitor` does not respect `overwrite_existing_annotations` for attributes
- CST validation error on multiline match-case-if blocks
- Failing to build wheel HOT 1
- Accessing .code attribute fails with "TypeError: CSTNode._codegen() missing 1 required positional argument: 'state'" HOT 3
- Importable libcst.tests.pyre
- [support] running test suite doesn’t find libcst.native HOT 2
- Recursion depth exceeded when applying MetadataWrapper on a very long concatenated string HOT 2
- test_parse_error for native parser HOT 1
- [Windows, Mac, Linux in future] Can't use call_if_inside/call_if_not_inside matcher decorators with multiprocessing
- Warnings persist in Context HOT 2
- Link https://libcst.me/ in the docs? HOT 1
- No mute option for reels
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libcst.