rustpython / parser

RustPython

A Python-3 (CPython >= 3.12.0) Interpreter written in Rust 🐍 😱 🤘.

Usage

Check out our online demo running on WebAssembly.

RustPython requires Rust latest stable version (e.g 1.67.1 at February 7th 2023). If you don't currently have Rust installed on your system you can do so by following the instructions at rustup.rs.

To check the version of Rust you're currently running, use rustc --version. If you wish to update, rustup update stable will update your Rust installation to the most recent stable release.

To build RustPython locally, first, clone the source code:

git clone https://github.com/RustPython/RustPython

Then you can change into the RustPython directory and run the demo (Note: --release is needed to prevent stack overflow on Windows):

$ cd RustPython
$ cargo run --release demo_closures.py
Hello, RustPython!

Or use the interactive shell:

$ cargo run --release
Welcome to rustpython
>>>>> 2+2
4

NOTE: For windows users, please set RUSTPYTHONPATH environment variable as Lib path in project directory. (e.g. When RustPython directory is C:\RustPython, set RUSTPYTHONPATH as C:\RustPython\Lib)

You can also install and run RustPython with the following:

$ cargo install --git https://github.com/RustPython/RustPython
$ rustpython
Welcome to the magnificent Rust Python interpreter
>>>>>

If you'd like to make https requests, you can enable the ssl feature, which also lets you install the pip package manager. Note that on Windows, you may need to install OpenSSL, or you can enable the ssl-vendor feature instead, which compiles OpenSSL for you but requires a C compiler, perl, and make. OpenSSL version 3 is expected and tested in CI. Older versions may not work.

Once you've installed rustpython with SSL support, you can install pip by running:

cargo install --git https://github.com/RustPython/RustPython --features ssl
rustpython --install-pip

You can also install RustPython through the conda package manager, though this isn't officially supported and may be out of date:

conda install rustpython -c conda-forge
rustpython

WASI

You can compile RustPython to a standalone WebAssembly WASI module so it can run anywhere.

Build

cargo build --target wasm32-wasi --no-default-features --features freeze-stdlib,stdlib --release

Run by wasmer

wasmer run --dir `pwd` -- target/wasm32-wasi/release/rustpython.wasm `pwd`/extra_tests/snippets/stdlib_random.py

Run by wapm

$ wapm install rustpython
$ wapm run rustpython
>>>>> 2+2
4

Building the WASI file

You can build the WebAssembly WASI file with:

cargo build --release --target wasm32-wasi --features="freeze-stdlib"

Note: we use the freeze-stdlib to include the standard library inside the binary. You also have to run once rustup target add wasm32-wasi.

JIT (Just in time) compiler

RustPython has a very experimental JIT compiler that compile python functions into native code.

Building

By default the JIT compiler isn't enabled, it's enabled with the jit cargo feature.

cargo run --features jit

This requires autoconf, automake, libtool, and clang to be installed.

Using

To compile a function, call __jit__() on it.

def foo():
    a = 5
    return 10 + a

foo.__jit__()  # this will compile foo to native code and subsequent calls will execute that native code
assert foo() == 15

Embedding RustPython into your Rust Applications

Interested in exposing Python scripting in an application written in Rust, perhaps to allow quickly tweaking logic where Rust's compile times would be inhibitive? Then examples/hello_embed.rs and examples/mini_repl.rs may be of some assistance.

Disclaimer

RustPython is in development, and while the interpreter certainly can be used in interesting use cases like running Python in WASM and embedding into a Rust project, do note that RustPython is not totally production-ready.

Contribution is more than welcome! See our contribution section for more information on this.

Conference videos

Checkout those talks on conferences:

• FOSDEM 2019
• EuroPython 2018

Use cases

Although RustPython is a fairly young project, a few people have used it to make cool projects:

• GreptimeDB: an open-source, cloud-native, distributed time-series database. Using RustPython for embedded scripting.
• pyckitup: a game engine written in rust.
• Robot Rumble: an arena-based AI competition platform
• Ruff: an extremely fast Python linter, written in Rust

Goals

• Full Python-3 environment entirely in Rust (not CPython bindings)
• A clean implementation without compatibility hacks

Documentation

Currently along with other areas of the project, documentation is still in an early phase.

You can read the online documentation for the latest release, or the user guide.

You can also generate documentation locally by running:

cargo doc # Including documentation for all dependencies
cargo doc --no-deps --all # Excluding all dependencies

Documentation HTML files can then be found in the target/doc directory or you can append --open to the previous commands to have the documentation open automatically on your default browser.

For a high level overview of the components, see the architecture document.

Contributing

Contributions are more than welcome, and in many cases we are happy to guide contributors through PRs or on Discord. Please refer to the development guide as well for tips on developments.

With that in mind, please note this project is maintained by volunteers, some of the best ways to get started are below:

Most tasks are listed in the issue tracker. Check issues labeled with good first issue if you wish to start coding.

To enhance CPython compatibility, try to increase unittest coverage by checking this article: How to contribute to RustPython by CPython unittest

Another approach is to checkout the source code: builtin functions and object methods are often the simplest and easiest way to contribute.

You can also simply run ./whats_left.py to assist in finding any unimplemented method.

Compiling to WebAssembly

See this doc

Community

Chat with us on Discord.

Code of conduct

Our code of conduct can be found here.

Credit

The initial work was based on windelbouwman/rspython and shinglyu/RustPython

Links

These are some useful links to related projects:

• https://github.com/ProgVal/pythonvm-rust
• https://github.com/shinglyu/RustPython
• https://github.com/windelbouwman/rspython

License

This project is licensed under the MIT license. Please see the LICENSE file for more details.

The project logo is licensed under the CC-BY-4.0 license. Please see the LICENSE-logo file for more details.

parser's Issues

Stack overflow when parsing pandas.io.formats.xml

I've been investigating it, but the problem is deep down in the generated python parser (parser/src/python.rs)
thread 'main' has overflowed its stack while parsing https://github.com/pandas-dev/pandas/blob/main/pandas/io/formats/xml.py

Parsing code:

let input = std::fs::read_to_string(xml_py_path).unwrap();
parse(input, Mode::Module, "<embedded>")

Missing syntax for context manager expressions

Summary

New syntax from 3.10 with (... as x, ... as y): of context managers is not supported. Checkout mtshiba/pylyzer#30.

Expected

Add it :c

Actual

Python Documentation

Issue: https://bugs.python.org/issue12782
Release notes: https://www.python.org/downloads/release/python-3100/

Port PEP-701 support from Ruff

New layout for `Arguments` node to enhance performance. (and more so if possible)

Currently the rust AST nodes layout is almost same as Python ast defined in Python side.

pub struct Arguments<R = TextRange> {
    pub range: OptionalRange<R>,
    pub posonlyargs: Vec<Arg<R>>,
    pub args: Vec<Arg<R>>,
    pub vararg: Option<Box<Arg<R>>>,
    pub kwonlyargs: Vec<Arg<R>>,
    pub kw_defaults: Vec<Expr<R>>,
    pub kwarg: Option<Box<Arg<R>>>,
    pub defaults: Vec<Expr<R>>,
}

Which is less parser-friendly than:

pub struct Arguments<R = TextRange> {
    range: ...,
    posonlyargs: Vec<Argument>,
    args: Vec<Argument>,
    vararg: Option<Box<Argument>>,
    kwonlyargs: Vec<Argument>,
    kwarg: Option<Box<Argument>>,
    type_comment: Option<String>,  // unparsed type_comment
    // no defaults and kw_defaults here
}

struct Argument {
    arg: Identifier,
    annotation: Option<Expr>,
    default: Option<Expr>,   // default is placed here unlike Arg
}

cc @MichaReiser

Parser incompatibility with "def woohoo(a, *, b): pass"

>>> ast.dump(ast.parse("def woohoo(a, *, b): pass"))
"Module(body=[FunctionDef(name='woohoo', args=arguments(posonlyargs=[], args=[arg(arg='a')], kwonlyargs=[arg(arg='b')], kw_defaults=[None], defaults=[]), body=[Pass()], decorator_list=[])], type_ignores=[])"

>>> ast.dump(rustpython_parse("def woohoo(a, *, b): pass"))
"Module(body=[FunctionDef(name='woohoo', args=arguments(posonlyargs=[], args=[arg(arg='a')], kwonlyargs=[arg(arg='b')], kw_defaults=[], defaults=[]), body=[Pass()], decorator_list=[])], type_ignores=[])"

kw_defaults=[None] vs kw_defaults=[]

Split parser core / compiler core

current compiler core includes codegen related features.

Add README

Would be a good idea to add a README describing why this is separate from main.

Add CI for RustPython/Ruff tests

Instead of only running Rust tests, we should also add jobs for running RustPythons (and Ruffs if there's the desire?) tests with every PR.

This will result in a larger wait time but should catch some confusing bugs, i.e RustPython/RustPython#4995

borrowed ast and default builtin types to immutable - String -> Box<str> etc

zero-copy parser

Most of text just exists on source code. We don't need to copy them until the source exist.

This is not trivial but worth to try if we really need to do.

How do I regenerate the parser?

I am using this library for a research project that requires me to modify the grammar of Python. I have been stuck trying to regenerate the parser since I am not sure what is means to "Run lalrpop src/python.lalrpop."

Any help is appreciated.
Thank you.

Move pyo3 feature out of ast crate

It was bad idea. It doesn't need to be a feature.

Give proper location for JoinedStr children

tracking issue for python/cpython#81639

Use of abbreviations and jargon

I assume that the use of abbreviating statement as stmt is for CPython compatibility. Another reason might be that it's slightly less to type but I think that's no longer a big concern, now that we have powerful IDEs like rust-analyzer.

The downside I see with using abbreviations (and chargon) is that it increases the barrier for new contributors for RustPython or downstream crates. Some of them may never have heard of stmt or what an expr is and it takes them, maybe only a little, time to understand that it is an abbreviation for statement.

That's why we at rome had the following two philosophies when it comes to naming:

• Utilize verbosity when naming commands and flags. No unnecessary and confusing abbreviations.
• Reduce jargon. Don’t assume that users will understand specific terminology. Strive to provide clear meaning for experts and beginners. For example, use “character” where you would traditionally use “token” when producing parser errors.

I'm curious to hear your perspective on this subject.

Add `Node` union

Add a new Node union that is an enum over all node types. This can be useful when implementing methods that can operate on any node. For example, Ruff's formatter has (roughly) the API format(node: AnyNode) -> String

I haven't figured out the full requirements yet, and I don't know yet if we'll need both the owned and reference versions:

#[derive(Debug, Clone, PartialEq, Eq, Hash)]
enum Node {
	IfStmt(ast::IfStmt),
	ConstantExpr(ast::ConstantExpr),
	...
}

#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash, is_macro)]
enum NodeRef<'a> {
	IfStmt(&'a ast::IfStmt),
	ConstantExpr(&'a ast::ConstantExpr)
	...
}

Note: An alternative naming more in line with Path and PathBuf would be to name the types NodeBuf (owned) and Node (reference)

The enums should have the following basic methods:

• if_stmt(self) -> Option<ast::IfStmt>
• as_if_stmt(&self) -> Option<&ast::IfStmt>
• const is_if_stmt(&self) -> bool

Node could also implement AsRef that returns a NodeRef

I may have time to work on this sometime soon but I wanted to put this up for discussion first to get feedback.

Question: How to walk/visit the AST?

I'm trying to extract all the imports from a python file. To do this I'm using the rustpython_parser::parse function to get an AST and then iterating over all the statements in the body to find Stmt::Import and Stmt::ImportFrom. This works for imports defined at the root of the file, however misses any imports defined inside e.g. function/class definitions etc. So I think what I really want to do is walk the AST and visit all the nodes. Is there some way to achieve this with one of the rustpython packages?

Publish on crates.io

Is there a plan to periodically publish updates of this crate to crates.io? Will like to use it externally without depending on git.

If this is not planned please feel free to close this issue.

Consolidate parsing of `Expr::Name` in `TypeAliasName`, `MatchName`, and `Atom<Goal>`

We have three parser descriptions for in the LALRPOP grammar for generating Expr::Name from an Identifier.

• TypeAliasName
• MatchName
• Atom<Goal>

These could be consolidated into a single reused identifier for simplicity and clarity.

Originally posted by @MichaReiser in #97 (comment)

Too many elif chain crashes parser

Code generator maybe able to generate it.
CPython cannot run this code anyway.

test_too_many_elif.py.zip

make unparse public

the unparse feature currently does not expose any additional functionality as the module is not public?

Parser/ast/src/lib.rs

Line 23 in b079666

mod unparse;

Hundread of files that are properly parsed(or not) in opposite to Python 3.11.2

d23611d

When parsing with this repo multiple files, I found that sometimes python cannot parse file but this library can and vice versa.

Command to check if python can parse file

python -m py_compile PY_FILE_TEST_427160.py

Code to check if parser can parse file

let rust_valid = parse(content.as_str(), Mode::Module,"").is_ok();

Only ~3% files from pack, can be parsed by python, but not rustpython parser

Pack 644 files - OUTPUT_FILES.zip

Example files and errors

return imprt

SyntaxError: 'return' outside function

\
 import _pl

Sorry: IndentationError: unexpected indent (79285088PY_FILE_TEST_5137609254.py, line 2)

# encoding: ut

SyntaxError: unknown encoding: ut

__p|ersion__ = '2.9.0'

SyntaxError: cannot assign to expression here. Maybe you meant '==' instead of '='?

from __future__ import un

SyntaxError: future feature un is not defined

Enable CI

Cannot compile with Rust 1.76.0

Works with 1.75.0, but here is the issue with 1.76.0

  --> /home/rfiszel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustpython-ast-0.3.0/src/impls.rs:60:1
   |
60 | static_assertions::assert_eq_size!(crate::Stmt, [u8; 160]);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: source type: `generic::Stmt` (1216 bits)
   = note: target type: `[u8; 160]` (1280 bits)
   = note: this error originates in the macro `static_assertions::assert_eq_size` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types
  --> /home/rfiszel/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustpython-ast-0.3.0/src/impls.rs:62:1
   |
62 | static_assertions::assert_eq_size!(crate::Pattern, [u8; 96]);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: source type: `generic::Pattern` (704 bits)
   = note: target type: `[u8; 96]` (768 bits)
   = note: this error originates in the macro `static_assertions::assert_eq_size` (in Nightly builds, run with -Z macro-backtrace for more info)

Investigate alternate PHF algorithms for detecting keywords

This relates to a discussion on discord a while back.

The basic idea is that we can create a custom hash function for our keywords that's based on their structure (see related video in discussion) and possibly gain a good perf boost by replacing rust_phf. It still needs some investigating and bench-marking to actually evaluate if this would result in any significant improvement.

Afaik, this usually results in a bigger table being generated (for a keyword set of 30 odd functions, we might need a table of 1024 or 2048 elements mostly filled with None's) but that's a small price to pay considering this table should only be generated for the keywords and nothing else.

Implement `ToString` for `Identifier` and `Constant`

Would it be useful to implement ToString for Identifier and Constant?

Reject generator without parenthesis

Ref RustPython/RustPython#3942

Generator expression without parenthesis does not raise SyntaxError

Feature

In CPython, generator expression without parenthesis raises following error

>>> foo(x, z for z in range(10), t, w)
  File "<stdin>", line 1
    foo(x, z for z in range(10), t, w)
           ^^^^^^^^^^^^^^^^^^^^
SyntaxError: Generator expression must be parenthesized

However, In RustPython, it does not matter

>>>>> foo(x, z for z in range(10), t, w)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'foo' is not defined

Python Documentation

Error recovery/resilience

Currently, all of the parse functions return Result<ast, ParseError>. This means that if the file has a syntax error, we get the error and nothing else. However, there are some use cases in which having access to parts which did parse successfully would be beneficial:

• Language servers usually want to keep functioning when the file is temporarily broken during editing.
• Linters might want to skip over broken parts and continue reporting lints, or maybe up to the syntax error.

On the other hand, interpreters usually don't have too much use for this AFAIK.

lalrpop seems to have some support for error recovery: https://lalrpop.github.io/lalrpop/tutorial/008_error_recovery.html
Alex Kladov has written about error recovery, most recently here: https://matklad.github.io/2023/05/21/resilient-ll-parsing-tutorial.html

Is RustPython/Parser interested in error recovery or is it out of scope?

Forked from RustPython main repository

This issue is to record how this repository can be related to history of its original owner's branch.

This repository is forked from RustPython's main repository

The forked trunk is this commit: RustPython/RustPython@306943a
The original fork is archived here:

• branch: https://github.com/RustPython/Parser/tree/fork-init
• commit: 7159de8

Current main commit e6eb49f filtered out codegen from the original commit above.

A macro to generating AST with python grammar

Inspired by https://docs.rs/pmutil/latest/pmutil/macro.smart_quote.html

e.g.

ExprKind::Tuple(ast::ExprTuple {
    elts: names
        .iter()
        .map(|&name| {
            create_expr(ExprKind::Constant(
                ast::ExprConstant {
                    value: Constant::Str(name.to_string()),
                    kind: None,
                },
            ))
        })
        .collect(),
    ctx: ExprContext::Load,
}))

can be rewritten to:

&python_ast! {
    Vars { names },
    "(*names)"
}

The grammar will be limited to legal python codes to leverage python parser.

Illegal "yield (from)" python syntax accepted

Originally reported as astral-sh/ruff#4992.

I'm not quite sure where the incorrectness here is, but I think this bug indicates an incorrect parsing of the yield from expression. The following is a python snippet with illegal syntax:

def e():
  x = yield from y = 5

We can verify that it is indeed illegal syntax with:

$ python yield-elision.py 
  File "/tmp/tmp.1z4PScoI6T/yield-elision.py", line 2
    x = yield from y = 5
        ^^^^^^^^^^^^
SyntaxError: assignment to yield expression not possible

Ruff, however, believes this is valid syntax:

$ ruff yield-elision.py --show-source --show-fixes
yield-elision.py:2:3: F841 [*] Local variable `x` is assigned to but never used
  |
2 | def e():
3 |   x = yield from y = 5
  |   ^ F841
  |
  = help: Remove assignment to unused variable `x`

yield-elision.py:2:18: F821 Undefined name `y`
  |
2 | def e():
3 |   x = yield from y = 5
  |                  ^ F821
  |

Found 2 errors.

And will attempt to fix it, leading to a self-reported parsing error:

$ ruff yield-elision.py  --fix
error: Autofix introduced a syntax error in `yield-elision.py` with rule codes F841: invalid syntax. Got unexpected token '=' at byte offset 24
---
def e():
  yield from y = 5

---
yield-elision.py:2:3: F841 Local variable `x` is assigned to but never used
yield-elision.py:2:18: F821 Undefined name `y`
Found 2 errors.

I believe this indicates a parsing bug.

Implement `PartialEq<Builtin> for str/String`

We should implement PartialEq<Builtin> for &str and PartialEq<Builtin> for String so that both ident == str and str == ident work.

Error building wheel for rustpython_ast (pyo3)

I cloned the repository, entered the ast-pyo3 directory, and ran pyproject-build:

% pyproject-build 
* Creating venv isolated environment...
* Installing packages in isolated environment... (maturin>=0.15,<0.16)
* Getting build dependencies for sdist...
* Building sdist...
Running `maturin pep517 write-sdist --sdist-directory /media/data/dev/griffe/rustpython_parser/ast-pyo3/dist`
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.11 at /tmp/build-env-m11693t3/bin/python3
📡 Using build options features from pyproject.toml
📦 Built source distribution to /media/data/dev/griffe/rustpython_parser/ast-pyo3/dist/rustpython_ast-0.0.1.tar.gz
rustpython_ast-0.0.1.tar.gz
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (maturin>=0.15,<0.16)
* Getting build dependencies for wheel...
* Building wheel...
Running `maturin pep517 build-wheel -i /tmp/build-env-ra719lal/bin/python --compatibility off`
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.11 at /tmp/build-env-ra719lal/bin/python
📡 Using build options features from pyproject.toml
   Compiling autocfg v1.1.0
   Compiling libc v0.2.153
   Compiling proc-macro2 v1.0.79
   Compiling cfg-if v1.0.0
   Compiling unicode-ident v1.0.12
   Compiling target-lexicon v0.12.14
   Compiling version_check v0.9.4
   Compiling once_cell v1.19.0
   Compiling ppv-lite86 v0.2.17
   Compiling ahash v0.8.11
   Compiling siphasher v0.3.11
   Compiling num-traits v0.2.18
   Compiling phf_shared v0.11.2
   Compiling libm v0.2.8
   Compiling zerocopy v0.7.32
   Compiling syn v1.0.109
   Compiling pyo3-build-config v0.19.2
   Compiling getrandom v0.2.12
   Compiling quote v1.0.35
   Compiling rand_core v0.6.4
   Compiling either v1.10.0
   Compiling hashbrown v0.14.3
   Compiling rand_chacha v0.3.1
   Compiling itertools v0.11.0
   Compiling rand v0.8.5
   Compiling ryu v1.0.17
   Compiling phf_generator v0.11.2
   Compiling phf_codegen v0.11.2
   Compiling unicode-width v0.1.11
   Compiling crunchy v0.2.2
   Compiling getopts v0.2.21
   Compiling malachite-base v0.4.5
   Compiling num-integer v0.1.46
   Compiling num-bigint v0.4.4
   Compiling anyhow v1.0.81
   Compiling log v0.4.21
   Compiling tiny-keccak v2.0.2
   Compiling paste v1.0.14
   Compiling unicode_names2_generator v1.2.2
   Compiling pyo3-ffi v0.19.2
   Compiling syn v2.0.53
   Compiling lock_api v0.4.11
   Compiling parking_lot_core v0.9.9
   Compiling unic-char-range v0.9.0
   Compiling unic-common v0.9.0
   Compiling memchr v2.7.1
   Compiling Inflector v0.11.4
   Compiling convert_case v0.4.0
   Compiling unicode_names2 v1.2.2
   Compiling rustpython-parser-vendored v0.3.1 (/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/local_dependencies/rustpython-parser-vendored)
   Compiling unic-ucd-version v0.9.0
   Compiling unic-char-property v0.9.0
   Compiling memoffset v0.9.0
   Compiling smallvec v1.13.2
   Compiling scopeguard v1.2.0
   Compiling derive_more v0.99.17
   Compiling rustpython-parser v0.3.1 (/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/local_dependencies/rustpython-parser)
   Compiling phf v0.11.2
   Compiling pyo3-macros-backend v0.19.2
   Compiling pyo3 v0.19.2
   Compiling malachite-nz v0.4.5
   Compiling is-macro v0.3.5
   Compiling static_assertions v1.1.0
   Compiling rustpython-parser-core v0.3.1 (/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/local_dependencies/rustpython-parser-core)
   Compiling parking_lot v0.12.1
   Compiling pyo3-macros v0.19.2
   Compiling unic-emoji-char v0.9.0
   Compiling unic-ucd-ident v0.9.0
   Compiling num-complex v0.4.5
   Compiling unindent v0.1.11
   Compiling lalrpop-util v0.20.2
   Compiling indoc v1.0.9
   Compiling rustc-hash v1.1.0
   Compiling malachite-q v0.4.5
   Compiling malachite v0.4.5
   Compiling malachite-bigint v0.2.0
   Compiling rustpython-ast v0.3.1 (/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/local_dependencies/rustpython-ast)
error[E0252]: the name `bigint` is defined multiple times
  --> local_dependencies/rustpython-ast/src/lib.rs:28:9
   |
26 | pub use malachite_bigint as bigint;
   |         -------------------------- previous import of the module `bigint` here
27 | #[cfg(feature = "num-bigint")]
28 | pub use num_bigint as bigint;
   |         ^^^^^^^^^^^^^^^^^^^^ `bigint` reimported here
   |
   = note: `bigint` must be defined only once in the type namespace of this module
help: you can use `as` to change the binding name of the import
   |
28 | pub use num_bigint as other_bigint;
   |         ~~~~~~~~~~~~~~~~~~~~~~~~~~

warning: unused import: `num_bigint as bigint`
  --> local_dependencies/rustpython-ast/src/lib.rs:28:9
   |
28 | pub use num_bigint as bigint;
   |         ^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

For more information about this error, try `rustc --explain E0252`.
warning: `rustpython-ast` (lib) generated 1 warning
error: could not compile `rustpython-ast` (lib) due to 1 previous error; 1 warning emitted
warning: build failed, waiting for other jobs to finish...
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit status: 101": `PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="/tmp/build-env-ra719lal/bin/python" PYTHON_SYS_EXECUTABLE="/tmp/build-env-ra719lal/bin/python" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/tmp/build-via-sdist-_6kryhu2/rustpython_ast-0.0.1/Cargo.toml" "--release" "--lib"`
Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/build-env-ra719lal/bin/python', '--compatibility', 'off'] returned non-zero exit status 1

ERROR Backend subprocess exited when trying to invoke build_wheel

Any hint on how to fix that 🙂?

As to why I'm trying to build it, well, I'd like to try it in a project of mine to see how well it performs and how much work it would need to use it instead of Python's ast.

Quick question by the way: it builds an Abstract Syntax Tree and not a Concrete Syntax Tree right?

Licencing parser

We currently license it under MIT while previously it was licensed under a dual MIT/CC-BY-4.0 license. It would probably be a good idea to use similar licenses.

Fix bool format

Related to RustPython/RustPython#4950

bool format doesn't work correctly.
I'll fix FormatSpec.format_bool.

Add a benchmarking suite

There's been a couple of PRs lately that have been opened in hopes of speeding things up. We should probably add a good set of benchmarks that can be re-used by everyone trying to make an optimization change so as to make changes like these easier to evaluate.

Ideally, we can also yank some of Ruffs benchmarks to also see how much a change affects it does heavily consume this library.

Add test case for malformed float

https://github.com/RustPython/RustPython/pull/3194/files#diff-f3c6d1f7423d91e0c39ccb0cb8e3cb98cc1fc27bf9bcb01d09fe76edee0a501aR495-R499

Implement Parse for located AST nodes

mtshiba/pylyzer#37 (comment)

PEP 695 – Type Parameter Syntax

Feature

Seeing as how Python 3.12 is now in beta, I feel like RustPython should also work on being able to parse the new syntax introduced in PEP 695, which makes it much easier to type-hint generic functions and types in the language. I ran into this while trying to start writing a custom type checker, and wanted to use rustpython_ast to parse Python 3.12 code that uses this new syntax. Some examples include:

def identity[T](x: T) -> T:
    return x

type Point = tuple[float, float]

type MappingOrSequence[T] = Mapping[int, T] | Sequence[T]

class Container[T](Sequence[T]): ...

Python Documentation or reference to CPython source code

• The type statement
• Generic functions
• Generic classes
• Generic type aliases

Incorrect f-string parsing

Originally reported as astral-sh/ruff#5044:

Not the best string ever, but ruff doesn't handle it like Python

λ cat test.py
print(f"$N_{{img}}\in\{{0,1,2\}}$")
λ python test.py
$N_{img}\in\{0,1,2\}$
λ ruff check test.py --select F401
error: Failed to parse test.py:1:25: f-string: unterminated string

Build fails on i386: error[E0512]: cannot transmute between types of different sizes, or dependently-sized types

error[E0512]: cannot transmute between types of different sizes, or dependently-sized types
  --> /wrkdirs/usr/ports/devel/pylyzer/work/pylyzer-0.0.51/cargo-crates/rustpython-ast-0.3.0/src/impls.rs:65:1
   |
65 | static_assertions::assert_eq_size!(crate::ExceptHandler, [u8; 64]);
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: source type: `generic::ExceptHandler` (288 bits)
   = note: target type: `[u8; 64]` (512 bits)
   = note: this error originates in the macro `static_assertions::assert_eq_size` (in Nightly builds, run with -Z macro-backtrace for more info)

For more information about this error, try `rustc --explain E0512`.
error: could not compile `rustpython-ast` (lib) due to 1 previous error

log

Improvement: Add `is_*` methods to `Tok`

Ruff has some lint rules that run on tokens. It would be useful to have is_* methods to avoid extensive matching.

Recover update_asdl and related test

The generated ast must be always aligned with Python.asdl

Consider using `Box` on type parameters to reduce statement allocation size

In #93, we add type_params: Vec to ClassDef and FunctionDef which increases the size of a statement. We may want to optimize this field to reduce the size of the statement struct.

The main reason for the assertion is to be aware of how changing the AST changes its size. This is important because ruff and RustPython hold many instances of these types and increasing their size mean:

• Each element in a Vec now requires more storage -> Overall increased memory consumption
• Reading or writing an element becomes slower because your computer must read or write more bytes. This is especially relevant for reading where the L1 cache short circuits reads by caching about 64 bytes of neighboring memory to avoid subsequent reads from (the very slow) memory.

There's not much we can do about this here. This could be a place where using tinyvec over a regular Vec would be beneficial, considering that the vec will be empty for almost all instances. But I think this is fine for now.

Originally posted by @MichaReiser in #93 (comment)

Add trait Parse and Parse::parse to use with each types.

With code

let parsed = parse(code, path, Mode::Exec)

The result parsed is always ast::ModModule. But it currently returns ast::Mod::Module.

suggestion:

let parsed = ast::ModModule::parse(code, path)

previous parse_* functions can remains for CPython friendly API

add full-parser, like full-lexer

Currently it does not seem to be possible to get an AST with comments attached?