rocky / python-spark

An Earley-Algorithm Context-free grammar Parser Toolkit

An Earley Algorithm Parser toolkit.

This package uses Jay Earley's algorithm for parsing context free grammars, and comes with some generic Abstract Syntax Tree routines. There is also a prototype scanner which does its job by combining Python regular expressions.

(SPARK stands for Scanning, Parsing, and Rewriting Kit. It is a poor name since it conflicts with a more popular package of the same name. In the future we will rename this.)

The original version of this was written by John Aycock for his Ph.d thesis and was described in his 1998 paper: "Compiling Little Languages in Python" at the 7th International Python Conference. The current incarnation of this code is maintained (or not) by Rocky Bernstein.

Note: Earley algorithm parsers are almost linear when given an LR grammar. These are grammars which are left-recursive.

Installation

This uses setup.py, so it follows the standard Python routine:

$ pip install -e .  # set up to run from source tree
                    # Or if you want to install instead
$ python setup.py develop

Example

The github example directory has a worked-out examples; Package uncompyle6 uses this and contains a much larger example.

See Also

• features
• http://pages.cpsc.ucalgary.ca/~aycock/spark/ (Old and not very well maintained)
• https://pypi.python.org/pypi/uncompyle6/

python-spark's Issues

Grammar checking

We should have a routine for warning about nonterminals that don't appear on the LHS (especially those that aren't all uppercase).

We should warn about right-recursive rules.

incorrect setup_requires in setup.py

s/setup_requires/tests_require/

Nose is a test only dependency. Using setup_requires forces it to be installed.
It should be a tests_require setting.

Allow | in input grammar

We can do at least a rudimentary version that doesn't have grouping inside including * or ?.

AST Formatting

See rocky/python-uncompyle6#24

Stack states on error stack could show LHS

Add omega memoization

To handle right recursion. See the Perl parser or the one Evan Pheonix did for inspiration.

Write some unit tests for Earley parser

Deprecation warning due to invalid escape sequences

Deprecation warnings are raised due to invalid escape sequences. This can be fixed by using raw strings or escaping the literals. pyupgrade also helps in automatic conversion : https://github.com/asottile/pyupgrade/

find . -iname '*.py' | grep -v example | xargs -P4 -I{} python3.8 -Wall -m py_compile {}
./spark_parser/spark.py:127: DeprecationWarning: invalid escape sequence \e
  _NULLABLE = '\e_'
./spark_parser/spark.py:200: DeprecationWarning: invalid escape sequence \s
  doc = os.linesep.join([s for s in doc.splitlines() if s and not re.match("^\s*#", s)])
./spark_parser/spark.py:272: DeprecationWarning: invalid escape sequence \s
  doc = os.linesep.join([s for s in doc.splitlines() if s and not re.match("^\s*#", s)])

Allow full EBNF as input

Perhaps this project could use itself to parse EBNF?

Note: doing this will fix Issue #10