jstuyts / jsontestsuite

A comprehensive test suite for RFC 8259 compliant JSON parsers

JSON Parsing Test Suite

A comprehensive test suite for RFC 8259 compliant JSON parsers

This repository was created as an appendix to the article Parsing JSON is a Minefield 💣.

/parsers/

This directory contains several parsers and tiny wrappers to turn the parsers into JSON validators, by returning a specific value.

• 0 the parser did accept the content
• 1 the parser did reject the content
• >1 the process did crash
• timeout happens after 5 seconds

/test_parsing/

The name of these files tell if their contents should be accepted or rejected.

• y_ content must be accepted by parsers
• n_ content must be rejected by parsers
• i_ parsers are free to accept or reject content

/test_transform/

These files contain weird structures and characters that parsers may understand differently, eg:

• huge numbers
• dictionaries with similar keys
• NULL characters
• escaped invalid strings

These files were used to produce results/transform.html.

/run_tests.py

Run all parsers with all files:

$ python3 run_tests.py

Run all parsers with a specific file:

$ python3 run_tests.py file.json

Run specific parsers with all files:

$ echo '["Python 2.7.10", "Python 3.5.2"]' > python_only.json
$ python3 run_tests.py --filter=python_only.json

The script writes logs in results/logs.txt.

The script then reads logs.txt and generates results/parsing.html.

/results/