loki-astari / jsonbenchmark

Json benchmark based on nativejson-benchmark

JsonBenchmark

Benchmark Results:

Conformance mac linux
Performance mac linux

Disclaimer

The original author (Milo Yip) of this benchmark is also the primary author of RapidJSON.
The next author (Loki Astari) of this benchmarks is also the primary author of ThorsSerializer

Introduction

These benchmarks were originally written by Milo in the project nativejson-benchmarkback between 2014 and 2016. Milo no longer seems to be actively updating the project so I have am trying to resurrect the project. I have taken it upon my self to re-write the wrapper code in cleaner C++ (this is why it is not a fork of the original) and port over as many of the original libraries as I could (with as little modification to the original testing code as possible).

The original test harness interface written by Milo has been maintained in the file src/ThirdParty/test.h. Also the original tests in data remain the same, though the directory structure inside data may have changed and the files in the directory "data/validate_*/*.json" contain both input and output rather than raw json. Though not directly visible the php code that generates the results html and all the javascript have been taken directly from Milo's original project.

Original Intent

You can look back at Milo's original project and read the README over there. But basically he was trying to provide both conformance and performance metrics.

New Intent

The intent of this library is to allow:

• New Json Libraries to be easily added for inclusion.
• New Test suites to be easily added for inclusion.
• New Test data for test suites.
• The results to be automatically published for both Mac and Linux.

We will allow readers of the tests to decide what the data means.

Flexibility

All updates to this project will be accepted via pull requests. Please check that the projects continue to build correctly. The easy way to do this is via a travis-ci account. This is free you simply need to link your github account and allow travis-ci accesses to the JsonBenchmark repository.

Create a fork of this project. Then checkout your fork and build it locally.

> git checkout [email protected]:<You>/JsonBenchmark.git
> cd JsonBenchmark
> git submodule update --init --recursive
> ./configure
> # To install standard packages that are used to run tests.
> ./initPlatform
> # To build all 3rd party JSON libraries.
> ./init3rdParty
> # Build the test platform.
> make
> ./runTests

Adding a new Json Library

Run the script scripts/AddJsonLibrary

Running this script will generate the file buildJsonLibrary. Running this script will add the appropriate files to automatically generate the appropriate libraies.

After running these two scripts you will simply need to fill out the C++ to use the the Json library.

Tests

Executing Class	Data Directory	SetUp/TearDown Name	TestBase API Used	TestDescription
FailChecker	jsonchecker_fail	<Dir>/<File>	Parse()	Tests that should all fail.
PassChecker	jsonchecker_pass	<Dir>/<File>	Parse()	Tests that should all pass.
ValidateFloat	validate_float	vector-double	ParseDouble()	Float Value (as a string) and a floating point value. Make sure the string is correctly converted to a floating point value.
ValidateString	validate_string	vector-string	ParseString()	String with escape sequences and utf-8 encoded string. Make sure the escaped string is correctly converted to utf-8.
RoundTripChecker	roundtrip	<Dir>/<File>	Parse() Stringify()	A Json object is read into internal representation then converted back to a string. Ignore space (not in a string) make sure they are the same.
PerformanceChecker	performance	<Dir>/<File>		Some big Json objects. See below:
PerformanceChecker	performance	<Dir>/<File>	Parse()	Parse: Parse Json string into object: Get execution time.
PerformanceChecker	performance	<Dir>/<File>	Parse() Stringify()	Stringify: Convert internal object to string (Object from Parse): Get execusion time.
PerformanceChecker	performance	<Dir>/<File>	Parse() Prettify()	Prettify: Like stringify but is supposed do generate neat Json: Get execution time.
PerformanceChecker	performance	<Dir>/<File>	Parse() Statistics()	Statistics: Make sure the correct number of objects are created by Parse.
PerformanceChecker	performance	<Dir>/<File>	SaxRoundtrip()	SaxRoundtrip: Needs documentation.
PerformanceChecker	performance	<Dir>/<File>	SaxStatistics()	SaxStatistics: Needs documentation.

Adding a new Test Suite

Adding a new test suite is a two part processes.

• Add a new directory under data to hold the test data.
• Add a test suite

Adding new Test data

Adding new test data to an existing test suite is as simple as adding a file to the appropriate directory under data.
The test harness will automatically loop through all test data files in the directory belonging to the test suite.
NOTE: All tests files must end in ".json" if the file name includes the name "EXCLUDE" it will be ignored.

jsonbenchmark's Issues

Stats table: duplicate column "FirstCommit"

In the stats table (https://lokiastari.com/Json/Stats.html), there are two columns named "FirstCommit":

I guess the second one should be "MostRecentCommit" or something similar.

Add JSON check

The nlohmann/json has a function accept which just checks whether an input is valid JSON. As no internal structure is built, this check is much faster than parsing. Maybe other libraries have this as well, and benchmarking it could be helpful.

Add test for maximal nesting of arrays/objects

Almost all JSON parsers have limitations in the maximal nesting of arrays or objects. While

[[[[[[[[[[[[[[[[[[[["Too deep"]]]]]]]]]]]]]]]]]]]]

certainly will break no library any more, having the same input with a few thousand more [] will. It would be great to know the limits of the libraries.

Question: Why having conformance test on Mac and Linux

Are there any differences running the conformance tests on different operating systems?

Add UTF-8 conformance tests

When implementing nlohmann/json, I struggled a lot with the UTF-8 validation. I'm now pretty sure that I will only accept correctly encoded inputs and reject anything else. It would be great to add such tests to the conformance tests, because it makes a difference whether a parser just copies everything between quotes or actually checks whether the strings make sense. This also includes decoding of \uxxxx escapes.

Please add version numbers to library names

Many libraries are in constant development, and it would be helpful to know which version was benchmarked.