Parse large JSONs using a stream parser about strawberryfield

[DiegoPino]

@giancarlobi pinging you here again, this is old but i keep this issues as my personal diary and when i come back and back to the same idea i have to conclude its worth persuing.

Question: Wonder if we can measure memory usage in PHP just based of different JSON string lenghts?

Now that i'm working closer on Importing XML and CSV directly into JSON i want to be sure we can handle anything.

Idea would be to decide at which point/string size we swap normal json_decode for one of these ones (specially salsify one, can't stop thinking this is something California related?).

@alliomeria i know this is not your realm but i will keep pinging you. The idea here is that when we take a very very large JSON String (like what you see in SBF raw is quite small) ,and i will share here a HUGE one coming from a Finding AID import or even when you have 1000 pages for a book and its not a PDF nor a Sub Graph/Linked Objects Graph (i hate the word compound) the JSON, that string loaded and transformed into a PHP Structure uses memory and some times a lot. This issue is to think when to change the way we decode JSON, the implications, the why. A JSON string can be up to 2Gbytes in our Database!

If we can use the Salsify code we could

 $listener = new InMemoryListener();
 $stream = fopen($testfile, 'r'); // $testfile here would need to be an streaming endpoint http or a pointer. We do not use files!
 try {
    $parser = new \JsonStreamingParser\Parser($stream, $listener);
    $parser->parse();
   fclose($stream);
} catch (Exception $e) {
    fclose($stream);
    throw $e;
}

 $actual = $listener->getJson(); // Same as a JSON_DECODE but lower memory footprint.

The real beauty lies here
https://github.com/salsify/jsonstreamingparser/tree/master/src/Listener
Basically when you parse like this, the Listener provides events and data as the JSON progresses, and allows us, e.g to only get the data we need and leave all the rest out from memory. But there are also Format specific listeners (Looking at @dmer) which means we can deal with some pretty intensive data coming from Huge GeoJSONs.

This example is gold
https://github.com/salsify/jsonstreamingparser/blob/master/example/example_regex.php

Thanks!

[DiegoPino]

PS: this is not for Archipelago (right now) but since @giancarlobi and i are using reactPHP for realtime background processing its good to mention that you can stream MYSQL

https://clue.engineering/2018/introducing-reactphp-mysql