PHP Fatal error: Allowed memory size exhausted about json-machine

[halaxa]

Hello. Please take a look at #3 (you probably using it wrong). If that's not your case, please provide a malfunctioning code sample and data. Thanks.

[milinmestry]

Thanks for your quick response @halaxa.

As per #3, it is true that I'm trying to read the entire json file.
What I need is to read the key and it's respective value from root level.
Because the subtree results is a key at root level it is giving Fatal error to me.
The subtree results is consists of an array with almost 60K records.
I can read the subtree results using json pointer, that works great!

Is there is a way I can skip the key results from the root level and read all other keys and its values. Fetch the value for specific keys like root/last-update-time, root/resultCount from the entire json file.

Here is the structure of the json file:

{
"last-update-time":"2019-04-09 01:00:00.0",
"timezone":"PDT",
"results":[
{ },
{ },
{ },
],
"resultCount":60638
}

[halaxa]

There is no feature to skip a key (results in your case). But I see why that could be useful. If you have control over the json structure, I'd suggest to put resultCount key before the results key and thus make the whole structure make more sense. Putting count at the end makes little sense to me, because I could have them counted by the time I reach the resultCount key. If you don't have control over the json structure and the order of the keys does not change across requests, I'd suggest to read first two keys and then stop. Then use json pointer and read the results key. You can manually count the results which means you don't need to get to resultCount.

JsonMachine is intended to be simple iterator over huge json structure. If one needs more complexity, I'd recommend https://github.com/salsify/jsonstreamingparser. It's more complicated to set up, but more low level and may better fit your needs.

Dou you think JsonMachine should have support skipping "big" keys? How would the function signature look like? Thanks.

[milinmestry]

Sorry, I don't have control on the json structure since it is coming from api response.
Since at first go I have to read the metadata from the json response. There is an another script, that reads the results json object and stored it in database table.

Since you mentioned

JsonMachine is intended to be simple iterator over huge json structure.

so I thought it should work start from the root level and not specific from subtree inside json response.

Definitely it will be helpful if we could only read the required keys from the json object. At this moment I haven't thought of how we can skip the large keys, but I will try it in spare time.

Thanks again.

[halaxa]

Why it does not work

so I thought it should work start from the root level and not specific from subtree inside json response.

If you wanted to start in document root and have control over depth of the document, you'd have to do tree traversing. Travevesing tree in client code always adds some complexity, that's why philosophy of Json machine avoids it. Thanks to that the initial example in readme is so simple. The main goal was just to make a library you throw some huge list on and iterate it via foreach. In most of the cases one just needs to process big list of items. That's what Json machine is for.

Let's imagine

It does not mean some inventive implementation could not take place. I have an idea for some time now. What if the value you are given in foreach wouldn't be parsed structure but rather lazy reference. It would not parse and load anything into memory until you for example access an index on it. Therefore you could iterate on the root level and just continue if you want to skip a big key. This implementation would be mostly backwards compatible with current solution and more flexible then adding another parameter to iterator. Not sure when I get to it though.

[halaxa]

Will be addressed in #36

[halaxa]

Long time no update from me. Currently one can utilize json pointer or recursive parsing waiting to mrege in #36 to achieve the result you want.