Big primitive field still seem to have memory footprint when ignored about stream-json

[uhop]

I didn't understand the business with disassembler(), so I removed it. The following code works for me:

const fs = require('fs');

const {chain} = require('stream-chain');

const {parser} = require('stream-json');
const {pick} = require('stream-json/filters/Pick');
const {ignore} = require('stream-json/filters/Ignore');
const {streamArray} = require('stream-json/streamers/StreamArray');

const main = () => {
  chain([
    fs.createReadStream('./data-112.json', {encoding: 'utf8'}),
    parser(),
    pick({filter: 'result'}),
    ignore({filter: /huge/}),
    streamArray(),
    data => console.log(data)
  ]);
};

try {
  main();
} catch (error) {
  console.error('ERROR:', error);
}

data-112.json:

{
  "result": [{"id": 1, "name": "Richard", "huge": "one huge string!"}]
}

It produces the expected result:

// $ node issue-112.js
{ key: 0, value: { id: 1, name: 'Richard' } }

Please let me know if I missed anything significant to your case.

[balthazar]

one huge string was just an example, you're supposed to replace it with an actual huge string, but I guess I wasn't explicit enough 😄

I published an example repo here, where the script memory is limited to 12mb.

You just have to run npm start and you'll see the memory issue I'm referencing above

[uhop]

I see what you mean. You should turn off packing values on Parser, which is on by default as a convenience. Now the problem is that filters usually operate on keys, so keys are assumed to be small enough to fit memory — they should be packed. Another problem is that streamers assume that individual objects fit the memory so they want all values to be packed. It all means that we need to add some trivial programming:

const fs = require('fs');

const {chain, none} = require('stream-chain');

const {parser} = require('stream-json');
const {pick} = require('stream-json/filters/Pick');
const {ignore} = require('stream-json/filters/Ignore');
const {streamArray} = require('stream-json/streamers/StreamArray');

const packedValues = {endString: 'stringValue', endNumber: 'numberEnd', endKey: 'keyValue'};

const packer = () => {
  let accumulator = '';
  return function* (chunk) {
    switch (chunk.name) {
      case 'stringChunk':
      case 'numberChunk':
      case 'keyChunk':
        accumulator += chunk.value;
        break;
      case 'endString':
      case 'endNumber':
      case 'endKey':
        const value = accumulator;
        accumulator = '';
        yield chunk;
        // add a value token
        yield {name: packedValues[chunk.name], value};
        return;
    }
    yield chunk;
  };
};

const main = () => {
  chain([
    fs.createReadStream('./data-112.json', {encoding: 'utf8'}),
    parser({packValues: false, packKeys: true}),
    pick({filter: 'result'}),
    ignore({filter: /huge/}),
    packer(),
    streamArray(),
    data => console.log(data)
  ]);
};

try {
  main();
} catch (error) {
  console.error('ERROR:', error);
}

As you can see:

• I told Parser to stop packing values save for keys.
• Created a trivial packer that glues values together.
  • stream-json guarantees that all tokens go in a strict order described in the documentation.

The parser options and the stream of tokens are described in the Parser documentation.

I probably should make it more clear in the documentation. Another idea is to add a packer to streamers and as a stand-alone utility.

[uhop]

Closing unless a follow up is needed.