[Feature Request] Support for stream source about cjs-module-lexer

[guybedford]

There has been previous discussion on this topic in guybedford/es-module-lexer#38.

The short of it though is that for smaller source files (under 500KB), the sync processing really is fastest since context switching costs would be more than just finishing the parse which is really quite fast.

The first step to take if we deem performance to be an issue (which already it is highly optimized), would be to use the native C version of this library instead of the JS version. There is a bit of work needed to land this though to the quality of C code needed in Node.js core. But that would get roughly a 1.5x speed improvement.

After that the next thing is optimizing the actual calls even further to find the bottlenecks.

After all that is done, I would rather consult those who are experts on C++ performance to see what next steps might be. I don't get the impression though that streaming would be on the list there.

[aduh95]

OK thanks for the info, good to know it's already been thought through.

But that would get roughly a 1.5x speed improvement.

According to the readme, WASM version is 1.5x the speed of JS; would that make the C version 2.25x faster than the JS version or do you mean C perf are roughly the same as WASM? Not that it matters here, I'm just wondering.

[guybedford]

Yes, actually the C version would likely be another 1.5x the speed of Wasm so you are probably right!