vscode: "Allocation failed - JavaScript heap out of memory" for large dataset about sanddance

[danmarshall]

Hi @bkmeneguello, thanks for posting this. There will of course, be some maximum limit of what can be loaded / displayed. Currently, we don't know exactly what that limit is, but it is probably around 500k rows.

A followup issue, is that the app should fail more gracefully than crashing.

[bkmeneguello]

Hi @danmarshall, thanks for your answer but where these limits come from? It's a js engine limitation or something that could be tuned? I have enough memory to accommodate all these data multiple times, so there could be possible to load lots of data (also for video memory). I understand if this can lead to performance issues since js is single threaded and workers aren't always an option.

[danmarshall]

Hi @bkmeneguello, searching that error message, I see the there is a 1GB limit for Node.js heap size: nodejs/node#17878 (comment)

We probably need to have a chunking strategy instead of loading the entire file all at once.

[danmarshall]

I'd be happy to take a PR for splitting the file into chunks.
The backend code is here, and the frontend code is here

That will get us around this specific error message, but we may have the same buffer problem in the frontend, eventually. Not sure how to deduce what the limits of one's browser should be.