Read/write from/to streams about parquet-wasm

[kylebarron]

Well actually a stream interface is possible and was added in #296!

[mf-bruce]

I should add, working with streams has the added benefit that streams can yield to the event loop. This is helpful when processing large amounts of data in a browser context as you need to yield to the event loop to keep the UI in a functional/responsive state.

[kylebarron]

From my knowledge of rust-wasm, I don't think a stream is possible. I couldn't even figure out a way to have an async iterator in JS that iterates on batches in the rust side.

The easiest workaround for now is to use the async functions to fetch one row group of data at a time. This only works when you're accessing a remote file; otherwise for a local file you still need to have all the source bytes in memory at once, though you can at least materialize the output arrow table in batches. The other big win for memory would be fleshing out the implementation of the Arrow C Data interface in JS, (I started that here), which would prevent a copy of the table from Arrow memory in Rust to the IPC buffer.