Error on encoding large binaries about erl-brotli

[hauleth]

@marmor157 another reason why I prefer using Erlang implementation for compressing via brotli:encode/{1,2} is that it create less stress on the schedulers (even dirty). Having NIF that will run few seconds can be problematic even with dirty schedulers.

[hauleth]

Will look into that.

[hauleth]

I think that the problem is different - brotli:encode/1 can be used with binaries that are at most 1.25 MiB (that is 1310721 bytes). Question is how we should handle such case. Whether disallow passing such huge binaries at once and force user to use streaming parsing, or rather "internally" split the input into smaller chunks and parse them separately.

[marmor157]

For brotli:encode/1 I would suggest using BrotliEncoderCompress as it performs one-shot memory-to-memory compression, and is essentialy what brotli:encode/1 is now, but also add brotli:encode_stream which would internally use BrotliEncoderCompressStream and would allow to handle output in chunks. But to do that there would have to be a way to handleavailable_in, next_in, next_out and total_out which are used to resume compression

[hauleth]

@marmor157 the problem is that BrotliEncoderCompress supports only small set of the options that brotli:encode/2 supports. Technically I could do some runtime checks to pick proper implementation, however that was easier.

[bblaszkow06]

Awesome, thanks @hauleth for the quick fix & update.