yjh0502 / erl-brotli Goto Github PK

Running into an issue with OTP 24.3/elixir 1.13.3 and brotli on an M1 Mac air.

:brotli.encode("brotli\n=====\n\nAn OTP library\n\nBuild\n-----\n\n")

[warning] 23:21:47.608 The on_load function for module brotli_nif returned:
{:error,
 {:bad_lib,
  'Failed to find library init function: \'dlsym(0x20506ac60, _nif_init): symbol not found\''}}

** (UndefinedFunctionError) function :brotli_nif.encoder_create/0 is undefined (module :brotli_nif is not available)
    (brotli 0.3.0) :brotli_nif.encoder_create()
    (brotli 0.3.0) /Users/Me/deps/brotli/src/brotli_encoder.erl:58: :brotli_encoder.new/1
    (brotli 0.3.0) /Users/Me/deps/brotli/src/brotli.erl:60: :brotli.encode/2

If I try to compile version 0.3.1 I get the following compile error message.

Undefined symbols for architecture arm64:
  "_enif_alloc", referenced from:
      _brotli_alloc in brotli_nif.o
  "_enif_alloc_binary", referenced from:
      _brotli_encoder_take_output in brotli_nif.o
      _brotli_decoder_take_output in brotli_nif.o
  "_enif_alloc_resource", referenced from:
      _brotli_encoder_create in brotli_nif.o
      _brotli_decoder_create in brotli_nif.o
  "_enif_compare", referenced from:
      _brotli_encoder_set_parameter in brotli_nif.o
      _brotli_encoder_compress_stream in brotli_nif.o
  "_enif_free", referenced from:
      _brotli_free in brotli_nif.o
  "_enif_get_resource", referenced from:
      _brotli_encoder_set_parameter in brotli_nif.o
      _brotli_encoder_has_more_output in brotli_nif.o
      _brotli_encoder_is_finished in brotli_nif.o
      _brotli_encoder_compress_stream in brotli_nif.o
      _brotli_encoder_take_output in brotli_nif.o
      _brotli_decoder_has_more_output in brotli_nif.o
      _brotli_decoder_is_finished in brotli_nif.o
      ...
  "_enif_get_uint", referenced from:
      _brotli_encoder_set_parameter in brotli_nif.o
  "_enif_get_ulong", referenced from:
      _brotli_max_compressed_size in brotli_nif.o
  "_enif_inspect_iolist_as_binary", referenced from:
      _brotli_encoder_compress_stream in brotli_nif.o
      _brotli_decoder_decompress_stream in brotli_nif.o
  "_enif_make_atom", referenced from:
      _brotli_init in brotli_nif.o
  "_enif_make_badarg", referenced from:
      _brotli_encoder_create in brotli_nif.o
      _brotli_encoder_set_parameter in brotli_nif.o
      _brotli_encoder_has_more_output in brotli_nif.o
      _brotli_encoder_is_finished in brotli_nif.o
      _brotli_encoder_compress_stream in brotli_nif.o
      _brotli_encoder_take_output in brotli_nif.o
      _brotli_decoder_create in brotli_nif.o
      ...
  "_enif_make_binary", referenced from:
      _brotli_encoder_take_output in brotli_nif.o
      _brotli_decoder_take_output in brotli_nif.o
  "_enif_make_int", referenced from:
      _brotli_version in brotli_nif.o
  "_enif_make_resource", referenced from:
      _brotli_encoder_create in brotli_nif.o
      _brotli_decoder_create in brotli_nif.o
  "_enif_make_string", referenced from:
      _brotli_decoder_error_description in brotli_nif.o
  "_enif_make_tuple", referenced from:
      _brotli_version in brotli_nif.o
  "_enif_make_ulong", referenced from:
      _brotli_max_compressed_size in brotli_nif.o
  "_enif_open_resource_type", referenced from:
      _brotli_init in brotli_nif.o
  "_enif_release_resource", referenced from:
      _brotli_encoder_create in brotli_nif.o
      _brotli_decoder_create in brotli_nif.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [/Users/ME/deps/brotli/priv/brotli.so] Error 1
===> Hook for compile failed!

Using version v0.3.0 the library returns an :error atom when trying to encode larger binary (in my case somewhere between 1M and 2M). This breaks https://github.com/hauleth/phoenix_bakery when a large JS bundle is compressed

Steps to reproduce (in Elixir)

First create a file

head -c 10M </dev/urandom >myfile   

then

iex(1)> Mix.install([:brotli])
:ok
iex(2)> f=File.read!("myfile")                                  
<<120, 120, 3, 37, 247, 145, 138, 157, 219, 235, 102, 30, 192, 84, 58, 149, 19,
  72, 76, 103, 47, 203, 12, 251, 96, 128, 89, 26, 4, 194, 36, 47, 46, 161, 11,
  144, 36, 248, 204, 55, 28, 221, 57, 160, 156, 106, 15, 38, 149, 242, ...>>
iex(3)> res = :brotli.encode(binary_part(f, 0, 1*1024*1024))    
{:ok,
 <<171, 255, 255, 15, 120, 120, 3, 37, 247, 145, 138, 157, 219, 235, 102, 30,
   192, 84, 58, 149, 19, 72, 76, 103, 47, 203, 12, 251, 96, 128, 89, 26, 4, 194,
   36, 47, 46, 161, 11, 144, 36, 248, 204, 55, 28, 221, 57, 160, ...>>}
iex(4)> res = :brotli.encode(binary_part(f, 0, 2*1024*1024))
:error

Suspected reason

The responsible atom comes from here:

Finishing the stream means encoding of all input passed to encoder and adding specific "final" marks, so stream decoder could determine that stream is complete. To perform finish set op to BROTLI_OPERATION_FINISH. Under some circumstances (e.g. lack of output stream capacity) this operation would require several calls to BrotliEncoderCompressStream. The method must be called again until both input stream is depleted and encoder has no more output (see BrotliEncoderHasMoreOutput) after the method is called.

Environment: Broken in Elixir 1.13.4, OTP 24. Working in Elixir 1.12.3, OTP 24
erl-brotli version: 0.3.1

To reproduce:

1. Start with a test project. Add the lib as a dependency to mix.exs: {:brotli, "~> 0.3.1"}
2. Fetch and compile deps
3. Confirm that brotli.so exists in the deps dir (deps/brotli/priv/brotli.so) but not in the build dir (_build/<env>/lib/brotli/priv/brotli.so)
4. The file will also be missing in the release when it is built with mix release. Running this release will result in this error:

{"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,brotli_nif,{error,{load_failed,\"Failed to load NIF library: 'Error loading shared library /app/lib/brotli-0.3.1/priv/brotli.so: No such file or directory'\"}}}}},{kernel,start,[normal,[]]}}}"}

I don't fully understand what changed between Elixir 1.12 and 1.13 with regards to rebar3 and the build system (something with rebar bare compile?). But I also found a similar issue in another repository which also fixed the issue on this project.

Maybe someone with more knowledge on this topic can chime in.

Hi, thank you for creating this package. I was wondering how you change the compression level from 1-11? Thanks.

Hi! I'm running into an issue that I'm hoping you can help me with. It looks like brotli doesn't compile when using OTP 26. I was running 25 locally where everything ran fine, and saw the error in our CI where we're running OTP 26. I was then able to reproduce the error locally when switching to Erlang version 26.0.2. Thanks in advance!

Steps to reproduce:

1. Switch to latest Erlang asdf local erlang 26.0.2
2. rm -rf _build deps
3. mix do deps.get, deps.compile

Error:

Error! Failed to eval: io:format("~s/erts-~s/include/", [code:root_dir(), erlang:system_info(version)]).

cc -O2 -g -fno-stack-check -O3 -std=c99 -finline-functions -Wall -fPIC -I  -I ./include/  -c brotli_nif.c -o brotli_nif.o
Error! Failed to eval: io:format("~s/erts-~s/include/", [code:root_dir(), erlang:system_info(version)]).

clang: warning: ./include/: 'linker' input unused [-Wunused-command-line-argument]
In file included from brotli_nif.c:32:
./include/brotli/decode.h:15:10: fatal error: 'brotli/port.h' file not found
#include <brotli/port.h>
         ^~~~~~~~~~~~~~~
1 error generated.
Error! Failed to eval: io:format("~s/erts-~s/include/", [code:root_dir(), erlang:system_info(version)]).

cc -O2 -g -fno-stack-check -O3 -std=c99 -finline-functions -Wall -fPIC -I  -I ./include/  -c common/constants.c -o common/constants.o
Error! Failed to eval: io:format("~s/erts-~s/include/", [code:root_dir(), erlang:system_info(version)]).

make: *** [brotli_nif.o] Error 1
make: *** Waiting for unfinished jobs....
clang: warning: ./include/: 'linker' input unused [-Wunused-command-line-argument]
In file included from common/constants.c:7:
In file included from common/./constants.h:15:
common/./platform.h:28:10: fatal error: 'brotli/port.h' file not found
#include <brotli/port.h>
         ^~~~~~~~~~~~~~~
1 error generated.
make: *** [common/constants.o] Error 1

Hi @yjh0502, I have tried to contact you via email (about a month ago), but I haven't received any response, so I will request here as well.

Hi, I have spotted that you have created Erlang library to provide API for Brotli compression. I allowed myself to provide few improvements and there is now a question - would you prefer to pass maintaining that library to someone else (I am willing to do so) or I should push these changes up to your repository? I could also handle the Hex.pm package maintenance.

The improvements added so far are:

• expanding test suite, property testing, more examples, etc.
• GitHub Actions for CI
• decompression support
• streaming API support
• cleanup of C code
• better errors
• support for more configuration options
• dirty schedulers configuration for CPU intensive tasks
• code formatting
• coverage reporting

I used this library in Phoenix Brotli Compressor library for Phoenix 1.6 and I would be willing to pick up on maintenance from You on Hex.pm. Would you be willing to pass the torch to me?

Observed Behavior

The Makefile in c_src sets the project name based on the parent directory name.

For most checkouts (via Git) this would be erl-brotli. This is fine and in line with the update I just made for brotli_nif.erl to look for erl-brotli.

However, if someone uses this library (say, in a Mix project) and names it brotli, then the compiled file will be brotli.so and not erl-brotli.so

Solution

Change the Makefile and remove line 6 and change line 7 to just assign a static string:

PROJECT = "erl-brotli"

Then the compiled library will be named the same thing, regardless of its directory name.

Hi, I would like to use Brotli for compressing a stream of data. For this reason it's necessary to be able to "flush" the stream at the end of a frame (whatever the app is doing to frame the data stream).

The issue is that if the stream sends some number of bytes to the encoder, the encoder might not necessarily emit enough data to the output to allow the decoder to fully decode that input. A "flush" forces the encoder to empty it's state and transmit any pending data (it hurts compression performance, but emits the buffered data)

This extra API call would make the brotli library useful for a wider array of use cases.

Thanks for listening