bagder / libbrotli

meta project to build libraries from the brotli source code

Hi, I'm Daniel!

I am and have been an avid open source supporter and developer since the mid 1990s. My code is almost guaranteed to run in your devices, in one way or another. I work exclusively on Open Source since many years.

I founded and lead the development of the curl project and I do spurious contributions in other network related projects like libssh2, c-ares and trurl.

I have authored the following books: Uncurled, everything curl, HTTP/3 explained and http2 explained.

File issues and pull-requests in any and all of the projects I participate in. Contact me via email at [email protected] on Mastodon or IRC in the #curl channel on libera.chat.

Everything about me and my projects are also available on my website.

🌟 GitHub star | 🏆 Polhem Prize | 🏆 Nordic Free Sofware Award | 🏆 Google Patch Award | 🏆 Sweden's 2nd best Developer

💚 Thank you awesome sponsors! 💚

libbrotli's Issues

Failed to execute process './autogen.sh'

Hello, I have an error while executing autogen :

root@pulse ~/libbrotli# ./autogen.sh
Failed to execute process './autogen.sh'. Reason:
exec: Exec format error
The file './autogen.sh' is marked as an executable but could not be run by the operating system.

Need a release

Hi

I want to help ngx_brotli easier to install on OSX using homebrew.
But before I can do it.
I will need a release like this.

Is there any plan for a release?

Thanks

Versioning

The current versioning of brotli is not clear to me.
There is a git tag 0.2.0, the draft having -05 and we have a define #define BROTLI_VERSION "0.1.0"

Maybe we could agree on something here.

build fails on the command line tool

From #43

../brotli/c/tools/brotli.c:20:10: fatal error: brotli/decode.h: No such file or directory
 #include <brotli/decode.h>
          ^~~~~~~~~~~~~~~~~
compilation terminated.

Building on Linux w/ libtool-2.4.6, autoconf-2.59 and automake-1.11.6

Hi,

I can upgrade versions if needed, but I'd rather not if not necessary.

From a clean git clone:

/usr/local/src/libbrotli# ./autogen.sh
Submodule 'brotli' (https://github.com/google/brotli) registered for path 'brotli'
Cloning into 'brotli'...
remote: Counting objects: 1772, done.
remote: Total 1772 (delta 0), reused 0 (delta 0), pack-reused 1772
Receiving objects: 100% (1772/1772), 9.05 MiB | 1.60 MiB/s, done.
Resolving deltas: 100% (1070/1070), done.
Submodule path 'brotli': checked out '1886f77ac76685689593c100362594aa648b1c30'
configure.ac:8: error: Autoconf version 2.62 or higher is required
/usr/share/aclocal-1.11/init.m4:26: AM_INIT_AUTOMAKE is expanded from...
configure.ac:8: the top level autom4te: /usr/bin/m4 failed with exit status: 63
aclocal: autom4te failed with exit status: 63
autoreconf: aclocal failed with exit status: 63

gcc is v4.7.2

Any help appreciated.
Thanks,

cc1plus: error: unrecognized command line option "-std=c++11"

Hi,

Output of "gcc --version"

[root@datanode2 lib]# gcc --version
gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
I have installed below package by yum:
devtoolset-2-gcc.x86_64 4.8.2-15.1.el6 @slc6-devtoolset

Output of "yum info gcc":

Installed Packages
Name : gcc
Arch : x86_64
Version : 4.4.7
Release : 18.el6
Size : 19 M
Repo : installed
From repo : base
Summary : Various compilers (C, C++, Objective-C, Java, ...)
URL : http://gcc.gnu.org
License : GPLv3+ and GPLv3+ with exceptions and GPLv2+ with exceptions
Description : The gcc package contains the GNU Compiler Collection version 4.4.
: You'll need this package in order to compile C code.

can anyone help me why I am getting this error. Default gcc package was gcc4.4.7. After that I installed gcc4.8.2 then also I am getting this error..

Any help would be appreciated..

cc1plus: error: unrecognized command line option "-std=c++11"

Trying to compile this for use with ngx_brotli on CentOS 6.7 and getting the following errors

yum list autoconf automake gcc -q
Installed Packages
autoconf.noarch     2.63-5.1.el6    @base
automake.noarch     1.11.1-4.el6    @base
gcc.x86_64          4.4.7-16.el6    @base

errors

*************************************************
install libbrotli
Initialized empty Git repository in /svr-setup/libbrotli/.git/
make: *** No rule to make target `clean'.  Stop.
No local changes to save
Already up-to-date.
Submodule 'brotli' (https://github.com/google/brotli) registered for path 'brotli'
Initialized empty Git repository in /svr-setup/libbrotli/brotli/.git/
Submodule path 'brotli': checked out '1886f77ac76685689593c100362594aa648b1c30'
libtoolize: putting auxiliary files in `.'.
libtoolize: linking file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: linking file `m4/libtool.m4'
libtoolize: linking file `m4/ltoptions.m4'
libtoolize: linking file `m4/ltsugar.m4'
libtoolize: linking file `m4/ltversion.m4'
libtoolize: linking file `m4/lt~obsolete.m4'
configure.ac:7: installing `./config.guess'
configure.ac:7: installing `./config.sub'
configure.ac:8: installing `./install-sh'
configure.ac:8: installing `./missing'
Makefile.am: installing `./depcomp'

----------------------------------------------------------------
Initialized build system. For a common configuration please run:
----------------------------------------------------------------

./configure

checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for gcc... ccache /usr/bin/gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether ccache /usr/bin/gcc accepts -g... yes
checking for ccache /usr/bin/gcc option to accept ISO C89... none needed
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by ccache /usr/bin/gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1966080
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from ccache /usr/bin/gcc object... ok
checking how to run the C preprocessor... ccache /usr/bin/gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if ccache /usr/bin/gcc supports -fno-rtti -fno-exceptions... no
checking for ccache /usr/bin/gcc option to produce PIC... -fPIC -DPIC
checking if ccache /usr/bin/gcc PIC flag -fPIC -DPIC works... yes
checking if ccache /usr/bin/gcc static flag -static works... no
checking if ccache /usr/bin/gcc supports -c -o file.o... yes
checking if ccache /usr/bin/gcc supports -c -o file.o... (cached) yes
checking whether the ccache /usr/bin/gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for style of include used by make... GNU
checking dependency style of ccache /usr/bin/gcc... gcc3
checking whether to enable maintainer-specific portions of Makefiles... no
checking whether we are using the GNU C++ compiler... yes
checking whether ccache /usr/bin/g++ accepts -g... yes
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether ccache /usr/bin/g++ accepts -g... (cached) yes
checking dependency style of ccache /usr/bin/g++... gcc3
checking how to run the C++ preprocessor... ccache /usr/bin/g++ -E
checking for ld used by ccache /usr/bin/g++... /usr/bin/ld -m elf_x86_64
checking if the linker (/usr/bin/ld -m elf_x86_64) is GNU ld... yes
checking whether the ccache /usr/bin/g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking for ccache /usr/bin/g++ option to produce PIC... -fPIC -DPIC
checking if ccache /usr/bin/g++ PIC flag -fPIC -DPIC works... yes
checking if ccache /usr/bin/g++ static flag -static works... no
checking if ccache /usr/bin/g++ supports -c -o file.o... yes
checking if ccache /usr/bin/g++ supports -c -o file.o... (cached) yes
checking whether the ccache /usr/bin/g++ linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking dependency style of ccache /usr/bin/g++... (cached) gcc3
configure: creating ./config.status
config.status: creating Makefile
config.status: creating libbrotlienc.pc
config.status: creating libbrotlidec.pc
config.status: creating config.h
config.status: executing libtool commands
config.status: executing depfiles commands
make  all-am
make[1]: Entering directory `/svr-setup/libbrotli'
  CC     brotli/dec/libbrotlidec_la-decode.lo
  CC     brotli/dec/libbrotlidec_la-bit_reader.lo
  CC     brotli/dec/libbrotlidec_la-huffman.lo
  CC     brotli/dec/libbrotlidec_la-state.lo
  CC     brotli/dec/libbrotlidec_la-streams.lo
  CC     brotli/dec/libbrotlidec_la-dictionary.lo
  CXX    brotli/enc/libbrotlienc_la-backward_references.lo
cc1plus: error: unrecognized command line option "-std=c++11"
  CXX    brotli/enc/libbrotlienc_la-histogram.lo
make[1]: *** [brotli/enc/libbrotlienc_la-backward_references.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
cc1plus: error: unrecognized command line option "-std=c++11"
make[1]: *** [brotli/enc/libbrotlienc_la-histogram.lo] Error 1
make[1]: Leaving directory `/svr-setup/libbrotli'
make: *** [all] Error 2
  CCLD   libbrotlidec.la
  CXX    brotli/enc/libbrotlienc_la-backward_references.lo
cc1plus: error: unrecognized command line option "-std=c++11"
make: *** [brotli/enc/libbrotlienc_la-backward_references.lo] Error 1

Adoption by brotli upstream?

I've asked for adoption by the upstream project: google/brotli#326

ARM - Ansi C

Can I use your lib in ARM Cortex M3 ?

VERSION_INFO unused

Makefile.am defines a LIBBROTLI{DEC,ENC}_VERSION_INFO = -version 1:0:0 variable, but this is never used. As a result, the library's name will be libbrotlienc.so.0 rather than the intended(?) libbrotlienc.so.1.
Consider removing LIBBROTLI{DEC,ENC}_VERSION_INFO, or using the variable in ..._la_LDFLAGS.

Makefile:305: ../brotli/c/tools/.deps/brotli-brotli.Po: No such file or directory

I have try to compile libbrotli on Centos 6, but it's getting error.

[root@centos-vhost libbrotli]# make
make all-recursive
make[1]: Entering directory /opt/libbrotli' Making all in . make[2]: Entering directory /opt/libbrotli'
CC brotli/c/dec/libbrotlidec_la-bit_reader.lo
CC brotli/c/dec/libbrotlidec_la-decode.lo
CC brotli/c/dec/libbrotlidec_la-huffman.lo
CC brotli/c/dec/libbrotlidec_la-state.lo
CC brotli/c/common/libbrotlidec_la-dictionary.lo
CCLD libbrotlidec.la
CC brotli/c/enc/libbrotlienc_la-backward_references.lo
CC brotli/c/enc/libbrotlienc_la-histogram.lo
CC brotli/c/enc/libbrotlienc_la-block_splitter.lo
CC brotli/c/enc/libbrotlienc_la-literal_cost.lo
CC brotli/c/enc/libbrotlienc_la-brotli_bit_stream.lo
CC brotli/c/enc/libbrotlienc_la-metablock.lo
CC brotli/c/enc/libbrotlienc_la-encode.lo
CC brotli/c/enc/libbrotlienc_la-static_dict.lo
CC brotli/c/enc/libbrotlienc_la-entropy_encode.lo
CC brotli/c/common/libbrotlienc_la-dictionary.lo
CC brotli/c/enc/libbrotlienc_la-utf8_util.lo
CC brotli/c/enc/libbrotlienc_la-compress_fragment.lo
CC brotli/c/enc/libbrotlienc_la-compress_fragment_two_pass.lo
CC brotli/c/enc/libbrotlienc_la-memory.lo
CC brotli/c/enc/libbrotlienc_la-backward_references_hq.lo
CC brotli/c/enc/libbrotlienc_la-bit_cost.lo
CC brotli/c/enc/libbrotlienc_la-dictionary_hash.lo
CC brotli/c/enc/libbrotlienc_la-cluster.lo
CCLD libbrotlienc.la
make[2]: Leaving directory /opt/libbrotli' Making all in tools make[2]: Entering directory /opt/libbrotli/tools'
Makefile:305: ../brotli/c/tools/.deps/brotli-brotli.Po: No such file or directory
make[2]: *** No rule to make target ../brotli/c/tools/.deps/brotli-brotli.Po'. Stop. make[2]: Leaving directory /opt/libbrotli/tools'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/opt/libbrotli'
make: *** [all] Error 2
[root@centos-vhost libbrotli]# locate brotli-brotli.Po
/opt/libbrotli/tools/$(top_srcdir)/brotli/c/tools/.deps/brotli-brotli.Po
[root@centos-vhost libbrotli]#

Do you have any idea about this issue?

libtool: warning: undefined symbols not allowed in i686-w64-mingw32 shared libraries; building static only

Try to compile libbrotli in windows with msys2 and mingw32
Then get the before during make, and no dlls is created.

Seems http://stackoverflow.com/a/22884787 is an workaround for me, but not sure if this should be added OS-independently.

Permissions erros on Debian

I'm trying this out on Debian Jessie and getting silly permissions errors on the final step of make install:

/bin/mkdir -p '/usr/local/lib'
 /bin/bash ./libtool   --mode=install /usr/bin/install -c   libbrotli.la '/usr/local/lib'
libtool: install: /usr/bin/install -c .libs/libbrotli.so.0.0.0 /usr/local/lib/libbrotli.so.0.0.0
/usr/bin/install: cannot create regular file '/usr/local/lib/libbrotli.so.0.0.0': Permission denied

/usr/local/lib/libbrotlienc.so：‘log2’ undefined reference

The full code is as follows：
root@ubuntu:/home/eww0216/working_dir/picotls# make
[ 4%] Built target picotls-core
[ 29%] Built target picotls-minicrypto
[ 33%] Built target test-fusion.t
[ 35%] Built target picotls-fusion
[ 64%] Built target test-openssl.t
[ 90%] Built target test-minicrypto.t
[ 92%] Built target picotls-openssl
[ 93%] Linking C executable cli
/usr/local/lib/libbrotlienc.so：‘log2’ undefined reference
collect2: error: ld returned 1 exit status
CMakeFiles/cli.dir/build.make:125: recipe for target 'cli' failed
make[2]: *** [cli] Error 1
CMakeFiles/Makefile2:363: recipe for target 'CMakeFiles/cli.dir/all' failed
make[1]: *** [CMakeFiles/cli.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

can anyone help me why I am getting this error.Any help would be appreciated..

Compiling error

Hey There,
After installing all dependency I got this error:
bruteforce:~/nginx/libbrotli# make make all-am make[1]: Entering directory '/root/nginx/libbrotli' CXX brotli/enc/libbrotlienc_la-backward_references.lo ./libtool: line 1125: g++: command not found Makefile:767: recipe for target 'brotli/enc/libbrotlienc_la-backward_references.lo' failed make[1]: *** [brotli/enc/libbrotlienc_la-backward_references.lo] Error 1 make[1]: Leaving directory '/root/nginx/libbrotli' Makefile:481: recipe for target 'all' failed make: *** [all] Error 2

Dose somebody have a hint for me?

-MuhPirat

Missing symbols

Thanks for getting this rolling, I'm guessing you don't use OS X.

dyld: Symbol not found: _kBrotliDictionary
  Referenced from: /usr/local/lib/libbrotli.0.dylib
  Expected in: flat namespace
 in /usr/local/lib/libbrotli.0.dylib
[1]    54387 trace trap  ./main.native

Use brotli as a submodule

Using brotli as a git submodule would alleviate the need for manually creating a symlink, and would also allow pinning known good revisions in case something done upstream broke the build.

Autogen.sh failed on MinGw32

I am getting this error:

$ ./autogen.sh
./autogen.sh: line 15: /c/Program: No such file or directory
./autogen.sh: line 16: /c/Program: No such file or directory
configure.ac:8: installing './install-sh'
configure.ac:8: installing './missing'
Makefile.am:5: error: Libtool library used but 'LIBTOOL' is undefined
Makefile.am:5:   The usual way to define 'LIBTOOL' is to add 'LT_INIT'
Makefile.am:5:   to 'configure.ac' and run 'aclocal' and 'autoconf' again.
Makefile.am:5:   If 'LT_INIT' is in 'configure.ac', make sure
Makefile.am:5:   its definition is in aclocal's search path.
Makefile.am: installing './depcomp'
/usr/local/share/automake-1.15/am/depend2.am: error: am__fastdepCC does not appear in AM_CONDITIONAL
/usr/local/share/automake-1.15/am/depend2.am:   The usual way to define 'am__fastdepCC' is to add 'AC_PROG_CC'
/usr/local/share/automake-1.15/am/depend2.am:   to 'configure.ac' and run 'aclocal' and 'autoconf' again
Makefile.am: error: C source seen but 'CC' is undefined
Makefile.am:   The usual way to define 'CC' is to add 'AC_PROG_CC'
Makefile.am:   to 'configure.ac' and run 'autoconf' again.
autoreconf: automake failed with exit status: 1

stream encode library support

I see that the new version of the broli has support for stream encoding. What is the plan to have a library for stream encoding?

Building on Debian

I'm not a GCC expert. Nothing in the ./configure step popped out as wrong, but I'm getting this error and have been for a couple of weeks. ngx_brotli requires libbrotli as a dependency.

make  all-am
make[1]: Entering directory '/opt/libbrotli'
  CC       brotli/dec/libbrotlidec_la-bit_reader.lo
  CC       brotli/dec/libbrotlidec_la-decode.lo
  CC       brotli/dec/libbrotlidec_la-huffman.lo
  CC       brotli/dec/libbrotlidec_la-state.lo
make[1]: *** No rule to make target 'brotli/enc/backward_references.cc', needed by 'brotli/enc/libbrotlienc_la-backward_references.lo'.  Stop.
make[1]: *** Waiting for unfinished jobs....
  CC       brotli/common/libbrotlidec_la-dictionary.lo
make[1]: Leaving directory '/opt/libbrotli'
Makefile:509: recipe for target 'all' failed
make: *** [all] Error 2
  CCLD     libbrotlidec.la
ar: `u' modifier ignored since `D' is the default (see `U')
make: *** No rule to make target 'brotli/enc/backward_references.cc', needed by 'brotli/enc/libbrotlienc_la-backward_references.lo'.  Stop.

Decoder / encoder in one or two libs?

(continued from here)

Thinking about it one step further, It might perhaps make sense to keep two separate ones. That makes it smaller for the ones who need only decoding.

(In addition to the fact that the encoder is C++ while the decoder is plain C...)

Separate encoder and decoder

As Daniel already suggested somewhere (couldn't find it again), we should have two libraries. One for encoding and one for decoding. Because...

1. encoder is C++, decoder is C (would like to see encoder in C as well, sigh)
2. the library is quite large, splitting it up would decrease load time for some apps

And we should than have one tool for decoding and one for encoding.

Brotli 0.4.0 released

Brotli 0.4.0 is released.
Please update libbrotli with the latest upstream release, because it has exciting new features and options.
The "Adoption by brotli upstream?" issue ,is still open, hence my question.

Encoder files missing

I copied your list of encoder/decoder files for brotlipy. However, when I came to start wrapping the encoder in a C API I discovered some weird linkage errors. This turned out to be because there were some files in my checkout that you didn't have in your list of C++ files for the encoder. I suspect if you tried to load the encoder library you'd find that the library load would fail because of missing symbols.

I was specifically missing brotli/enc/dictionary.cc and brotli/enc/utf8_util.cc.

Building on Linux

From HEAD trying to build on Debian Jessie,

 CC       brotli/dec/libbrotlidec_la-bit_reader.lo
./libtool: 1566: ./libtool: preserve_args+= --silent: not found
./libtool: 1578: ./libtool: preserve_args+= --tag CC: not found
./libtool: 1: eval: base_compile+= gcc: not found
./libtool: 1: eval: base_compile+= -DHAVE_CONFIG_H: not found
./libtool: 1: eval: base_compile+= -g: not found
./libtool: 1: eval: base_compile+= -O2: not found
./libtool: 1: eval: base_compile+= -MT: not found
./libtool: 1: eval: base_compile+= brotli/dec/libbrotlidec_la-bit_reader.lo: not found
./libtool: 1: eval: base_compile+= -MD: not found
./libtool: 1: eval: base_compile+= -MP: not found
./libtool: 1: eval: base_compile+= -MF: not found
./libtool: 1: eval: base_compile+= brotli/dec/.deps/libbrotlidec_la-bit_reader.Tpo: not found
./libtool: 1: eval: base_compile+= -c: not found
libtool: compile: you must specify a compilation command
libtool: compile: Try `libtool --help --mode=compile' for more information.
Makefile:687: recipe for target 'brotli/dec/libbrotlidec_la-bit_reader.lo' failed
make[2]: *** [brotli/dec/libbrotlidec_la-bit_reader.lo] Error 1