esiegerman / lz4

Extremely Fast Compression algorithm

LZ4 - Extremely fast compression

LZ4 is lossless compression algorithm, providing compression speed at 400 MB/s per core, scalable with multi-cores CPU. It features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.

Speed can be tuned dynamically, selecting an "acceleration" factor which trades compression ratio for more speed up. On the other end, a high compression derivative, LZ4_HC, is also provided, trading CPU time for improved compression ratio. All versions feature the same decompression speed.

LZ4 library is provided as open-source software using BSD 2-Clause license.

Branch	Status
master
dev

Branch Policy:

• The "master" branch is considered stable, at all times.
• The "dev" branch is the one where all contributions must be merged before being promoted to master.
  • If you plan to propose a patch, please commit into the "dev" branch, or its own feature branch. Direct commit to "master" are not permitted.

Benchmarks

The benchmark uses lzbench, from @inikep compiled with GCC v6.2.0 on Linux 64-bits. The reference system uses a Core i7-3930K CPU @ 4.5GHz. Benchmark evaluates the compression of reference Silesia Corpus in single-thread mode.

Compressor	Ratio	Compression	Decompression
memcpy	1.000	7300 MB/s	7300 MB/s
LZ4 fast 8 (v1.7.3)	1.799	911 MB/s	3360 MB/s
LZ4 default (v1.7.3)	2.101	625 MB/s	3220 MB/s
LZO 2.09	2.108	620 MB/s	845 MB/s
QuickLZ 1.5.0	2.238	510 MB/s	600 MB/s
Snappy 1.1.3	2.091	450 MB/s	1550 MB/s
LZF v3.6	2.073	365 MB/s	820 MB/s
Zstandard 1.1.1 -1	2.876	330 MB/s	930 MB/s
Zstandard 1.1.1 -3	3.164	200 MB/s	810 MB/s
zlib deflate 1.2.8 -1	2.730	100 MB/s	370 MB/s
LZ4 HC -9 (v1.7.3)	2.720	34 MB/s	3240 MB/s
zlib deflate 1.2.8 -6	3.099	33 MB/s	390 MB/s

LZ4 is also compatible and well optimized for x32 mode, for which it provides +10% speed performance.

Documentation

The raw LZ4 block compression format is detailed within lz4_Block_format.

To compress an arbitrarily long file or data stream, multiple blocks are required. Organizing these blocks and providing a common header format to handle their content is the purpose of the Frame format, defined into lz4_Frame_format. Interoperable versions of LZ4 must respect this frame format.

Other source versions

Beyond the C reference source, many contributors have created versions of lz4 in multiple languages (Java, C#, Python, Perl, Ruby, etc.). A list of known source ports is maintained on the LZ4 Homepage.