python-lz4 / python-lz4

LZ4 bindings for Python

python-lz4

Status

Introduction

This package provides python bindings for the LZ4 compression library.

The production ready bindings provided in this package cover the frame format, and the block format specifications. The frame format bindings are the recommended ones to use, as this guarantees interoperability with other implementations and language bindings.

Experimental bindings for the the streaming format specification are also included, but further work on those is required.

The API provided by the frame format bindings follows that of the LZMA, zlib, gzip and bzip2 compression libraries which are provided with the Python standard library. As such, these LZ4 bindings should provide a drop-in alternative to the compression libraries shipped with Python. The package provides context managers and file handler support.

The bindings drop the GIL when calling in to the underlying LZ4 library, and is thread safe. An extensive test suite is included.

Documentation

Full documentation is included with the project. The documentation is generated using Sphinx. Documentation is also hosted on readthedocs.

master

http://python-lz4.readthedocs.io/en/stable/

development

http://python-lz4.readthedocs.io/en/latest/

Homepage

The project homepage is hosted on Github. Please report any issues you find using the issue tracker.

Licensing

Code specific to this project is covered by the BSD 3-Clause License

python-lz4's Issues

s* instead of s# with PyArg_ParseTupleAndKeywords

In python-lz4 s# is used in e.g. the lz4.decompress method. I've got bytearray objects I'm decompression and I currently have to create an immutable bytes object out of them for compatibility with lz4. While I understand that lz4 will threat the bytes to be decompressed as immutable, accepting only immutable bytes objects forces a memory copy when the data is in a bytearray. Any posibility s# could be changed to s*?

Size is int32?

Running

python -c "import lz4.frame; lz4.frame.compress(b'a' * (2 ** 31))"

Will give me OverflowError: size does not fit in an int

Looks like it is using int32 to store size?

Allow unknown uncompressed data size in lz4.block.decompress()

In some cases you get lz4 compressed chunk of data without knowing uncompressed data size.

I suggest to allow uncompressed_size (parameter of lz4.block.decompress) accept zero value to indicate that data size is unknown.

Fails to build with Python: nightly

Hey there,

Just so that you are aware, setup.py fails for me with Python: nightly.
https://travis-ci.org/decibelcooper/proio/jobs/341279761#L447

Reflect frame.decompress() return change in doc

Hi @jonathanunderwood et al, thanks for the new release and the great work. With the new release I find that frame.decompress() now returns a tuple, with the byte string and an integer, but the documentation doesn't seem to reflect the change.

It would be nice for windows users if pre-compiled Wheels were published

Many Windows users don't have compilers installed, and if they do have some C++ compiler it is likely NOT to be the correct one for building C modules for their installed Python version.

Pre-built wheels would make this much easier for Windows users.

Provide with handler support

Support read and write via with file handler as gzip.open and BZ2File do. Eg.:

with lz4.open(filename, 'wb') as f:

0.9.0 failing: test_compress_begin_update_end_no_auto_flush_not_defaults_threaded

In Alpine Linux we have just started running unit tests for all packages. In python2 & python3 only one test is failing (with a segfault) for:

test_compress_begin_update_end_no_auto_flush_not_defaults_threaded

---

• Failing test log:

*** running python2 tests ***
running test
running egg_info
writing lz4.egg-info/PKG-INFO
writing top-level names to lz4.egg-info/top_level.txt
writing dependency_links to lz4.egg-info/dependency_links.txt
reading manifest file 'lz4.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'src/*.h'
warning: no previously-included files found matching 'tests/*.pyc'
writing manifest file 'lz4.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-2.7/lz4/_version.so -> lz4
copying build/lib.linux-x86_64-2.7/lz4/block/_block.so -> lz4/block
copying build/lib.linux-x86_64-2.7/lz4/frame/_frame.so -> lz4/frame
/usr/lib/python2.7/site-packages/nose/config.py:264: RuntimeWarning: Option 'with-coverage' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
/usr/lib/python2.7/site-packages/nose/config.py:264: RuntimeWarning: Option 'cover-package' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
nose.loader: DEBUG: load from . (None)
nose.loader: DEBUG: load from dir /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests
nose.loader: DEBUG: load from /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py (None)
nose.loader: DEBUG: Load from module <module 'test_block' from '/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py'>
test_block_format (test_block.TestLZ4Block) ... ok
test_random (test_block.TestLZ4Block) ... ok
test_random_fast1 (test_block.TestLZ4Block) ... ok
test_random_fast2 (test_block.TestLZ4Block) ... ok
test_random_fast3 (test_block.TestLZ4Block) ... ok
test_random_fast_no_store_size (test_block.TestLZ4Block) ... ok
test_random_hc1 (test_block.TestLZ4Block) ... ok
test_random_hc2 (test_block.TestLZ4Block) ... ok
test_random_hc3 (test_block.TestLZ4Block) ... ok
test_random_hc_no_store_size (test_block.TestLZ4Block) ... ok
test_random_no_store_size (test_block.TestLZ4Block) ... ok
test_threads (test_block.TestLZ4Block) ... ok
test_threads_fast1 (test_block.TestLZ4Block) ... ok
test_threads_fast2 (test_block.TestLZ4Block) ... ok
test_threads_fast3 (test_block.TestLZ4Block) ... ok
test_threads_fast_no_store_size (test_block.TestLZ4Block) ... ok
test_threads_hc1 (test_block.TestLZ4Block) ... ok
test_threads_hc2 (test_block.TestLZ4Block) ... ok
test_threads_hc3 (test_block.TestLZ4Block) ... ok
test_threads_hc_no_store_size (test_block.TestLZ4Block) ... ok
test_decompress_truncated (test_block.TestLZ4BlockModern) ... ok
test_decompress_ui32_overflow (test_block.TestLZ4BlockModern) ... ok
test_decompress_with_trailer (test_block.TestLZ4BlockModern) ... ok
test_decompress_without_leak (test_block.TestLZ4BlockModern) ... ok
nose.loader: DEBUG: load from /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_frame.py (None)
nose.loader: DEBUG: Load from module <module 'test_frame' from '/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_frame.py'>
test_LZ4FrameCompressor (test_frame.TestLZ4Frame) ... ok
test_LZ4FrameCompressor_reset (test_frame.TestLZ4Frame) ... ok
test_compress (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_2 (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_not_defaults (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_not_defaults_threaded (test_frame.TestLZ4Frame) ... Segmentation fault
*** running python3 tests ***
running test
Searching for nose>=1.0
Reading https://pypi.python.org/simple/nose/
Downloading https://pypi.python.org/packages/58/a5/0dc93c3ec33f4e281849523a5a913fa1eea9a3068acfa754d44d88107a44/nose-1.3.7.tar.gz#md5=4d3ad0ff07b61373d2cefc89c5d0b20b
Best match: nose 1.3.7
Processing nose-1.3.7.tar.gz
Writing /tmp/easy_install-wme5wdjf/nose-1.3.7/setup.cfg
Running nose-1.3.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install-wme5wdjf/nose-1.3.7/egg-dist-tmp-2jabmaz8
no previously-included directories found matching 'doc/.build'
creating /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py3.6.egg
Extracting nose-1.3.7-py3.6.egg to /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs

Installed /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py3.6.egg
running egg_info
writing lz4.egg-info/PKG-INFO
writing dependency_links to lz4.egg-info/dependency_links.txt
writing top-level names to lz4.egg-info/top_level.txt
reading manifest file 'lz4.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'src/*.h'
warning: no previously-included files found matching 'tests/*.pyc'
writing manifest file 'lz4.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-3.6/lz4/_version.cpython-36m-x86_64-linux-gnu.so -> lz4
copying build/lib.linux-x86_64-3.6/lz4/block/_block.cpython-36m-x86_64-linux-gnu.so -> lz4/block
copying build/lib.linux-x86_64-3.6/lz4/frame/_frame.cpython-36m-x86_64-linux-gnu.so -> lz4/frame
/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py3.6.egg/nose/config.py:264: RuntimeWarning: Option 'with-coverage' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py3.6.egg/nose/config.py:264: RuntimeWarning: Option 'cover-package' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
nose.loader: DEBUG: load from . (None)
nose.loader: DEBUG: load from dir /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests
nose.loader: DEBUG: load from /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py (None)
/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py:196: DeprecationWarning: invalid escape sequence \d
  with self.assertRaisesRegexp(ValueError, '^(Input source data size too small|Corrupt input at byte \d+|Decompressor wrote \d+ bytes, but \d+ bytes expected from header)'):
nose.loader: DEBUG: Load from module <module 'test_block' from '/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py'>
test_block_format (test_block.TestLZ4Block) ... ok
test_random (test_block.TestLZ4Block) ... ok
test_random_fast1 (test_block.TestLZ4Block) ... ok
test_random_fast2 (test_block.TestLZ4Block) ... ok
test_random_fast3 (test_block.TestLZ4Block) ... ok
test_random_fast_no_store_size (test_block.TestLZ4Block) ... ok
test_random_hc1 (test_block.TestLZ4Block) ... ok
test_random_hc2 (test_block.TestLZ4Block) ... ok
test_random_hc3 (test_block.TestLZ4Block) ... ok
test_random_hc_no_store_size (test_block.TestLZ4Block) ... ok
test_random_no_store_size (test_block.TestLZ4Block) ... ok
test_threads (test_block.TestLZ4Block) ... ok
test_threads_fast1 (test_block.TestLZ4Block) ... ok
test_threads_fast2 (test_block.TestLZ4Block) ... ok
test_threads_fast3 (test_block.TestLZ4Block) ... ok
test_threads_fast_no_store_size (test_block.TestLZ4Block) ... ok
test_threads_hc1 (test_block.TestLZ4Block) ... ok
test_threads_hc2 (test_block.TestLZ4Block) ... ok
test_threads_hc3 (test_block.TestLZ4Block) ... ok
test_threads_hc_no_store_size (test_block.TestLZ4Block) ... ok
test_decompress_truncated (test_block.TestLZ4BlockModern) ... /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py:196: DeprecationWarning: Please use assertRaisesRegex instead.
  with self.assertRaisesRegexp(ValueError, '^(Input source data size too small|Corrupt input at byte \d+|Decompressor wrote \d+ bytes, but \d+ bytes expected from header)'):
ok
test_decompress_ui32_overflow (test_block.TestLZ4BlockModern) ... ok
test_decompress_with_trailer (test_block.TestLZ4BlockModern) ... ok
test_decompress_without_leak (test_block.TestLZ4BlockModern) ... ok
nose.loader: DEBUG: load from /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_frame.py (None)
nose.loader: DEBUG: Load from module <module 'test_frame' from '/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_frame.py'>
test_LZ4FrameCompressor (test_frame.TestLZ4Frame) ... ok
test_LZ4FrameCompressor_reset (test_frame.TestLZ4Frame) ... ok
test_compress (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_2 (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_not_defaults (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_not_defaults_threaded (test_frame.TestLZ4Frame) ... Segmentation fault

---

• Test Log without test_compress_begin_update_end_no_auto_flush_not_defaults_threaded:

*** running python2 tests ***
running test
Searching for nose>=1.0
Reading https://pypi.python.org/simple/nose/
Downloading https://pypi.python.org/packages/58/a5/0dc93c3ec33f4e281849523a5a913fa1eea9a3068acfa754d44d88107a44/nose-1.3.7.tar.gz#md5=4d3ad0ff07b61373d2cefc89c5d0b20b
Best match: nose 1.3.7
Processing nose-1.3.7.tar.gz
Writing /tmp/easy_install-FPsAiB/nose-1.3.7/setup.cfg
Running nose-1.3.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install-FPsAiB/nose-1.3.7/egg-dist-tmp-BDyS4C
no previously-included directories found matching 'doc/.build'
creating /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py2.7.egg
Extracting nose-1.3.7-py2.7.egg to /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs

Installed /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py2.7.egg
running egg_info
writing lz4.egg-info/PKG-INFO
writing top-level names to lz4.egg-info/top_level.txt
writing dependency_links to lz4.egg-info/dependency_links.txt
reading manifest file 'lz4.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'src/*.h'
warning: no previously-included files found matching 'tests/*.pyc'
writing manifest file 'lz4.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-2.7/lz4/_version.so -> lz4
copying build/lib.linux-x86_64-2.7/lz4/block/_block.so -> lz4/block
copying build/lib.linux-x86_64-2.7/lz4/frame/_frame.so -> lz4/frame
/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py2.7.egg/nose/config.py:264: RuntimeWarning: Option 'with-coverage' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py2.7.egg/nose/config.py:264: RuntimeWarning: Option 'cover-package' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
nose.loader: DEBUG: load from . (None)
nose.loader: DEBUG: load from dir /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests
nose.loader: DEBUG: load from /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py (None)
nose.loader: DEBUG: Load from module <module 'test_block' from '/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py'>
test_block_format (test_block.TestLZ4Block) ... ok
test_random (test_block.TestLZ4Block) ... ok
test_random_fast1 (test_block.TestLZ4Block) ... ok
test_random_fast2 (test_block.TestLZ4Block) ... ok
test_random_fast3 (test_block.TestLZ4Block) ... ok
test_random_fast_no_store_size (test_block.TestLZ4Block) ... ok
test_random_hc1 (test_block.TestLZ4Block) ... ok
test_random_hc2 (test_block.TestLZ4Block) ... ok
test_random_hc3 (test_block.TestLZ4Block) ... ok
test_random_hc_no_store_size (test_block.TestLZ4Block) ... ok
test_random_no_store_size (test_block.TestLZ4Block) ... ok
test_threads (test_block.TestLZ4Block) ... ok
test_threads_fast1 (test_block.TestLZ4Block) ... ok
test_threads_fast2 (test_block.TestLZ4Block) ... ok
test_threads_fast3 (test_block.TestLZ4Block) ... ok
test_threads_fast_no_store_size (test_block.TestLZ4Block) ... ok
test_threads_hc1 (test_block.TestLZ4Block) ... ok
test_threads_hc2 (test_block.TestLZ4Block) ... ok
test_threads_hc3 (test_block.TestLZ4Block) ... ok
test_threads_hc_no_store_size (test_block.TestLZ4Block) ... ok
test_decompress_truncated (test_block.TestLZ4BlockModern) ... ok
test_decompress_ui32_overflow (test_block.TestLZ4BlockModern) ... ok
test_decompress_with_trailer (test_block.TestLZ4BlockModern) ... ok
test_decompress_without_leak (test_block.TestLZ4BlockModern) ... ok
nose.loader: DEBUG: load from /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_frame.py (None)
nose.loader: DEBUG: Load from module <module 'test_frame' from '/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_frame.py'>
test_LZ4FrameCompressor (test_frame.TestLZ4Frame) ... ok
test_LZ4FrameCompressor_reset (test_frame.TestLZ4Frame) ... ok
test_compress (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_2 (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_not_defaults (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_not_defaults (test_frame.TestLZ4Frame) ... ok
test_compress_huge_with_size (test_frame.TestLZ4Frame) ... ok
test_compress_huge_without_size (test_frame.TestLZ4Frame) ... ok
test_compress_not_defaults_1 (test_frame.TestLZ4Frame) ... ok
test_compress_not_defaults_2 (test_frame.TestLZ4Frame) ... ok
test_compress_not_defaults_3 (test_frame.TestLZ4Frame) ... ok
test_compress_not_defaults_4 (test_frame.TestLZ4Frame) ... ok
test_create_and_free_compression_context (test_frame.TestLZ4Frame) ... ok
test_get_frame_info (test_frame.TestLZ4Frame) ... ok
test_threads (test_frame.TestLZ4Frame) ... ok
test_LZ4FrameCompressor_fails (test_frame.TestLZ4FrameModern) ... ok
test_checksum_failure (test_frame.TestLZ4FrameModern) ... ok
test_decompress_trailer (test_frame.TestLZ4FrameModern) ... ok
test_decompress_truncated (test_frame.TestLZ4FrameModern) ... ok

----------------------------------------------------------------------
Ran 45 tests in 7.541s

OK
*** running python3 tests ***
running test
Searching for nose>=1.0
Reading https://pypi.python.org/simple/nose/
Downloading https://pypi.python.org/packages/58/a5/0dc93c3ec33f4e281849523a5a913fa1eea9a3068acfa754d44d88107a44/nose-1.3.7.tar.gz#md5=4d3ad0ff07b61373d2cefc89c5d0b20b
Best match: nose 1.3.7
Processing nose-1.3.7.tar.gz
Writing /tmp/easy_install-ietl2k7c/nose-1.3.7/setup.cfg
Running nose-1.3.7/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ietl2k7c/nose-1.3.7/egg-dist-tmp-gdyx_mdr
no previously-included directories found matching 'doc/.build'
creating /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py3.6.egg
Extracting nose-1.3.7-py3.6.egg to /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs

Installed /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py3.6.egg
running egg_info
writing lz4.egg-info/PKG-INFO
writing dependency_links to lz4.egg-info/dependency_links.txt
writing top-level names to lz4.egg-info/top_level.txt
reading manifest file 'lz4.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'src/*.h'
warning: no previously-included files found matching 'tests/*.pyc'
writing manifest file 'lz4.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-3.6/lz4/_version.cpython-36m-x86_64-linux-gnu.so -> lz4
copying build/lib.linux-x86_64-3.6/lz4/block/_block.cpython-36m-x86_64-linux-gnu.so -> lz4/block
copying build/lib.linux-x86_64-3.6/lz4/frame/_frame.cpython-36m-x86_64-linux-gnu.so -> lz4/frame
/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py3.6.egg/nose/config.py:264: RuntimeWarning: Option 'with-coverage' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/.eggs/nose-1.3.7-py3.6.egg/nose/config.py:264: RuntimeWarning: Option 'cover-package' in config file 'setup.cfg' ignored: excluded by runtime environment
  warn(msg, RuntimeWarning)
nose.loader: DEBUG: load from . (None)
nose.loader: DEBUG: load from dir /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests
nose.loader: DEBUG: load from /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py (None)
/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py:196: DeprecationWarning: invalid escape sequence \d
  with self.assertRaisesRegexp(ValueError, '^(Input source data size too small|Corrupt input at byte \d+|Decompressor wrote \d+ bytes, but \d+ bytes expected from header)'):
nose.loader: DEBUG: Load from module <module 'test_block' from '/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py'>
test_block_format (test_block.TestLZ4Block) ... ok
test_random (test_block.TestLZ4Block) ... ok
test_random_fast1 (test_block.TestLZ4Block) ... ok
test_random_fast2 (test_block.TestLZ4Block) ... ok
test_random_fast3 (test_block.TestLZ4Block) ... ok
test_random_fast_no_store_size (test_block.TestLZ4Block) ... ok
test_random_hc1 (test_block.TestLZ4Block) ... ok
test_random_hc2 (test_block.TestLZ4Block) ... ok
test_random_hc3 (test_block.TestLZ4Block) ... ok
test_random_hc_no_store_size (test_block.TestLZ4Block) ... ok
test_random_no_store_size (test_block.TestLZ4Block) ... ok
test_threads (test_block.TestLZ4Block) ... ok
test_threads_fast1 (test_block.TestLZ4Block) ... ok
test_threads_fast2 (test_block.TestLZ4Block) ... ok
test_threads_fast3 (test_block.TestLZ4Block) ... ok
test_threads_fast_no_store_size (test_block.TestLZ4Block) ... ok
test_threads_hc1 (test_block.TestLZ4Block) ... ok
test_threads_hc2 (test_block.TestLZ4Block) ... ok
test_threads_hc3 (test_block.TestLZ4Block) ... ok
test_threads_hc_no_store_size (test_block.TestLZ4Block) ... ok
test_decompress_truncated (test_block.TestLZ4BlockModern) ... /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_block.py:196: DeprecationWarning: Please use assertRaisesRegex instead.
  with self.assertRaisesRegexp(ValueError, '^(Input source data size too small|Corrupt input at byte \d+|Decompressor wrote \d+ bytes, but \d+ bytes expected from header)'):
ok
test_decompress_ui32_overflow (test_block.TestLZ4BlockModern) ... ok
test_decompress_with_trailer (test_block.TestLZ4BlockModern) ... ok
test_decompress_without_leak (test_block.TestLZ4BlockModern) ... ok
nose.loader: DEBUG: load from /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_frame.py (None)
nose.loader: DEBUG: Load from module <module 'test_frame' from '/home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_frame.py'>
test_LZ4FrameCompressor (test_frame.TestLZ4Frame) ... ok
test_LZ4FrameCompressor_reset (test_frame.TestLZ4Frame) ... ok
test_compress (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_2 (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_no_auto_flush_not_defaults (test_frame.TestLZ4Frame) ... ok
test_compress_begin_update_end_not_defaults (test_frame.TestLZ4Frame) ... ok
test_compress_huge_with_size (test_frame.TestLZ4Frame) ... ok
test_compress_huge_without_size (test_frame.TestLZ4Frame) ... ok
test_compress_not_defaults_1 (test_frame.TestLZ4Frame) ... ok
test_compress_not_defaults_2 (test_frame.TestLZ4Frame) ... ok
test_compress_not_defaults_3 (test_frame.TestLZ4Frame) ... ok
test_compress_not_defaults_4 (test_frame.TestLZ4Frame) ... ok
test_create_and_free_compression_context (test_frame.TestLZ4Frame) ... ok
test_get_frame_info (test_frame.TestLZ4Frame) ... ok
test_threads (test_frame.TestLZ4Frame) ... ok
test_LZ4FrameCompressor_fails (test_frame.TestLZ4FrameModern) ... /home/stuart/aports/community/py-lz4/src/lz4-0.9.0/tests/test_frame.py:270: DeprecationWarning: Please use assertRaisesRegex instead.
  with self.assertRaisesRegexp(RuntimeError, r'compress called after flush'):
ok
test_checksum_failure (test_frame.TestLZ4FrameModern) ... ok
test_decompress_trailer (test_frame.TestLZ4FrameModern) ... ok
test_decompress_truncated (test_frame.TestLZ4FrameModern) ... ok

----------------------------------------------------------------------
Ran 45 tests in 7.807s

---

Perhaps this error is due to Alpine using musl libc (not glibc) - if I can provide any other information to help debug this issue please let me know.

Import modules from a library.tar.lz4

This might be beyond the scope of this project, but Is it possible to use this to import from a library.tar.lz4 like you can using pythons zipimporter?

With standard python you can create a zip file (ex. library.zip) with your python packages, add the zip file to the sys.path, and then import directly from the zip file (cx_Freeze uses this for distributed apps) .

I think this would be really useful for the python-for-android project. Currently p4a apps don't use zip import because importing from a .zip on a phone is slow (due to slow decompression speed). Using lz4 might be a good alternative.

Thanks!

(De)compressor context managers potentially incomplete

Hi Jonathan / @jgubmll / @jonathanunderwood, congratulations on frame support & reaching production status!

We provide context manager classes LZ$FrameCompressor and LZ4FrameDecompressor,

Judging by what's on top of master I think you might be missing something there since both __enter__ and __exit__ methods are empty which is a No-Op and basically means the following two do exactly the same thing:

with lz4.frame.LZ4FrameCompressor() as compressor:
   # do stuff with compressor
   pass
# At this point compressor is still in scope so it's no like
# the context has caused it to be GC'ed.

and

compressor = lz4.frame.LZ4FrameCompressor()
# do stuff with compressor

You also mention:

The compression context is created with an appropriate destructor, so no need to del it here

Maybe I'm missing something - where is the appropriate destructor? (I see e.g. no __del__ method.)

lz4 doesn't handle larger file sizes: stream protocol implementation needed

lz4 doesn't handle larger file sizes

$ pip list | grep lz4
lz4 (0.8.2)

$ python -c "import lz4; lz4.compress(b'a' * (2 ** 31))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OverflowError: size does not fit in an int

memory leak in lz4.frame.decompress

> python --version
Python 3.6.3
> pip freeze | fgrep lz4
lz4==0.18.1

I'm encountering an issue where when I run the script below, I can see the memory increasing. At the end, the memory using is ~4GB looking at htop.

import lz4.frame
import tracemalloc
import time


tracemalloc.start()

input_data = 'a' * 1024 * 1024
compressed = lz4.frame.compress(input_data.encode('utf-8'))

prev_snapshot = None

for i in range(5000):
    decompressed = lz4.frame.decompress(compressed)

    if i % 100 == 0:
        snapshot = tracemalloc.take_snapshot()
        top_stats = snapshot.statistics('lineno')

        print(" top: {}".format(top_stats[0]))

        if prev_snapshot:
            top_stats = snapshot.compare_to(prev_snapshot, 'lineno')
            print("diff: {}".format(top_stats[0]))

        print()
        time.sleep(0.5)

        prev_snapshot = snapshot

print('done')
time.sleep(10)

 top: mem-look.py:14: size=101 MiB, count=101, average=1024 KiB
diff: mem-look.py:14: size=101 MiB (+100 MiB), count=101 (+100), average=1024 KiB
...
 top: mem-look.py:14: size=4901 MiB, count=4901, average=1024 KiB
diff: mem-look.py:14: size=4901 MiB (+100 MiB), count=4901 (+100), average=1024 KiB

compression failing with 0.9.2

Can you help me understand what changed from 0.9.1 to 0.9.2? We compress a lot of output from numpy, eg:

lz4.compressHC(np.packbits(np.ones(len(df), dtype='uint8')))

with previous versions of lz4 this was fine, and now in our python 3.5/3.6 environments this fails with the following exception:

ValueError: Incorrect type for source object

Memory leak in v0.19.1

import numpy as np
import glob

import lz4.frame

frames=glob.glob('frames_compressed/*.npy.lz4')



for frame in frames:
    print(frame)
    with lz4.frame.open(frame) as f1:
        data=np.load(f1)
        

this leads to huge memory usage (due to >50k files in the folder), the same code with lzma works with constant memory usage so I assume there is some leak in lz4

Path to 1.0

• Update quickstart and docs for file handler support, and updated lz4.frame.decompress_chunk API
• Add more tests for the frame bindings, including tests for exceptions, and decompression of input containing data after the frame
• Working with handler support for file reading and writing
• Check frame module constants for unneeded ones, and remove (we replaced settings with booleans where sensible).
• Implement block checksums for LZ4 >=1.8.0
• Add tests/fixtures for block checksums. Nb. tests with block_checksum=True will raise RuntimeError if LZ4 < 1.8.0, so need to take care of that, preferably in fixture.
• Provide binding for LZ4F_resetDecompressionContext()
• Add quickstart sections to documentation
• block - add missing error checking for resize operations
• block - use Py_ssize_t where appropriate
• block - improve/remove checking for maximum header size
• Remove deprecated functions
• Make LZ4VERSION attribute deprecated (or remove it altogether)
• Add function (as lazily evaulated @property) for returning the underlying lz4 semver
• Verify that we're complying with https://packaging.python.org/tutorials/distributing-packages/
• Make windows wheels available on PyPi (pull from Appveyor)
• Frame: API : from 1.7.5 on, LZ4F_compressBound(0) provides upper bound of *flush() and *End() (see issues #290, #280 of lz4). Make use of this - will need a conditional.
• Fix missing docstrings from the RTD docs - unclear why eg. lz4.frame.create_decompression_context doesn't show up. Other formatting screw-ups abound, too.
• Define a minimum supported LZ4 library version. If a system library is found that is older than the minimum required, build against the bundled version. This needs to be as lenient as possible.
• Add binding for LZ4F_flush(). Available from 1.8.0 onwards? Check. The LZ4FrameCompressor method flush will need renaming to end, and a new flush method adding.

Fix depreciated API use in tests/bench.py

lz4.uncompress -> lz4.block.decompress

lz4.compress -> lz4.block.compress

Otherwise stdout gets a line for each of the 200k loop iterations on each test. Didn't seem worth making a PR over, but annoying enough an upstream fix seemed appropriate.

Package fails to install on systems without `pkg-config`

On systems without pkg-config, pip install lz4 fails with the following output:

Collecting lz4 (from ***)
  Downloading lz4-0.8.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 20, in <module>
      File "/tmp/pip-build-D9kUKL/lz4/setup.py", line 19, in <module>
        liblz4_found = subprocess.call(cmd) == 0
      File "/usr/lib/python2.7/subprocess.py", line 522, in call
        return Popen(*popenargs, **kwargs).wait()
      File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
        errread, errwrite)
      File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
        raise child_exception
    OSError: [Errno 2] No such file or directory

I've investigated this a little bit, and it seems like the following chunk of code on the setup.py file isn't handling all exceptional cases correctly:

# Check to see if we have a lz4 library installed on the system and
# use it if so. If not, we'll use the bundled library. If lz4 is
# installed it will have a pkg-config file, so we'll use pkg-config to
# check for existence of the library.
pkg_config_exe = os.environ.get('PKG_CONFIG', None) or 'pkg-config'
cmd = '{0} --exists liblz4'.format(pkg_config_exe).split()
liblz4_found = subprocess.call(cmd) == 0

In this specific case, the pkg-config binary doesn't exist on the system, making subprocess.call() fail with an OSError exception instead of a non-zero return code, and this exception isn't being handled by the code, making the installation fail.

This can be pretty common on minimal installations. In this particular situation, I'm running pip inside an ubuntu docker container, and it seems like the default docker ubuntu image doesn't contain pkg-config by default:

$ sudo docker run --rm -it ubuntu bash
root@f8481c78cd7b:/# pkg-config
bash: pkg-config: command not found

In my case I think I'll try to solve this by installing the pkg-config package, but since this might trip other people I've also decide to open this issue.

Support for pypy

The current lz4 library crashes on pypy:

→ pypy
Python 2.7.10 (5f8302b8bf9f53056e40426f10c72151564e5b19, Nov 21 2015, 04:15:18)
[PyPy 4.0.1 with GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.76)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>> import lz4.frame
>>>> lz4.frame.compress(b'1234')
[1]    23731 segmentation fault  pypy

I don't know enough about pypy development right now to debug, so apologies for no PR (because this is a great library!).

pytest-runner a setup requirement ?

Hi,
Just wondering...
Why is pytest-runner a setup requirement ?
Shouldn't it be a test requirement ?

tests_require = [
    'pytest',
    'pytest-runner',
    'psutil',
],

instead of

setup_requires=[
        'setuptools_scm',
        'pkgconfig',
        'pytest-runner',
    ],

Cheers.
Elias.

Ask LZ4 project to reference this project, not the old project

Homepage lists bindings: http://lz4.github.io/lz4/

Windows build instructions

I understand how to build package "manually", but I want to know where to put lz4 and py3c so pip install lz4 can find it.

Cannot seem to read from node-lz4

Your library produces this when compressing 'a':

b'\x01\x00\x00\x00\x10a'

node-lz4 produces this when compressing 'a':

b'\x04"M\x18dp\xb9\x02\x00\x00\x00\x10a\x00\x00\x00\x00Vt\rU'

Neither library can decode the other's output. Is there any chance you could try to make these libraries compatible, since both are supposed to use the same compression?

See: pierrec/node-lz4#48

Use doctest to test code snippets in docs

Ensure all examples given work as expected.

Compile failure on MSVC

MS Visual Studio 2008 (which is used for the default python.org Python 2.7 builds) fails to compile lz4 with the following error:

src/python-lz4.c(131) : error C2143: syntax error : missing ';' before 'type'

This is because MSVC 2008 doesn't permit mixing declarations with code. The fix is simple, and a patch is forthcoming.

Update to lz4 v1.8.1.2

There's a new upstream release available. Seems to have some new features, and some new compression levels. Needs a bit of investigation.

fix building under mingw

--- a/setup.py	2017-03-16 13:54:10.038666000 +0700
+++ b/setup.py	2017-03-16 13:56:20.881360400 +0700
@@ -84,7 +84,7 @@
 
 if compiler == 'msvc':
     extra_compile_args = ['/Ot', '/Wall']
-elif compiler == 'unix':
+elif compiler in ('unix', 'mingw32'):
     if liblz4_found:
         extra_compile_args = []
     else:

Also the sys.exit in the "else" error case is failing with an import error because sys is not imported anywhere.

Empty string error with lz4.frame

lz4.frame fails with an empty string (py27, macOS, lz4 0.10.1)

In [32]: import lz4.frame

In [33]: lz4.frame.decompress(lz4.frame.compress(b''))
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-33-e1bd89fc259b> in <module>()
----> 1 lz4.frame.decompress(lz4.frame.compress(b''))

RuntimeError: LZ4F_getFrameInfo failed with code: ERROR_frameType_unknown

Integrate with test coverage service

It would be really helpful if we could integrate with a service which reports on test coverage (eg. Coveralls) as an aid to increasing and upkeep of test suite coverage of the codebase.

Update bundled lz4 to 1.7.5

https://github.com/lz4/lz4/releases

Add support for:

lz4hc : new high compression mode, by @inikep : levels 10-12 compress more (and slower), 12 is highest level

API : lz4frame : LZ4F_compressBound(0) provides upper bound of *flush() and *End() (#290, #280)

testing takes abnormally long with python 3.6, but is reasonable with 2.7

Hi,

in an attempt to package lz4 for the openSUSE project, we see very differing testing times between python 2.7 and 3.6 builds.

Here's an example:

[  105s] ============================= test session starts ==============================
[  105s] platform linux2 -- Python 2.7.15, pytest-3.6.3, py-1.5.3, pluggy-0.6.0
[  105s] rootdir: /home/abuild/rpmbuild/BUILD/lz4-2.0.2, inifile: setup.cfg
[  114s] collected 18893 items
[...] 
[  405s] ================== 18889 passed, 4 skipped in 300.56 seconds ===================

[  410s] ============================= test session starts ==============================
[  410s] platform linux -- Python 3.6.5, pytest-3.6.3, py-1.5.3, pluggy-0.6.0
[  410s] rootdir: /home/abuild/rpmbuild/BUILD/lz4-2.0.2, inifile: setup.cfg
[  418s] collected 18893 items
[...]
[ 9610s] ======================= 18893 passed in 9200.23 seconds ========================

Sure, we have quicker hosts in the build cluster, but 5 minutes versus 2 and a half hours is an abnormal deviation. It might be due to the 4 skipped tests, but having builds taking this long is painful. Of course, you can argue, that testing isn't necessary, but we're building for a varying number of different interpreters and real world distributions, where having tested all the builds is a good thing (tm).

Is that observed timing behaviour to be expected?

For reference, here's the build log, use the Download logfile link for a full build log.

Missing `lz4.dumps` in successful install

After install there is no lz4.dumps, or any other method. See below.

$ pip install -U lz4
Collecting lz4
Installing collected packages: lz4
Successfully installed lz4-1.1.0
$ python
Python 3.6.3 |Anaconda, Inc.| (default, Oct 13 2017, 12:02:49)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lz4
>>> lz4.dumps
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'lz4' has no attribute 'dumps'
>>> dir(lz4)
['VERSION', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_version', 'library_version_number', 'library_version_string', 'version']
>>> lz4.VERSION
'1.1.0'
>>> lz4.library_version_string()
'1.8.1'
>>> lz4.library_version_number()
10801
>>> 

I'm not sure how this can be a legitimate issue, but (and because!) I have tested this on two separate machines (Debian and CentOS), across different python versions (2.7 and 3.6). I have even tried installing older versions of the lz4 module (going back to 0.23.2). They all have this issue, which makes me think it is on my end, but... not sure what it is across multiple machines??

Update bundled lz4 to version 1.8.3

https://github.com/lz4/lz4/releases/tag/v1.8.3

latest version 0.9.0 does not allow compression of empty string

Previous versions of python-lz4 allowed compression of the empty string:

In [1]: lz4.compress('')
Out[1]: b'\x00\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x88/\xc9\x0b'

the latest version fails:

In [2]: lz4.compress('')
lz4/__init__.py:18: DeprecationWarning: Call to deprecated function or method compress (use lz4.block.compress instead).
  def compress(*args, **kwargs):
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last) in <module>()
----> 1 lz4.compress('')

lz4/deprecated.py in new_func(*args, **kwargs)
     37             warnings.warn_explicit(msg, category=DeprecationWarning, filename=filename, lineno=lineno)
     38             warnings.simplefilter('default', DeprecationWarning)  # reset filter
---> 39             return cls_or_func(*args, **kwargs)
     40 
     41         return new_func

lz4/__init__.py in compress(*args, **kwargs)
     17 @deprecated('use lz4.block.compress instead')
     18 def compress(*args, **kwargs):
---> 19     return lz4.block.compress(*args, **kwargs)
     20 
     21 @deprecated('use lz4.block.decompress instead')

ValueError: Input source data size invalid: 0 bytes

Was there a reason for this change? We use lz4 for compression of data we write to a database. Some of the records might contain empty strings. If I were to upgrade to the latest version of lz4 I'd no longer be able to support datasets with empty strings, and I'd also no longer be able to decompress the compressed data that's already been written.

Gracefully handle old versions of liblz4

Unable to pip install lz4 if pkgconfig python module is already installed and an old version of liblz4 is present. Details below.

Expected Behavior

pip install lz4 should assume liblz4_found = False when pkgconfig fails to find the library for any reason.

Current Behavior

$ pip install lz4==2.0.0
    100% |################################| 133kB 53.8MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "tmp/pip-build-pp4a_V/lz4/setup.py", line 22, in <module>
        liblz4_found = pkgconfig.installed('liblz4', LZ4_REQUIRED_VERSION)
      File "tmp/python-lz4/venv/lib/python2.7/site-packages/pkgconfig/pkgconfig.py", line 155, in installed
        raise ValueError(msg)
    ValueError: >= 1.7.5 is not a correct version specifier

Possible Solution

Improve error handling around pkgconfig.installed() call in setup.py.

diff --git a/setup.py b/setup.py
index 73a8e2a..2cd413a 100644
--- a/setup.py
+++ b/setup.py
@@ -21,8 +21,9 @@ else:
     try:
         liblz4_found = pkgconfig.installed('liblz4', LZ4_REQUIRED_VERSION)
         py3c_found = pkgconfig.installed('py3c', PY3C_REQUIRED_VERSION)
-    except EnvironmentError:
+    except Exception:
         # Windows, no pkg-config present
+        # or pkgconfig call failed
         pass

Maybe both liblz4_found and py3c_found should be set to False too.

Steps to Reproduce

1. pip install pkgconfig=1.3.1
2. <pkg-manager> install liblz4-dev
3. edit liblz4.pc file, replace version with Version: "1.5.r128"
4. attempt to pip install lz4==2.0.0

Context

Before ~ 2016 liblz4 used slightly unconventional versioning schema ...->125->126->127 see
https://github.com/lz4/lz4/blob/r128/lib/liblz4.pc.in

The library was repackaged internally using https://github.com/lz4/lz4/blob/r128/cmake_unofficial/CMakeLists.txt that constructs versions in the format of X.Y.r###, e.g. 1.5.r128 in the liblz4.pc file:

Name: lz4
Description: fast lossless compression algorithm library
URL: http://code.google.com/p/lz4/
Version:  "1.5.r128"
Libs: -L${libdir} -llz4
Cflags: -I${includedir}

Python pkgconfig module installed() call fails with invalid literal for int() with base 10: 'r128', because of the assumption it makes about the version format here

Add support for block decompression without a size hint

It would be useful to be able to decompress an lz4 block without knowing the decompressed size up front (i.e. not passed in by the user nor given in a uint32_t header). Should be doable by dynamically resizing as it goes. I haven't looked at the lz4 API enough to know if it's possible to realloc and resume, or if it needs to start over, but in either case this function would be a fallback and doesn't need to be super efficient.

It'd also be nice to have an initial size + growth factor exposed, but not strictly necessary.

I should mention that I already have workarounds for my use case so this is not high priority. Just wanted to file it here for consideration.

Add / fix support for text read mode

In [17]: lz4.frame.open(input_file, mode='rt', newline='', encoding='utf-8')
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-17-ca68d3c17eba> in <module>()
----> 1 lz4.frame.open(input_file, mode='rt', newline='', encoding='utf-8')

/usr/local/lib/python3.6/site-packages/lz4/frame/__init__.py in open(filename, mode, encoding, errors, newline, block_size, block_linked, compression_level, content_checksum, block_checksum, auto_flush, return_bytearray, source_size)
    842         block_checksum=block_checksum,
    843         auto_flush=auto_flush,
--> 844         return_bytearray=return_bytearray,
    845     )
    846

/usr/local/lib/python3.6/site-packages/lz4/frame/__init__.py in __init__(self, filename, mode, block_size, block_linked, compression_level, content_checksum, block_checksum, auto_flush, return_bytearray, source_size)
    513             self._pos = 0
    514         else:
--> 515             raise ValueError('Invalid mode: {!r}'.format(mode))
    516
    517         if sys.version_info > (3, 6):

ValueError: Invalid mode: 'rt'

Looking at the source, it looks like mode is passed from

python-lz4/lz4/frame/__init__.py

Line 837 in 603c112

mode=mode,

to

python-lz4/lz4/frame/__init__.py

Line 500 in 603c112

if mode in ('r', 'rb'):

open() is happy with a mode of 'rt' and it passes it along unchanged to the LZ4FrameFile class which does not like 'rt'. Perhaps the mode should be changed to 'rb' when passing to the init method, or perhaps the init method should be happy with 'rt'.

Either way it breaks.

Segfault on frame decompress

There is an issue in the code for the frame.
Here: https://github.com/python-lz4/python-lz4/blob/master/lz4/frame/_frame.c#L1087
You set to 1 the stableDst even if max_length is -1 (meaning that PyMem_Realloc can be called) which is incorrect.

Remove bundled py3c

• As py3c nears a 1.0 release, it is setting up to be available as a package on PyPi, at which point there's no need to bundle it.
• Once 1.0 is released, we can drop the Py_UNUSED stuff from _frame.c and _block.c

high_compression mode called compress_fast instead of lz4hc

#24

When calling lz4.block.compress(xxx, mode='high_compression'), it calls fast compress instead of HC

I have made a PR to fix it.

Test Failure under musl c in 2.0.0

Version 2.0.0 builds successfully under Alpine Linux (musl c) - but the tests fail at the end:

...............................................................................................................................................                     [ 95%]
tests/frame/test_frame_5.py F

================================================================================ FAILURES =================================================================================
_________________________________________________________________ test_frame_decompress_mem_usage[data0] __________________________________________________________________

data = b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'

    def test_frame_decompress_mem_usage(data):
        tracemalloc = pytest.importorskip('tracemalloc')
    
        tracemalloc.start()
    
        compressed = lz4.frame.compress(data)
        prev_snapshot = None
    
        for i in range(1000):
            decompressed = lz4.frame.decompress(compressed)  # noqa: F841
    
            if i % 100 == 0:
                snapshot = tracemalloc.take_snapshot()
    
                if prev_snapshot:
                    stats = snapshot.compare_to(prev_snapshot, 'lineno')
>                   assert stats[0].size_diff < (1024 * 4)
E                   AssertionError: assert 11264 < (1024 * 4)
E                    +  where 11264 = <StatisticDiff traceback=<Traceback (<Frame filename='/usr/lib/python3.6/tracemalloc.py' lineno=113>,)> size=11264 (+11264) count=128 (+128)>.size_diff

compressed = b'\x04"M\x18H@\x00\x00\x10\x00\x00\x00\x00\x00g\x0b\x01\x00\x00\x1fa\x01\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x...\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xe8Paaaaa\x00\x00\x00\x00'
data       = b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
decompressed = b'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
i          = 200
prev_snapshot = <tracemalloc.Snapshot object at 0x7f661e2d40f0>
snapshot   = <tracemalloc.Snapshot object at 0x7f661e2d43c8>
stats      = [<StatisticDiff traceback=<Traceback (<Frame filename='/usr/lib/python3.6/tracemalloc.py' lineno=113>,)> size=11264 (+...src/lz4-2.0.0/.eggs/pytest-3.6.1-py3.6.egg/_pytest/python.py' lineno=263>,)> size=842040 (-648) count=11695 (-9)>, ...]
tracemalloc = <module 'tracemalloc' from '/usr/lib/python3.6/tracemalloc.py'>

tests/frame/test_frame_5.py:35: AssertionError
=========================================================== 1 failed, 18108 passed, 1 skipped in 299.80 seconds ===========================================================
>>> ERROR: py-lz4*: check failed
>>> ERROR: py-lz4: all failed

Test on travis with Python 3.7

This is a tracker - at present Python 3.7 is not available on travis. See:

travis-ci/travis-ci#9815

LZ4_VERSION mismatch

1. According to 1a13ecc, LZ4_VERSION hardcoded in setup.py should be updated, or it will mislead people.
2. The setup.py should be improved. If the version of liblz4 found is lower than requirements, bundled LZ4 library should be used.

Upload source release from Travis rather than AppVeyor

At present the source release and Windows wheels are all uploaded from AppVeyor. The Appveyor builds are much slower though, so it would be better to publish the source packge from Travis.

Can't install from pip to a fresh virtualenv: No module named 'pkgconfig'

The lz4 package fails to install to a fresh virtualenv with the error. Here is a simple runnable script that demonstrates:

#!/bin/bash

rm -rf venv
python3 -m venv venv
. venv/bin/activate
pip install lz4

Actual result:

$ ./runtest.sh 
Collecting lz4
  Using cached lz4-0.14.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-vbudxya6/lz4/setup.py", line 7, in <module>
        import pkgconfig
    ModuleNotFoundError: No module named 'pkgconfig'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-vbudxya6/lz4/

Expected result:

Should install.

LZ4F_getFrameInfo failed with code: ERROR_frameType_unknown

I compressed the data using lz4 java library (https://github.com/lz4/lz4-java) and want to decompress it in python using your library. And the compressed data is available in redis cluster. When I execute the following code:

from rediscluster import StrictRedisCluster
startup_nodes = [{"host": "my_host", "port": "7002"}]
rc = StrictRedisCluster(startup_nodes=startup_nodes, decode_responses=False)
result=rc.hget("key","field")
from lz4.frame import compress, decompress
decompress(result)

I am getting the error:"LZ4F_getFrameInfo failed with code: ERROR_frameType_unknown"

What am I doing wrong?

PyCapsule_New calls have no destructor - memory leak

None of the calls to PyCapsule_New are passing in a destructor. The result is that the various heap-allocated chunks inside these capsules will eventually leak.

For example, calling lz4.frame.makePrefs() in a loop will eventually consume all memory even if the result is not bound to a variable.

Feature Request: enable simple frame compression without content-size header

The lz4 frame specification makes the content-size header field optional. I would love to be able to lz4.frame.compress(foo, content_size=False) and have the encoded frame not include the C.Size FLG and the optional 8 bytes of size data.

For example, the lz4f package does not include the content-size header and results in different bits:

In [1]: import lz4f
In [2]: lz4f.compressFrame(b'1234')
Out[2]: '\x04"M\x18`@\x82\x04\x00\x00\x801234\x00\x00\x00\x00'
In [3]: import lz4.frame
In [4]: lz4.frame.compress(b'1234')
Out[4]: '\x04"M\x18h@\x04\x00\x00\x00\x00\x00\x00\x00\xcd\x04\x00\x00\x801234\x00\x00\x00\x00'

0.9 release?

Any chance for a 0.9 release? Been waiting for it for a long time, and there havent been any new commits in a month and a half.

decompress() shouldn't expect the data to start with the size

Unity3d uses lz4 to compress blocks of data. The size is stored separately. In order to use this library to decompress the blocks in question, a buffer has to be recreated which is unnecessarily slow.

Could you add a way to specify the data size through an argument?

Build wheels for Mac OSX on Travis - publish to PyPi

Post 1.0, it would be handy to publish wheels for OSX on PyPi (similar to how we do for Windows Wheels from Appveyor).

Travis has support for OSX builds. The PyGame folks have this working, so that would be a good place to start. See: the pygame .travis.yml