boludoz / lz4 Goto Github PK

What steps will reproduce the problem?
1. when extracting compressed data
2.
3.

What is the expected output? What do you see instead?
LZ4_uncompress output differ

What version of the product are you using? On what operating system?
linuxmint maya x64, kernel 3.4.4

Please provide any additional information below.
lz4 rev68, compressed with lz4hc, some bytes differ when decompressing
compressed size 638MB, uncompressed size around 1.8GB

different 1

extracted_file1:
03EE DCE0: 32 42 39 44 41 38 44 38  39 45 44 39 30 39 34 35  2B9DA8D8 9ED90945 
extracted_file2: 
03EE DCE0: 32 42 31 44 41 38 44 38  39 45 44 39 30 39 34 35  2B1DA8D8 9ED90945  

different 2

extracted_file1:
3D36 DF20: CD C6 B2 FF CE C8 BB FF  D0 CA BE FF D2 CC C1 FF  ........ ........  
extracted_file2:
3D36 DF20: CD C6 BA FF CE C8 BB FF  D0 CA BE FF D2 CC C1 FF  ........ ........  

different 3

extracted_file1:
3EA7 6F20: A1 D7 17 BC 06 6A 9C 11  2C 4F 31 D6 16 64 74 B4  .....j.. ,O1..dt.  
extracted_file2:
3EA7 6F20: A1 D7 13 BC 06 6A 9C 11  2C 4F 31 D6 16 64 74 B4  .....j.. ,O1..dt.  

different 4

extracted_file1:
528F 9F60: 83 78 10 00 74 12 8B 45  FC 8B 40 18 8B 4D FC 8B  .x..t..E [email protected]..  
extracted_file2:
528F 9F60: 83 78 18 00 74 12 8B 45  FC 8B 40 18 8B 4D FC 8B  .x..t..E [email protected]..  

some different more, and they looks like randomly, but no idea why.

regards,
Mark

What steps will reproduce the problem?
1. make clean
2. make

What is the expected output? What do you see instead?
lz4demo.c: In function ‘decode_file’:
lz4demo.c:294:9: warning: ignoring return value of ‘fwrite’, declared with 
attribute warn_unused_result
lz4demo.c: In function ‘compress_file’:
lz4demo.c:202:8: warning: ignoring return value of ‘fwrite’, declared with 
attribute warn_unused_result
lz4demo.c:223:9: warning: ignoring return value of ‘fwrite’, declared with 
attribute warn_unused_result

What version of the product are you using? On what operating system?
gcc-4.5.3, glibc-2.14.1-r3, linux

Please provide any additional information below.
return value of fwrite is not checked anywhere, at all.
which means compression or decompression will silently fail on a full disk
also output may be corrupted due to temporary disk error, e.g. comp/decomp 
output to a network file system

The Free Software Foundation is no longer at 675 Mass Ave, Cambridge, MA.

Our new address is 51 Franklin Street, Suite 500, Boston, MA 02110.

You can confirm this for yourself here: http://www.fsf.org/about/contact/

Please update all references to our old address in your code so people can 
continue to contact us (we haven't been at the old address for more than a 
decade at this point)

Kind regards,

matt

--
Matt Lee
Campaigns Manager
Free Software Foundation

Hi,

The header for LZ4_Decompress() states:

// Note : The decoding functions LZ4_uncompress() and 
LZ4_uncompress_unknownOutputSize()
//              are safe against "buffer overflow" attack type.
//              They will never write nor read outside of the provided input 
and output buffers.
//              A corrupted input will produce an error result, a negative int, 
indicating the position of the error within input stream.

However, the following demo program shows that this is not always the case:

#include <stdio.h>
#include <string.h>
#include "lz4.h"

int main(int argc, char** argv)
{
        char *input, *output;

        input = (char *)malloc(3);
        input[0] = 0xf0;
        input[1] = 0xfe;
        input[2] = 0x00;

        output = (char *)malloc(4096);

        int ret = LZ4_uncompress_unknownOutputSize(input, output, 3, 4096);
        printf("%d\n", ret);
        return 0;
}

Running it with Valgrind reveals:

==26795== Invalid read of size 8
==26795==    at 0x400750: LZ4_uncompress_unknownOutputSize 
(/home/sesse/lz4-read-only/lz4.c:744)
==26795==    by 0x400667: main (/home/sesse/lz4-read-only/democase.c:16)
==26795==  Address 0x51b6072 is 2 bytes inside a block of size 3 alloc'd
==26795==    at 0x4C2A4D6: malloc 
(/home/samsonov/valgrind-variant/valgrind-test/coregrind/m_replacemalloc/vg_repl
ace_malloc.c:263)
==26795==    by 0x40063A: main (/home/sesse/lz4-read-only/democase.c:9)
==26795== 
==26795== Invalid read of size 8
==26795==    at 0x400761: LZ4_uncompress_unknownOutputSize 
(/home/sesse/lz4-read-only/lz4.c:744)
==26795==    by 0x400667: main (/home/sesse/lz4-read-only/democase.c:16)
==26795==  Address 0x51b607a is 7 bytes after a block of size 3 alloc'd
==26795==    at 0x4C2A4D6: malloc 
(/home/samsonov/valgrind-variant/valgrind-test/coregrind/m_replacemalloc/vg_repl
ace_malloc.c:263)
==26795==    by 0x40063A: main (/home/sesse/lz4-read-only/democase.c:9)

This is with LZ4 as of r57, 64-bit x86, GCC 4.4.3.

Hi Yann,

If you give LZ4 HC a sequence such as

[1, 2, 3, 4, 5, 6, [...65530 incompressible bytes...], 1, 2, 3, 4, 5, 6, [other 
bytes]]

LZ4 HC won't be able to compress it because it always cheks ref > 
ip-MAX_DISTANCE instead of ref >= ip-MAX_DISTANCE (see attached patch for more 
details).

Thank you!

There are two API calls that work with context - LZ4_compressCtx and 
LZ4_compress64kCtx. But there is not API to free context at the end.

What steps will reproduce the problem?
1. svn co http://lz4.googlecode.com/svn/
2. cd svn/trunc
3. gmake CC="gcc -m64" lz4demo

What is the expected output? What do you see instead?
Succesfully built 64 bit lz4demo.
Instead I get : 
In file included from bench.c:56:0:
/usr/include/stdio.h:333:29: error: unknown type name 'off64_t'
/usr/include/stdio.h:334:1: error: unknown type name 'off64_t'
gmake: *** [lz4demo] Error 1


What version of the product are you using? On what operating system?
Latest. Solaris 10 on sparc.



Please provide any additional information below.

The fix is to change 

#if (defined(__sun__) && (!defined(__LP64__)))   // Sun Solaris 32-bits 
requires specific definitions
#  define _LARGEFILE_SOURCE 
#  define FILE_OFFSET_BITS=64
#else
#  define _LARGEFILE64_SOURCE
#endif


to
#if (defined(__sun__) && (!defined(__LP64__)))   // Sun Solaris 32-bits 
requires specific definitions
#  define _LARGEFILE_SOURCE 
#  define FILE_OFFSET_BITS=64
#elif ! defined(__LP64__)         // No point defining Large file for 64 bit
#  define _LARGEFILE64_SOURCE
#endif

What steps will reproduce the problem?
1. make all 

Actual result:
make all
gcc -m32 -Os -march=native -I. -std=c99 -Wall -W -Wundef 
-Wno-implicit-function-declaration lz4hc.c lz4.c bench.c lz4demo.c -o lz4demo32
In file included from /usr/include/features.h:385,
                 from /usr/include/stdlib.h:25,
                 from lz4hc.c:83:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
In file included from /usr/include/features.h:385,
                 from /usr/include/stdlib.h:25,
                 from lz4.c:139:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
In file included from /usr/include/features.h:385,
                 from /usr/include/stdlib.h:25,
                 from bench.c:48:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
In file included from /usr/include/features.h:385,
                 from /usr/include/stdio.h:28,
                 from lz4demo.c:40:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
make: *** [lz4demo32] Error 1

What is the expected output? What do you see instead?
Binaries created

What version of the product are you using? On what operating system?
libgcc-4.4.6-4.el6.x86_64
gcc-4.4.6-4.el6.x86_64
libgcc-4.4.6-4.el6.i686

With Visual Studio 2012, changing:
            *op++ = *ref++;
            *op++ = *ref++;
            *op++ = *ref++;
            *op++ = *ref++;
into:
            op[0] = ref[0];
            op[1] = ref[1];
            op[2] = ref[2];
            op[3] = ref[3];
            op += 4, ref += 4;

gives a 1.5% speed boost in my benchmark.

What steps will reproduce the problem?
1. Compile with 
cc -xc99 -fast -I. -m32 lz4.c lz4hc.c bench.c lz4demo.c -o lz4demo.exe 
-xmemalign=1i -erroff=E_WHITE_SPACE_IN_DIRECTIVE
2. run ./lz4demo.exe ./lz4demo.exe target && ./lz4demo.exe -d target back && 
cmp ./lz4demo.exe back


What is the expected output? What do you see instead?
No errors from cmp.
Instead:
./lz4demo.exe back differ: char 44, line 1


What version of the product are you using? On what operating system?
Solaris 10 sparc. cc: Sun C 5.12 SunOS_sparc Spica 2011/07/21



Please provide any additional information below.

This can be fixed by moving the #include directives to the top.
Solaris Studio needs to include some files  before it can have sensible values 
for BIG_ENDIAN or BYTE_ORDER.


I know I have reported this before and not followed up.
This time I promise to follow up and do all tests you ask of me.

> uname -v
Darwin Kernel Version 11.4.0: Mon Apr  9 19:33:05 PDT 2012; 
root:xnu-1699.26.8~1/RELEASE_I386

LZ4 stopped building after revision r71:

> svn checkout -q http://lz4.googlecode.com/svn/trunk/ lz4-trunk && cd 
lz4-trunk && make
gcc      -O3 -I. -std=c99 -Wall -W -Wundef -Wno-implicit-function-declaration 
lz4hc.c lz4.c bench.c lz4demo.c -o LZ4Demo.exe
Undefined symbols for architecture x86_64:
  "_LZ4_compress64kCtx", referenced from:
      _LZ4_compress_limitedOutput in ccJ02lMQ.o
      _LZ4_compress in ccJ02lMQ.o
  "_LZ4_compressCtx", referenced from:
      _LZ4_compress_limitedOutput in ccJ02lMQ.o
      _LZ4_compress in ccJ02lMQ.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
make: *** [lz4demo] Error 1

Revisions before r71 build ok:

> svn checkout -q -r 70 http://lz4.googlecode.com/svn/trunk/ lz4-ok && cd 
lz4-ok && make
gcc      -O3 -I. -std=c99 -Wall -W -Wundef -Wno-implicit-function-declaration 
lz4hc.c lz4.c bench.c lz4demo.c -o LZ4Demo.exe

I received a failure report [1] from the CPAN testers in which __builtin_ctz 
was an undefined symbol. A bit of searching shows that was added in GCC 3.4 and 
the tester's environment had version 3.3.3. I suggest testing for the version 
of GCC and providing a fallback function if the builtin isn't available in the 
user's version.

[1] http://www.cpantesters.org/cpan/report/7b034a9c-581e-11e1-a156-f932921a7b64

Currently the users have to estimate the worst-case compressed size by 
following the guidance in the comment to LZ4_compress; moreover, it's not 
possible to use a smaller buffer (i.e. we expect the data to be at least 50% 
compressible, so we want to save memory in the common case).

It would be nice to have:
1. A function for converting input size to worst-case output size (i.e. see 
compressBound() in zlib)
2. A function for safe compression with a specified output buffer size that's 
guaranteed to never write past the end; return negative number or zero to 
indicate error
3. (related) A function for safe decompression with a specified input buffer 
size that's guaranteed to never read past the end of input buffer.

Any ideas for improving compress performance on ARM?  I'm running some 
benchmarks on an OpenVPN packet dataset (IPv4 packets, < 1500 bytes each, 
reflecting real-world web browsing sessions) and LZ4 is taking twice as much 
time as LZO to achieve roughly comparable compression ratios.

James

What steps will reproduce the problem?
1. Compress data.
2. Recompress the same input with output limited to exactly the length of the 
previous run.

What is the expected output? What do you see instead?
Compression sometimes fails when there is exactly enough space.

What version of the product are you using? On what operating system?
svn r77, Arch Linux x86_64

Please provide any additional information below.
The calculation of the encoded run length was not exact, and could differ from 
the actual run length by a small amount in some cases. op + (encoded length) 
was also being compared as greater than or equal to oend, but a result exactly 
equal to oend means that the encoded data will exactly fill the buffer. I 
believe the attached patch corrects all length tests in lz4.c properly. lz4demo 
-b shows a small cost in compression speed, about 2.5% for larger data and 
about 7% for the 64k algorithm.

Hello,

It appears that there is an issue with the compression routine reading past the 
end of the array for certain inputs.

Please see the attached fuzz test. It will crash on the second iteration.

What steps will reproduce the problem?
1. LZ4_uncompress returns the number of input bytes processed

What is the expected output? What do you see instead?
I would expect it to return the number of bytes decompressed. I already know 
the size of my input and that is what always shows in the return value.

What version of the product are you using? On what operating system?
Using on iOS 6, iPad 3

Please provide any additional information below.
I've modified the return value to be this:

return 5 + (int)((char*)op - dest); // return the num bytes output

For my purposes this returns the number of bytes uncompressed, although I have 
to add 5 because the op pointer doesn't seem to be at the real end when the 
algorithm finishes.

Hi,

I am doing some tests with lz4 trunk and I am having trouble with 
uncompress_unknownOutputSize and compressHC. More specifically, I am testing it 
on random char streams (same likelyhood for every char in the [0, 256[ range), 
then compress with compressHC and uncompress_unknownOutputSize always returns 
an uncompressed size of 4096 when the size of the original stream size is ~4200 
or more.

On the other hand, it works perfectly fine:
 - when the original size of the stream is < 4000,
 - when I decompress with uncompress, whatever the original stream size is,
 - with a non-random stream, whatever the original stream size is.

When compiling lz4 on NetBSD, the build fails because of the obsolete ftime() 
function. It still does exist on NetBSD, but you would have to link against 
-lcompat.
Even on Linux, the ftime man page says:
"This  function  is  obsolete.  Don't use it." and recommends gettimeofday() 
instead, which is provided by BSDs since 4.2BSD and has been added to the POSIX 
standard in 2001.

The attached patch replaces ftime() with over to gettimeofday().

Functions LZ4_uncompress() and LZ4_uncompress_unknownOutputSize() use restrict 
keyword with local variables op and ref:

const BYTE* restrict ref;

BYTE* restrict op = (BYTE*) dest;

As far as I understand how 'restrict' works, this seems like an invalid use 
that can fool the compiler. 

In fact, when using icc version 11.0 (with plenty of optimizations on), this 
results in wrong decompression output (fixed by getting rid of restrict on both 
variables).

Hi Yann,

If you give LZ4 HC a sequence such as

[1, 2, 3, 4, 5, 6, [...65530 incompressible bytes...], 1, 2, 3, 4, 5, 6, [other 
bytes]]

LZ4 HC won't be able to compress it because it always cheks ref > 
ip-MAX_DISTANCE instead of ref >= ip-MAX_DISTANCE (see attached patch for more 
details).

Thank you!

1. Run the stress test (Windows) http://directnet-drive.net/lz4_test.cpp
2. Program crashes
3. #define LZ4_WORKAROUND to workaround the bug

lz4 latest svn source, compiled using VS2010, run on Windows 7

lz crashes on the line
for(;length>0;length-=4) { *(U32*)op = *(U32*)anchor; op+=4; anchor+=4; }
It tries to access 4 bytes at anchor address but it points to the end of the 
input stream minus 1/2/3 bytes.

What steps will reproduce the problem?
1. Download code, gmake

What is the expected output? What do you see instead?
Successful compile.

In file included from bench.c:50:0:
/usr/include/stdio.h:333:29: error: unknown type name 'off64_t'
/usr/include/stdio.h:334:1: error: unknown type name 'off64_t'

Fixed by removing the #define _LARGEFILE64_SOURCE line (line 33)
Could maybe replace with
#ifndef __LP64__
#ifdef __sun__
#define _LARGEFILE_SOURCE 
#define FILE_OFFSET_BITS=64
#else
#define _LARGEFILE64_SOURCE 
#endif
#endif

Or better move it into the makefile with with a `getconf LFS_CFLAGS` on the 
compile line.

This is a first attempt at limited output for LZ4HC. Passes fuzzer tests, slows 
compression by <3% on i7.

What steps will reproduce the problem?
1. cmake cmake
2. make


What is the expected output? What do you see instead?

Expected a working build. Got:

rhino ➜  ~/projects/lz4-read-only  make
[ 16%] Building C object CMakeFiles/lz4demo32.dir/lz4.c.o
[ 33%] Building C object CMakeFiles/lz4demo32.dir/bench.c.o
[ 50%] Building C object CMakeFiles/lz4demo32.dir/lz4demo.c.o
Linking C executable lz4demo32
CMakeFiles/lz4demo32.dir/lz4demo.c.o: In function `compress_file':
lz4demo.c:(.text+0x25c): undefined reference to `LZ4_compressHC'
CMakeFiles/lz4demo32.dir/bench.c.o: In function `BMK_benchFile':
bench.c:(.text+0x138): undefined reference to `LZ4_compressHC'
collect2: ld returned 1 exit status
make[2]: *** [lz4demo32] Error 1
make[1]: *** [CMakeFiles/lz4demo32.dir/all] Error 2
make: *** [all] Error 2


What version of the product are you using? On what operating system?

subversion checkout.

Please provide any additional information below.

When I changed CMakeLists.txt from

set(LZ4_SRCS_LIB ${SRC_DIR}lz4.c ${SRC_DIR}lz4.h )

to

set(LZ4_SRCS_LIB ${SRC_DIR}lz4.c ${SRC_DIR}lz4hc.c ${SRC_DIR}lz4.h )

the build proceeded correctly.

On gcc and clang, we can use:

#define forceinline __inline__ __attribute__((always_inline))
#define ensure_forceinline __attribute__((always_inline)) // inline or die

I have compiled the latest code using Visual Studio 2005 and the generic CE 6.0 
SDK, as well as Pocket PC 2003 and a specific handheld device, all ARM4 based. 
_FORCE_SW_BITCOUNT is defined (though generates the warning "unary minus 
operator applied to unsigned type, result still unsigned".)

The LZ4_uncompress function throws an exception at LZ4_WILDCOPY (line 657 of 
lz4.c.)

According to the msdn page: 
http://msdn.microsoft.com/en-us/library/ms253923%28v=vs.80%29.aspx

VC device compiler are not support bit-scan intrinsics.

So LZ4_FORCE_SW_BITCOUNT should be defined if _MSC_VER and _WIN32_WCE are both 
defined.

What steps will reproduce the problem?
1. gmake
2. run ./lz4demo.exe ./lz4demo.exe /dev/null


What is the expected output? What do you see instead?
Successful compress. Bus  error. Core dump.


What version of the product are you using? On what operating system?
Solaris 10. sparc.

Please provide any additional information below.


When compiling with SolarisStudio I am forced to compile with -xmemalgin=1i 
anyway, and the problem does not appear. Guess it is a performance killer.
I have not found a similar option for gcc.

When I changed some *(U32*)x to A32(x) it worked .

212c211
< #define HASH_VALUE(p)         HASH_FUNCTION(A32(p))

---
> #define HASH_VALUE(p)         HASH_FUNCTION(*(U32*)(p))
345c344
<                   if (A32(ref) == A32(ip))

---
>               if (*(U32*)ref == *(U32*)ip)
388c387
<                   if (A32(ref) == A32(ip))

---
>               if (*(U32*)ref == *(U32*)ip)

1. Compile lz4 for ARM cpu
2. Turn on SIGBUS signals generation in case of unaligned access (on linux can 
be done via "echo 5 > /proc/cpu/alignment")
3. Run lz4 test

I'm trying it on linux 2.4.21.
The core of the problem is that ARM/MIPS cpu does not allow unaligned access, 
so that the following code crashes:
char a[8];
*(int*) &a[1] = 0; // crash because of unaligned write to mem
int b = *(int*) &a[1]; // crash because of unaligned read from mem

In lz4 there is a plenty of such lines:
do { *(U32*)op = *(U32*)anchor; op+=4; anchor+=4; } while (op<l_end) ;
in case of op pointer not aligned to 32-bit boundary that would cause crash 
(SIGBUS).

The possible solution could be using __attribute__ ((packed)) if compiling 
using gcc (that's almost always true for ARM/MIPS).
In case of x86 & x64 that won't affect performance at all, the generated code 
should remain the same. In case of ARM/MIPS that forces compiler to generate 
per-byte access code. Plz take a look at the patch:
http://directnet-drive.net/lz4.20.patch
(result here: http://directnet-drive.net/lz4.c)
That fixes the problem in my case and ARM tests succeed. x86 and x64 code is 
not affected at all.

Although ARM/MIPS cpus are not so widespread now they are becoming more and 
more popular. If you need any kind of my help please do not hesitate to ask for 
it.

That's not the duplicate of the previous issue. I've changed the data being 
compressed and this happened again.

1. Run the stress test (Windows) http://directnet-drive.net/lz4_test.cpp (I've 
uploaded a new version of the test)
2. Program crashes
3. #define LZ4_WORKAROUND to workaround the bug

lz4 latest svn source (r25), compiled using VS2010, run on Windows 7
lz crashes on the lines
do { *(U32*)op = *(U32*)ip; op+=4; ip+=4; } while (op<ref) ;
and
ip++; forwardH = HASH_VALUE(ip);

What steps will reproduce the problem?
1. call LZ4_uncompress_unknownOutputSize with an osize that is exactly the size 
of the uncompressed data. It will return a decompressed size that is too small.


What is the expected output? I expected 65535 What do you see instead? I got 
65531


The bug is in line 534. The line following a memcpy.  ip += length; Since 
LZ4_uncompress_unknownOutputSize uses op to calculate the length unless there 
is an error this line should be op += length;

What steps will reproduce the problem?
1. Use ubuntu 8.04 with gcc (gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu4))
2. Just run make


I use sources from svn on ubuntu 8.04. I use gcc 4.2.4, but that don't have 
__builtin_bswap32. And i have errors after run make.

gcc      -O3 -I. -std=c99 -Wall -W -Wundef -Wno-implicit-function-declaration 
lz4hc.c lz4.c bench.c lz4demo.c -o lz4demo64
/tmp/ccqSPzan.o: In function `decode_file':
lz4demo.c:(.text+0x3dd): undefined reference to `__builtin_bswap32'
lz4demo.c:(.text+0x468): undefined reference to `__builtin_bswap32'
/tmp/ccqSPzan.o: In function `compress_file':
lz4demo.c:(.text+0x756): undefined reference to `__builtin_bswap32'
lz4demo.c:(.text+0x771): undefined reference to `__builtin_bswap32'
lz4demo.c:(.text+0x7ae): undefined reference to `__builtin_bswap32'
collect2: ld returned 1 exit status
make: *** [lz4demo64] Error 1

I have patched your sources for fix that problem.

--- lz4demo.c   (revision 67)
+++ lz4demo.c   (working copy)
@@ -58,8 +58,13 @@
 #if defined(_MSC_VER)    // Visual Studio
 #define swap32 _byteswap_ulong
 #elif GCC_VERSION >= 402
+#include <byteswap.h>
+#if defined(__builtin_bswap32)
 #define swap32 __builtin_bswap32
 #else
+#define swap32 __bswap_32
+#endif
+#else

I'm attaching a patch which properly declares source arrays as const.

This is needed to prevent warnings or errors when passing const source data to 
LZ4 methods.

James

With Visual Studio 2012, changing "BYTE token" to "unsigned token" in 
"LZ4_uncompress" removes one byte=>dword instruction and makes the decompressor 
approx 2.5% faster in my benchmark.

What steps will reproduce the problem?
1. Take 5 gb logfile
2. Compress with LZ4
3. Decompress

What is the expected output? What do you see instead?
Expected to see original file, got a 0byte file

What version of the product are you using? On what operating system?
LZ4 official windows binary, v1.3, on win7, win server2008r2

Please provide any additional information below.
Files 1 to 1.5 gb work perfeclty fine.

What steps will reproduce the problem?
1. No lz4.so library available

What is the expected output? What do you see instead?

More of a feature request, I suppose. I'd fork it and make a pull request if 
you were on github:) I'd like to be able to link to lz4.so from Haskell.

The comments for LZ4_uncompress() in lz4.h state:

"This function never writes beyond dest + osize, and is therefore protected 
against malicious data packets"

I don't see how can this be true, if it doesn't even take isize as a parameter. 
Surely a malicious attacker could truncate the input data, causing reads beyond 
the end of the input buffer for any given expected osize, not to mention more 
sophisticated attacks "fishing" data past the end of the input buffer.

Is there something I'm missing?

I know there's LZ4_uncompress_unknownOutputSize for safe decoding, but if I'm 
right I think this should be clarified.

In some situation, LZ4_compress_limitedOutput() may write beyond provided 
output buffer.

unpack source distro:

python setup.py build
Traceback (most recent call last):
  File "setup.py", line 13, in <module>
    long_description=open('README.rst', 'r').read(),
IOError: [Errno 2] No such file or directory: 'README.rst'

STEPS:
1. Create a Win32 console project
2. Add LZ4 source files to the project
3. Start build

EXPECTED RESULT:
Successfull build

ACTUAL RESULT:
Build failed with error:
lz4hc.obj : error LNK2019: unresolved external symbol __BitScanForward 
referenced in function _LZ4_NbCommonBytes

A user reported invalid roundtrip results for the Perl interface under Ubuntu 
amd64. I was able to able to reproduce using Virtualbox with the plain C 
library. The error is not present when compiling as a 32-bit executable with 
-m32.

Attached are the c program I used and the input file.

$ uname -a
Linux ubuntu 3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 
x86_64 x86_64 GNU/Linux

On NetBSD, including stdlib.h pulls in bswap.h which defines bswap16.

Reference error: 
http://www.cpantesters.org/cpan/report/34d82f1a-71a5-11e1-8755-e63c2a028b78

You might be able to borrow some of the portability tweaks from csnappy: 
https://github.com/zeevt/csnappy/blob/master/csnappy_internal_userspace.h

As part of testing out LZ4 I have compiled and run it on a
collection of machines on our build cluster including some with
very old compilers.

I needed to make a number of changes in order for lz4.c to
compile and work everywhere.  I did not test lz4hc.c, but it
probably needs a matching set of changes.

The build cluster includes:
   x86 linux on Debian: 32 & 64bit versions of all releases since 3.0
   x86 freebsd: all 32 & 64bit versions since 6.0
   netbsd 1.6.1 on 32bit x86
   netbsd 4.0 on 32/64bit x86
   OpenBSD 4.2 on x86
   linux on ia64 & PowerPC
   Solaris on x86 and Sparc
   SCO OpenServer R5 on x86
   MacOS 10.3 PowerPC
   MacOS 10.4 x86
   AIX 5 on PowerPC
   HP-UX on PARISC and IA-64

   and probably some other oddballs that I forgot about

Here were the changes I needed to make

64-bit detection:

  The define for 64-bit code was missing __ia64__

BIG_ENDIAN:

  For big endian I added __hpux__ && __sparc__ 
  I didn't test this part, but it would probably be a good idea to
  also add a test of:
    && !(defined(__BYTE_ORDER__) &&
     (__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__))

  The powerpc can actually be either, but in most common cases
  it is big endian so that part of the test seems to work.

Packed structures:

  The '#pragma pack(1)' directive for the U32_S structures doesn't
  work in older GCCs or with the ia64 architecture.
  In my searching I found the __packed__ attribute was more portable,
  so I switched to using that instead.  Not sure if that will
  work with MSVC or not as that is not one of my targets.

stdint.h

  The stdint.h is too new to be included on all of these machines so
  used typedefs directly. (why #defines before?)
  In theory these defines can be wrong, but in practice it works on
  every architecture in common use.  

You need to add:
#pragma intrinsic(_BitScanForward)

Otherwise VC will complain an error (unresolved symbol) at link time of DEBUG 
Version.

This is because the "/Oi" option are not specified while compiling the debug 
version.

Other bit-scan intrinsics are similar.

In some cases, LZ4_uncompress does not return error codes when the input is 
invalid.

For example uncompressing [16, 42, 1, 0, 80, 42, 42, 42, 42, 42, 42] (which is 
a compressed form of an array of length 10 whose elements are all 42) with 
osize == 11 won't return an error, although the uncompressed length of the 
array is 10.

Additionally, LZ4 does not return errors when ref == op (dec == 0).

LZ4 library should be refactored to support streaming compression and 
decompression similar to how zlib works.

There would be an lz4Init(), lz4(), and lz4End() similar to how the inflate 
functions work in zlib. You would pass around a pointer to the stream for each 
function. Call to lz4() would save the state of the compression/decompression 
machine for subsequent calls to lz4(). Here is zlib's stream structure.

typedef struct z_stream_s {
    z_const Bytef *next_in;     /* next input byte */
    uInt     avail_in;  /* number of bytes available at next_in */
    uLong    total_in;  /* total number of input bytes read so far */

    Bytef    *next_out; /* next output byte should be put there */
    uInt     avail_out; /* remaining free space at next_out */
    uLong    total_out; /* total number of bytes output so far */

    z_const char *msg;  /* last error message, NULL if no error */
    struct internal_state FAR *state; /* not visible by applications */

    alloc_func zalloc;  /* used to allocate the internal state */
    free_func  zfree;   /* used to free the internal state */
    voidpf     opaque;  /* private data object passed to zalloc and zfree */

    int     data_type;  /* best guess about the data type: binary or text */
    uLong   adler;      /* adler32 value of the uncompressed data */
    uLong   reserved;   /* reserved for future use */
} z_stream;

What steps will reproduce the problem?
1. char * c = "abcd";
2. c[2] = '\0';
3. compress c & fail.

I'd like to be able to compress arbitrary data.

https://github.com/mwotton/lz4hs has more detailed test cases.

What steps will reproduce the problem?
1. Read attached file as byte array
2. Compress it
3. Decompress Unknown output size

What is the expected output? What do you see instead?
Decompressed output should match input.
The decompressed array is too short.

What version of the product are you using? On what operating system?
I am actually using the LZ4Sharp port but changing line 850-ish from 
if (op == oend) break;    // Check EOF (should never happen, since last 5 bytes 
are supposed to be literals)
to 
if (op == oend) goto _output_error;
fixes the problem because then whatever is looping and increasing the output 
array size knows to run it again with a bigger size. The comment also indicates 
this is an error condition. 


Please provide any additional information below.
The data file is attached.

The error may only show up when increasing the output array in certain size 
increments. I was using input array * 4 to start and multiplying by 4 each time 
through the loop.

What steps will reproduce the problem?
1. svn checkout http://lz4.googlecode.com/svn/trunk/ lz4-read-only

Actual result:
svn: Server sent unexpected return value (307 Temporary Redirect) in response 
to OPTIONS request for 'http://lz4.googlecode.com/svn/trunk'

What is the expected output? What do you see instead?
Get the cource code.

What version of the product are you using? On what operating system?
tigervnc-1.0.90-0.17.20110314svn4359.el6.x86_64