stangelandcl / lz4sharp

Unmaintained port of LZ4 Compression algorithm to C#

This port is unmaintained and unlikely to be updated. I have personally switched to using Yann Collet’s C code via C++/CLI. For a .NET LZ4 solution try http://lz4net.codeplex.com/ which has multiple configuration and includes LZ4HC.

LZ4Sharp

Port of Yann Collet’s LZ4 algorithm to C# by Clayton Stangeland.

Original LZ4 algorithm can be found at (http://code.google.com/p/lz4) it was created by Yann Collet. See license.txt for project license.

Build

Building in Debug or Release will use the already generated C# files

LZ4Compressor.c and LZ4Decompressor.c are updated by manually comparing to the original LZ4 code and fixing any C# differences from C but leaving some of the macros to make comparing easier.

Then building in the Generated configuration will only build the LZ4SharpGenerator project which has a prebuild event that uses the MinGW gcc from CodeBlocks to generate the C# files from LZ4Compressor.c and LZ4Decompressor.c.

Timings

Compression times do not include disk I/O.

P4 = Pentium 4 2GHz 32 bit Windows XP

i5 = Intel i5 2.67 GHZ 64 bit Windows 7 running LZ4Sharp in 64 bit

LZ4Sharp = Timings for ‘webster’ from silesia corpus (http://sun.aei.plsl.pl/~sdeor/index.php?page=silesia)

memcpy and LZ4 = Timing for file ‘webster’ in silesia corpus in m2mark.exe from http://sd-1.archive-host.com/membres/up/182754578/m2mark.zip (or from link to benchmark program at http://code.google.com/p/lz4)

gzip = GZip from the Intel compiler built as a dll and accessed from C# by PInvoke

Ratio = compressed size / decompressed size (lower is better)

Computer	Type	Compression (MB/s)	Decompression (MB/s)	Ratio (%)
P4	memcpy	217
P4	LZ4	82	304
P4	LZ4Sharp	43	150
P4	LZ4Sharp	49	181
i5	memcpy	1658
i5	LZ4	270	1184
i5	LZ4Sharp	207	758	49
i5	LZ4Sharp	267	838	47
i5	gzip	48	266	33

Conclusion:

LZ4 C# is about 2/3 the speed of the c version. (Also, LZ4C# is slightly faster on the whole silesia corpus than on just the ‘webster’ file) LZ4 C# Compressed silesia corpus is 47% of uncompressed LZ4 C# Compressed ‘webster’ file is 49% of uncompressed

lz4sharp's Issues

Compiles, but does not run in Compact Framework 3.5

Hi!
I've tried the current source code to build and run in Compact Framework and I did succeed, but the execution halts at some point and the process quits without throwing any managed Exceptions.

What do you think the problem is?

I'v uploaded here a ZIPped test project: http://www.filedropper.com/lz4cf35bug

Please take a look at it!

Thanks!

High Compression mode

Have you done anything in porting the High Compression Mode of LZ4?

System.OverflowException thrown while decompressing data

LZ4Sharp source revision: a2de5bf (Feb 24, 2012)
Build tools: Microsoft Visual Studio 2008, MinGW gcc 4.6.1
Code:

    byte[] src = new byte[50];
    ILZ4Compressor compressor = new LZ4Compressor32();
    byte[] dst = compressor.Compress(src);
    LZ4Decompressor32 decompressor = new LZ4Decompressor32();
    decompressor.Decompress(dst); // System.OverflowException here

Not compatible with C version

Hi, I'm finding that files generated by this library are not decompressed by the original implementation http://fastcompression.blogspot.com.au/p/lz4.html. Files compiled by the original lz4, generate runtime errors in this library. Known/intentional? :)

Put LZ4Sharp up on NuGet

Hi,

just wondering if you have any plans on putting it up on NuGet?

//Daniel

LZ4 Algorithm in C#

We are going to develop a instant messenger application for LAN with data compression. We decided to use LZ4 algorithm for data compression in file transferring and we are going to use C# language. If you could give an example of the compression and decompression process through a diagram and a pseudo code. Thank you

Decompression fails on some portions of data.

Decompression of some compressed blocks fails.

Problem to run

I just couldn't figured out to run this project could you help me little bit thank you.

I got this error: "This will time compressing and decompressing all the files in 'directory' ignoring file read time" in very beginning

or

this error in using generator program.cs : "Please build the project and retry, or set the OutputPath and AssemblyName properties appropriately to point at the correct location for the target assembly."

System.AccessViolationException

We used LZ4Sharp for xml string - at most 40Kbyte - compression application. It calls compress method about 50 million times. But we got System.AccessViolationException during the execution. There might be some problem about algorithm. Of course, it might be occured according to our usage.

Exception Info: System.AccessViolationException
Stack:
at LZ4Sharp.LZ4Compressor32.Compress(Byte_, Byte_, Int32, Int32)
at LZ4Sharp.LZ4Compressor32.Compress(Byte[], Byte[])
at LZ4Sharp.LZ4Compressor32.Compress(Byte[])
at Common.XmlCompresser.Compress(System.String)

Our Usage:

public class XmlCompresser
{
    private static ILZ4Compressor compressor = LZ4CompressorFactory.CreateNew();

    private static string Compress(string xml)
    {
        var decompressedDataBytes = Encoding.UTF8.GetBytes(xml);
        var compressedDataBytes = compressor.Compress(decompressedDataBytes);
        return Convert.ToBase64String(compressedDataBytes);
    }
}

Invalid array length after decompression

LZ4Sharp source revision: c2316f5 (Apr 20, 2012)
Build tools: Microsoft Visual Studio 2008, MinGW gcc 4.6.1
Code:

            byte[] src = File.ReadAllBytes(<PATH_TO_DATA_FILE(see links below)>);
            ILZ4Compressor compressor = new LZ4Compressor32();
            byte[] dst = compressor.Compress(src);
            LZ4Decompressor32 decompressor = new LZ4Decompressor32();
            byte[] src1 = decompressor.Decompress(dst);
            Debug.Assert(src.Length == src1.Length);

Data files located at:
https://docs.google.com/open?id=0B6JvQNrlfBqrSFJ1WUlMZ1BqMFE
https://docs.google.com/open?id=0B6JvQNrlfBqrYVBZMzhsNGYtb1U

Add dependencies required to build in lib folder

Would it be possible to add the minimal required gcc executables and libs to a /lib folder (or similar) and reference those? This is to avoid people having install a lot of stuff to build the project.

Data loss during text (de?)compression (output smaller than input)

Great port! Found a big problem though:
I tried many different JSON strings (~300MB) and found at least one, where it seems the last block is missing after decompression (the output is much shorter than the input, ~300 vs. 350k), so it is unusuable. Tried the latest version from ~5h ago using the Debug version (i.e. now non generated). Cannot test generated version, but did test "multithreaded win32" version from original LZ4 homepage and it works! So the problem is in the port.

input: http://snipurl.com/24vzsc1
output http://snipurl.com/24vzu01

Please help!

Thanks!

P.S.: some benchmarks on my machine (all JSON, Vista x64, Sandy Bridge @ 4,6GHz):

Name;%;In;Out;C MB/s;D MB/s;
GZIP ModelleStammdaten;10.82%;292950;31708;39.35;69.84;
LZF ModelleStammdaten;17.66%;292950;51749;93.13;99.78;
LZ4 ModelleStammdaten;15.84%;292950;46402;253.98;155.21;
QLZ1 ModelleStammdaten;13.42%;292950;39315;103.47;84.66;
QLZ3 ModelleStammdaten;11.30%;292950;33094;14.78;103.47;
GZIP ModelleETInfo;9.44%;240525;22700;53.34;120.73;
LZF ModelleETInfo;15.01%;240525;36098;99.73;134.93;
LZ4 ModelleETInfo;14.72%;240525;35401;286.73;286.73;
QLZ1 ModelleETInfo;12.29%;240525;29558;109.23;127.43;
QLZ3 ModelleETInfo;10.20%;240525;24545;14.61;163.84;
GZIP Artikelersetzung;12.59%;2303948;290043;60.53;118.77;
LZF Artikelersetzung;20.64%;2303948;475465;88.96;100.79;
LZ4 Artikelersetzung;18.20%;2303948;419282;278.13;296.92;
QLZ1 Artikelersetzung;14.63%;2303948;337132;122.75;130.01;
QLZ3 Artikelersetzung;14.41%;2303948;332008;11.67;104.63;
GZIP ModelleFahrzeug;8.48%;20268197;1718578;69.83;121.41;
LZF ModelleFahrzeug;15.03%;20268197;3046920;107.21;140.99;
LZ4 ModelleFahrzeug;12.35%;20268197;2502983;274.95;315.84;
QLZ1 ModelleFahrzeug;9.87%;20268197;2001167;129.55;145.01;
QLZ3 ModelleFahrzeug;8.52%;20268197;1726858;14.61;164.64;
GZIP Artikel;7.38%;123633224;9128337;75.54;130.96;
LZF Artikel;12.65%;123633224;15634502;106.13;146.03;
LZ4 Artikel;8.74%;123633224;10806478;296.62;357.29;
QLZ1 Artikel;7.47%;123633224;9240482;133.97;149.27;
QLZ3 Artikel;6.90%;123633224;8536067;15.83;147.73;

Provide stream API

It'd be nice to have a stream-api also. QuickLZ has wrappers that seem to do the trick (except for some length stuff missing in LZ4): http://www.quicklz.com/QuickLZCompressionStream.cs http://www.quicklz.com/QuickLZDecompressionStream.cs