mat-codehaus / lzo-java

Pure Java implementation of the liblzo2 LZO compression algorithm

LZO for Java

Introduction

There is no version of LZO in pure Java. The obvious solution is to take the C source code, and feed it to the Java compiler, modifying the Java compiler as necessary to make it compile.

This package is an implementation of that obvious solution, for which I can only apologise to the world.

It turns out, however, that the compression performance on a single 2.4GHz laptop CPU is in excess of 500Mb/sec, and decompression runs at 815Mb/sec, which seems to be more than adequate. Run PerformanceTest on an appropriate file to reproduce these figures.

Example

Compression:

	OutputStream out = ...;
	LzoAlgorithm algorithm = LzoAlgorithm.LZO1X;
	LzoCompressor compressor = LzoLibrary.getInstance().newCompressor(algorithm, null);
	LzoOutputStream stream = new LzoOutputStream(out, compressor, 256);
	stream.write(...);

Decompression:

	InputStream in = ...;
	LzoAlgorithm algorithm = LzoAlgorithm.LZO1X;
	LzoDecompressor decompressor = LzoLibrary.getInstance().newDecompressor(algorithm, null);
	LzoInputStream stream = new LzoInputStream(in, decompressor);
	stream.read(...);

Documentation

The JavaDoc API is available.

Hadoop Notes

Notes on BlockCompressionStream, as of Hadoop 0.21.x:

• If you write 1 byte, then a large block, BlockCompressorStream will flush the single-byte block before compressing the large block. This is inefficient.

• If you write a large block to a fresh stream, BlockCompressorStream will flush existing data, which will write a zero uncompressed length to the file, but follow it with no blocks, thus breaking the ulen-clen-data format. This is wrong. There is no contract for the finished() method to avoid this, since it must return false at the top of write(), then must (with no other mutator calls) return true in BlockCompressorStream.finish() in order to avoid the empty block; having returned true there, compress() must be able to return a nonempty block, even though we have no data. This is wrong.

• Large blocks are written (ulen (clen data)) not (ulen clen data) due to the loop in compress(). This is not the same as the format for lzop, thus a data file written using LzopCodec cannot be read by lzop. See lzop-1.03/src/p_lzo.c method lzo_compress, which contains a single very simple loop, which is how Hadoop's BlockCompressorStream should be written. This is both inefficient and wrong.

• If the LZO compressor needs to use its holdover field (or, equivalently in other people's code, setInputFromSavedData()), then the ulen-clen-data format is broken because getBytesRead() MUST return the full number of bytes passed to setInput(), not just the number of bytes actually compressed so far; then if there is holdover data, there is nowhere for it to go but into the returned data from a second call to compress(), at which point the API has forced us to break ulen-clen-data, as per lzop's file format. This is wrong, and badly designed.

• The number of uncompressed bytes is written to the stream in lzop. There is therefore no excuse for a "Buffer too small" error in decompression. However, this value is NOT used to resize the decompressor's output buffer, and so the error occurs. One cannot, as a rule, know the size of output buffer required to decompress a given file, so Hadoop must be configured by trial and error. This is badly designed, and harder to use.