mcmilk / zstdmt

Multithreading Library for Brotli, Lizard, LZ4, LZ5, Snappy and Zstandard

Multithreading Library for Brotli, Lizard, LZ4, LZ5, Snappy-c, LZFSE and Zstandard

Description

• works with skippables frame id 0x184D2A50 (12 bytes per compressed frame)
• brotli is supported the same way, it will encapsulate the real brotli stream within an 16 byte frame header

Generic skippable frame definition

• the frame header for Lizard, LZ4, LZ5 and Zstandard is like this:

size	value	description
4 bytes	0x184D2A50U	magic for skippable frame
4 bytes	4	size of skippable frame
4 bytes	compressed size	size of the following frame (compressed data)

Brotli frame definition

• the frame header for brotli is defined a bit different:

size	value	description
4 bytes	0x184D2A50U	magic for skippable frame (like zstd)
4 bytes	8	size of skippable frame
4 bytes	compressed size	size of the following frame (compressed data)
2 bytes	0x5242U	magic for brotli "BR"
2 bytes	uncompressed size	allocation hint for decompressor (64KB * this size)

Snappy-c frame definition

• the frame header for Snappy-c is defined a bit different:

size	value	description
4 bytes	0x184D2A50U	magic for skippable frame (like zstd)
4 bytes	8	size of skippable frame
4 bytes	compressed size	size of the following frame (compressed data)
2 bytes	0x5053U	magic for Snappy-c "SP"
2 bytes	uncompressed size	allocation hint for decompressor (64KB * this size)

LZFSE frame definition

• the frame header for Lzfse is defined a bit different:

size	value	description
4 bytes	0x184D2A50U	magic for skippable frame (like zstd)
4 bytes	8	size of skippable frame
4 bytes	compressed size	size of the following frame (compressed data)
2 bytes	0x464CU	magic for Lzfse "LF"
2 bytes	uncompressed size	allocation hint for decompressor (64KB * this size)

Usage of the Testutils

• see programs

Usage of the Library

• see lib

/TR 2020-10-15

zstdmt's Issues

Add support for newer large window features.

Brotli has implemented a new feature "Large window brotli" can support of the feature through zstdmt be possible? In my tests it gave brotli a nice improvement in ratio.

--keep should be default

brotli-mt window size option

There is an option in brotli to use the window size. I couldn't find window option in brotli-mt.

Is there a way to specify the window size for brotli-mt and does it use LZ77?

  -w NUM, --lgwin=NUM         set LZ77 window size (0, 10-24)
                              window size = 2**NUM - 16
                              0 lets compressor choose the optimal value

And what does frame refer to in your code?

Thanks!

Long windows aren't supported

I.e. --large_window for brotli and --long for zstd.

Version information

Version information is somewhat confusing. It says: zstd-mt version v1.5.5, zstdmt v0.8, where 1.5.5 is in fact zstd version, and 0.8 is zstd-mt.
In my opinion it should be program version first, then libzstd version second. Perfectly in next line.

zstd-mt 0.8
libzstd 1.5.5

Copyright...

It's logical to display actual program version first, then its library.

Other program/s, using external library in same/similar way do it like that:

plzip 1.10
Using lzlib 1.14
Copyright...

Same would apply to the other programs from suite.

new version of lz4.

Unable to decompress with the original brotli？

When I compress with brotli-mt, brotli-mt can decompress while brotli can not decompress

brotli-mt test
brotli -d -S brot test.brot
Corrupt input [test. brot]

brotli-mt: Make -k default because why would you delete the input file on default????

Zero length Windows EXEs

All of the Windows releases (v0.6 - 0.8) appear to contain multiple zero-byte binaries.

snappy multithreading

Are there plans to add compression algorithms? I want to help. And I made snappy multithreading.

Any plan to support Linux platform? ( such as Debian, Arch, etc )

As title.

Checksum

Is it possible to add checksum to the archive? At least optionally.