Comments (3)
BZIP operates much better on small files. Lrzip on large ones. There are
1000 or so files and all are small. The long range compression routine
does not work as well on smaller files. Even though you tar the files
together, the bzip compression window is much smaller and operates more
efficiently with this type.
Interestingly, if you tar up the Kernel source tree, you get much better
results.
On 05/14/2016 02:41 PM, phiresky wrote:
See here:
|880M 14. Mai 21:25 files.tar 11M 14. Mai 21:25 files.tar.bz2 21M 14.
Mai 21:31 files.tar.lrz-bz2 22M 14. Mai 21:33 files.tar.lrz-lzma 51M
14. Mai 21:29 files.tar.lrz-nocompress 21M 14. Mai 21:29
files.tar.lrz-nocompress.bz2 19M 14. Mai 21:35 files.tar.lrz-zpaq |Using just bzip2, the 880MB file compresses to 11MB. Using |lrzip -b|,
it is 21MB.
How can this happen / should this happen?Here is the file (.zip so github accepts the upload):
files.tar.zip https://github.com/ckolivas/lrzip/files/264669/files.zip—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#51
Peter Hyman
http://www.linkedin.com/in/peterh
C: +1(609)598-0262
C: +1(612)440-7383 (Pete)
from lrzip.
The json files are very similar, they can basically be seen as a single file (same file structure, same words, just different order). The result stays the same when instead of tarring them up using cat *.json
.
from lrzip.
"Should this happen?" - on rare occasions the underlying compression does a better job on the file due to the tightly packed redundancy, and separating out the dictionary from the matches is counterproductive. The only thing lrzip offers for this particular archive is much faster compression and decompression.
from lrzip.
Related Issues (20)
- How to read the output HOT 1
- lrzip -t file.lrz fails when run from write-protected dir
- Streaming issue "No space left on device" in lrzip 0.651 HOT 1
- "Warning, low memory for chosen compression settings" for small target files HOT 12
- lrzip 0.651 test issue HOT 3
- Use of uninitialized memory bug HOT 1
- 41e8014 Add a -Q/--very-quiet option: Suppresses INFO. It shouldn't
- memory error in fill_buffer lrzip/stream.c HOT 1
- ZPAQ Segfault with incompressible blocks
- lrztar shows spurious "illegal option" with long options HOT 4
- heap-buffer-overflow in libzpaq/libzpaq.cpp:1208:25 libzpaq::PostProcessor::write(int) HOT 3
- Issue with -p or --threads HOT 12
- Incomplete fix of Issue #206 makes use-after-free still possible HOT 4
- "Unable to allocate enough memory for operation" on x86 when 10 threads are used. HOT 1
- Unable to stat file error on broken symlinks HOT 2
- Suggestion to support encrypted Stdin/Stdout when password provided on command line
- CPU detection does not account for CPU affinity HOT 8
- autoconf generates warnings on deprecated macros
- autoreconf: not found
- Makefile.am:34: error: Libtool library used but 'LIBTOOL' is undefined HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lrzip.