kaitz / paq8pxd Goto Github PK
View Code? Open in Web Editor NEWLicense: GNU General Public License v2.0
License: GNU General Public License v2.0
Hi! This is an absolutely amazing compression program. It is the best ratio I've ever seen.
You say that files over 2 GB are not supported.
Do you think a 9 GB file would be possible to compress? Or would I be better off breaking it up using 7zip and compressing the pieces?
Hi, I'm testing this utility on a Windows 11 ARM VM and I can't get it to work, I tried to drag and drop a folder with an handful of jpegs inside, but the terminal window briefly appears and then automatically closes with no result.
If I manually call the script and pass the arguments it just exists with no output:
paq8pxd_v106.exe C:\test-folder test.paq8pxd
paq8pxd_v106.exe .\test-folder .\test.paq8pxd
paq8pxd_v106.exe C:\test-folder C:\test.paq8pxd
paq8pxd_v106.exe C:\test-folder
None of the above produced any output. What am I doing wrong?
I also tried with v105 but I get the same result.
Use struct's for all detectable data types. Examples are TextInfo and bmpInfo.
So something I've noticed during my tests is that the overall memory allocated to paq8pxd does not match up with what is being stated at the end when it finished.
Two examples compressed individually: one JPEG and one uncompressed 24-bit image.
Baseline JPEG (-s9):
Task Manager and Process Explorer reports 2.8GiB allocated.
Reported memory usage by paq8pxd is: 3712 MB (3893306956 bytes)
Uncompressed 24-bit Image (-s9):
Task Manager and Process Explorer reports 1.2GiB allocated.
Reported memory usage by paq8pxd is: 3764 MB (3947871882 bytes)
It gets even wilder at (-s15) where I can manage to trigger and "Out of Memory" error at the very end of the compression if I start doing other things on my system while it's running. This is why I'm thinking it's some kind of memory allocation issue.
I really like this compressor and was just wondering if you planned on updating it still.
Why? It makes it harder to find and/or fix issues if all the code is in a one file. Is it for perfomance (somehow)? Was the one-file style borrowed from older codebase (AFAIK paq1 is 22 years old)?
I found a file (possibly damaged) that doesn't restore losslessly
EDIT: This is NOT the correct file; see the update.
Right now they can only be found on the forum. It would be great if you could add them to the releases here.
paq8pxd has a small bug in the detection routine: it may happen when it skips an image -> it will stop detecting more.
https://encode.su/threads/1464-Paq8pxd-dict?p=70995&viewfull=1#post70995
There are some variables common. So when two threads try to use wrt transform then program will crash with SIGSEV.
$ g++ paq8pxd.cpp -DWINDOWS -DMT -msse2 -O3 -s -static -o paq8pxd.exe
paq8pxd.cpp:554:0: warning: "MT" redefined
#define MT //uncomment for multithreading, compression only
:0:0: note: this is the location of the previous definition
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17588): undefined reference to inflateInit_' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17630): undefined reference to
deflateInit2_'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x1787a): undefined reference to deflate' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x178f0): undefined reference to
inflate'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17bc5): undefined reference to deflateEnd' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17be0): undefined reference to
inflateEnd'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17de5): undefined reference to inflateInit_' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17e8c): undefined reference to
inflate'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17f6c): undefined reference to deflateEnd' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x17fc7): undefined reference to
inflateInit2_'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x18022): undefined reference to inflateInit2_' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x182c2): undefined reference to
deflateInit2_'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x183ad): undefined reference to deflate' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x184a4): undefined reference to
deflateEnd'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x2845a): undefined reference to inflate' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x28469): undefined reference to
inflateEnd'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x2b332): undefined reference to inflate' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0x2b376): undefined reference to
inflateEnd'
C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0xc773): undefined reference to inflateInit_' C:\msys64\tmp\cclTlQgb.o:paq8pxd.cpp:(.text+0xc793): undefined reference to
inflateInit2_'
collect2.exe: error: ld returned 1 exit status
I'm attempting to preprocess (-s0) compact disc images stored in (BIN+CUE) or (BIN+GDI) formats and paq8pxd v104 is producing a corrupted output.
paq8px v205fix1 doesn't seem to have this problem so I've been using that for now, but I'm running into the file size limit of 2GiB which is too limiting for what I want to use it for.
I am running Windows 10 and using the provided binary here on the github.
As a side note, anything I extract always produces this message:
"DeCompressing FileDisk: unable to open file (No such file or directory)"
A way to check archives without extracting them or somewhere along those lines.
The only way to verify the validity of an archive right now is to extract it and compare it to the original manually.
LZWDecode
Detecting MSZIP streams is present.
https://stackoverflow.com/questions/39390314/deflating-data-from-mszip-format
https://blogs.kde.org/2008/01/04/kcabinet-mostly-working
computer@computer-Mac-mini paq8pxd-master % g++ paq8pxd.cpp -O2 -DUNIX -s -lz -o paq8pxd
paq8pxd.cpp:694:50: warning: format specifies type 'unsigned long' but the argument has type 'unsigned long long' [-Wformat]
double(clock()-start_time)/CLOCKS_PER_SEC, ((maxmem)/1024)/1024,(maxmem));
^~~~~~~~~~~~~~~~~~~~
paq8pxd.cpp:694:71: warning: format specifies type 'unsigned long' but the argument has type 'U64' (aka 'unsigned long long') [-Wformat]
double(clock()-start_time)/CLOCKS_PER_SEC, ((maxmem)/1024)/1024,(maxmem));
^~~~~~~~
paq8pxd.cpp:2076:7: warning: expression result unused [-Wunused-value]
((U32 *) &tx[nx],a);
^ ~~~~~~~
paq8pxd.cpp:2076:23: warning: expression result unused [-Wunused-value]
((U32 *) &tx[nx],a);
^
paq8pxd.cpp:2081:7: warning: expression result unused [-Wunused-value]
((U64 *) &tx[nx],a);
^ ~~~~~~~
paq8pxd.cpp:2081:23: warning: expression result unused [-Wunused-value]
((U64 *) &tx[nx],a);
^
In file included from paq8pxd.cpp:2786:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/immintrin.h:40:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/wmmintrin.h:15:
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:36:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:56:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:76:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:96:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/__wmmintrin_aes.h:113:1: error: function definition is not allowed here
{
^
In file included from paq8pxd.cpp:2786:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/immintrin.h:44:
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/clflushoptintrin.h:21:34: error: function definition is not allowed here
_mm_clflushopt(void const * __m) {
^
In file included from paq8pxd.cpp:2786:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/immintrin.h:48:
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/clwbintrin.h:32:27: error: function definition is not allowed here
_mm_clwb(void const *__p) {
^
In file included from paq8pxd.cpp:2786:
In file included from /Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/immintrin.h:52:
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:61:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:79:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:97:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:115:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:134:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:153:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:171:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:189:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:208:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:227:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:246:1: error: function definition is not allowed here
{
^
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.3/include/avxintrin.h:265:1: error: function definition is not allowed here
{
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
6 warnings and 20 errors generated.
After hearing that it would be difficult for files above 2GiB to be supported, I'd figure I'd leave this here for reference for anyone to comment on.
paq8pxd, and by extension the other paq8 variants, have an incredibly powerful preprocessor that would benefit from having large file & archive support above 2GiB. This way you can use paq8pxd to preprocess your data and use a different compression scheme than the one included.
I'll be realistic here and say that I do not expect this to happen.
P.S. I am not a programmer so please don't berate me with comments like "add it yourself."
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.