kjn / lbzip2 Goto Github PK

View Code? Open in Web Editor NEW

130.0 130.0 17.0 2.57 MB

Parallel bzip2 utility

License: GNU General Public License v3.0

Shell 1.02% Perl 2.50% C 94.18% Makefile 0.96% M4 1.35%

lbzip2's People

Contributors

Stargazers

Watchers

Forkers

orf1888 jlec vinsonzou ofer4321 weewillie shimondoodkin apergos zbhfut khuramali isgasho logological mattst88 sssseb fweimer-rh aonez cs558i maxugly

lbzip2's Issues

Fails to build because gnulib is missing

Formally present gnulib is removed, which triggers build errors. Is this intended?

lbzip2-2.3/src/common.h:33:48: fatal error: xalloc.h: No such file or directory
#include "xalloc.h" /* XMALLOC() */
^

Limit thread upper bound

We can specify the thread number to be applied with -n. However there is no upper bound limit for it. If the value is huge, lbzip2 will report unable to create a POSIX thread: Resource temporarily unavailable and hang there.

We think it's reasonable to have some upper bound check for the thread number.

Does not build on Fedora Rawhide

Hello there!

Upcoming changes in glibc 2.28 broke the package.

https://koji.fedoraproject.org/koji/taskinfo?taskID=28542910

https://kojipkgs.fedoraproject.org//work/tasks/2910/28542910/build.log

gcc -DHAVE_CONFIG_H -I.     -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -c -o printf-args.o printf-args.c
fseterr.c: In function 'fseterr':
fseterr.c:77:3: error: #error "Please port gnulib fseterr.c to your platform! Look at the definitions of ferror and clearerr on your system, then report this to bug-gnulib."
  #error "Please port gnulib fseterr.c to your platform! Look at the definitions of ferror and clearerr on your system, then report this to bug-gnulib."
   ^~~~~
make[3]: *** [Makefile:1474: fseterr.o] Error 1
make[3]: *** Waiting for unfinished jobs....

Thanks for help.

lbzip2 changes file permission

When I'm not the owner of a file and when using lbzip2 as a compression tool, the file permissions get changed! Can this be fixed?

verbose output not working on iMac

Dear,

Using lbzip2 from the command line fails when adding either -v or --verbose to the command. In that case, it makes a bz2 file, but in there, there is only a .cpgz file to be found. The command line throws out the following error: Illegal instruction: 4.

This is on High Sierra with an iMac 21.5-inch, Mid 2011 with an 2,8 GHz Intel Core i7.

I tried to compile it myself and to install it using homebrew with no difference whatsoever.

The same iMac running on Yosemite does the job as it should, so I suppose this problem is due to an instruction that High Sierra doesn't understand.

Any way to fix this?

Thank you in advance,
Erwin

Make sure that dummy trees meet Kraft equality

Some bz2 decoders (at least 7-Zip) may enforce Kraft equality. Lets make sure that dummy trees (generated in case only one tree is used in a block) are complete.

lbzip2-2.0 - Segf with args -d -c --

On my gentoo system with lbzip2-2.0 I get an segf for

lbzip2 -d -c -- automake-1.11.1.tar.bz2 > automake-1.11.1.tar

(gdb) set args -d -c -- automake-1.11.1.tar.bz2 > asd
(gdb) run
Starting program: /usr/bin/lbzip2 -d -c -- automake-1.11.1.tar.bz2 > asd
[Thread debugging using libthread_db enabled]
[New Thread 0x2aaaab698700 (LWP 32031)]
[New Thread 0x2aaaab899700 (LWP 32032)]
[New Thread 0x2aaaaba9a700 (LWP 32033)]
[Thread 0x2aaaab899700 (LWP 32032) exited]
[New Thread 0x2aaaabd9c700 (LWP 32034)]
[New Thread 0x2aaab0200700 (LWP 32035)]
[New Thread 0x2aaab05cf700 (LWP 32036)]
[New Thread 0x2aaab07d0700 (LWP 32037)]
[New Thread 0x2aaab09d1700 (LWP 32038)]
[New Thread 0x2aaab0bd2700 (LWP 32039)]
[New Thread 0x2aaab0dd3700 (LWP 32040)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x2aaaaba9a700 (LWP 32033)]
work_retrieve (ispec=0x7fffffffc5a0, w2m_q=, sw2w_q=0x2aaaab697c40, ibits_left=8, ibitbuf=134217728, ipos=4528, s2w_blk=0x2aaaabd9d010) at lbunzip2.c:1118
1118 lbunzip2.c: No such file or directory.
in lbunzip2.c
(gdb) bt
#0 work_retrieve (ispec=0x7fffffffc5a0, w2m_q=, sw2w_q=0x2aaaab697c40, ibits_left=8, ibitbuf=134217728, ipos=4528, s2w_blk=0x2aaaabd9d010) at lbunzip2.c:1118
#1 work_scan (ispec=0x7fffffffc5a0, w2m_q=, sw2w_q=0x2aaaab697c40, s2w_blk=) at lbunzip2.c:1316
#2 work (ispec=0x7fffffffc5a0, w2m_q=, sw2w_q=0x2aaaab697c40) at lbunzip2.c:1330
#3 work_wrap (v_work_arg=) at lbunzip2.c:1359
#4 0x00002aaaaaedff1c in start_thread () from /lib64/libpthread.so.0
#5 0x00002aaaab1d921d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

version 0.23 is fine.

Portage 2.2.0_alpha77 (default/linux/amd64/10.0, gcc-4.6.2-asneeded, glibc-2.14.1-r0, 3.1.1-lh x86_64)

System uname: Linux-3.1.1-lh-x86_64-Intel-R-_Core-TM-i7_CPU_860@_2.80GHz-with-gentoo-2.1
Timestamp of tree: Wed, 23 Nov 2011 07:00:01 +0000
ccache version 3.1.6 [enabled]
app-shells/bash: 4.2_p10
dev-java/java-config: 2.1.11-r3
dev-lang/python: 2.5.4-r4, 2.6.7-r2, 2.7.2-r3, 3.1.4-r3, 3.2.2
dev-util/ccache: 3.1.6
dev-util/cmake: 2.8.6-r3
dev-util/pkgconfig: 0.26
sys-apps/baselayout: 2.1
sys-apps/openrc: 0.9.4
sys-apps/sandbox: 2.5
sys-devel/autoconf: 2.13, 2.68
sys-devel/automake: 1.9.6-r3, 1.10.3, 1.11.1-r1
sys-devel/binutils: 2.22
sys-devel/gcc: 4.2.4-r1, 4.3.6-r1, 4.4.6-r1, 4.5.3-r1, 4.6.2
sys-devel/gcc-config: 1.5-r2
sys-devel/libtool: 2.4.2
sys-devel/make: 3.82-r3
sys-kernel/linux-headers: 2.6.39 (virtual/os-headers)
sys-libs/glibc: 2.14.1

[license] [question] Consider switching to LGPL

I was wondering, if it would be possible to use the library under the terms of the LGPL instead of the GPL for open source projects.

Trouble is, that I don't want to use GPL, but more like BSD as a license for my stuff and I'm worried, that I had to relicense the code according to GPL when using this library. Thus, I'm trying to find out, if dynamic linking and LGPL would work.

lbzip2.org it not resolving

host lbzip2.org
Host lbzip2.org not found: 3(NXDOMAIN)

Has been this way for a few days at least. Perhaps host the bootstrapped tarball elsewhere?

macOS: lbzip2 -v crashes with "illegal hardware instruction"

Using the parameter -v causes lbzip2 to crash for me. Weirdly after that happens, any call to lbzip2 crashes for a while. After a few minutes (and doing something else) it works again.

$ lbzip2 -v README.md
[1]    80520 illegal hardware instruction  lbzip2 -v README.md
$ lbzip2 README.md
[1]    80755 illegal hardware instruction  lbzip2 README.md

This is on lbzip2 2.5 installed from Homebrew. It's built with ./configure --disable-debug --disable-dependency-tracking in case that matters.

Here's the crash log:

Process:               lbzip2 [80520]
Path:                  /usr/local/Cellar/lbzip2/2.5/bin/lbzip2
Identifier:            lbzip2
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        zsh [70006]
Responsible:           lbzip2 [80520]
User ID:               501

Date/Time:             2018-04-25 12:44:04.527 +0300
OS Version:            Mac OS X 10.13.2 (17C88)
Report Version:        12
Bridge OS Version:     3.0 (14Y661)
Anonymous UUID:        E44E2E84-CDBC-B3B7-16B1-EB8100EA5BA1

Sleep/Wake UUID:       06491AE0-82A8-407A-97AF-E1E3867DB275

Time Awake Since Boot: 1900000 seconds
Time Since Wake:       82000 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_INSTRUCTION (SIGILL)
Exception Codes:       0x0000000000000001, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Illegal instruction: 4
Termination Reason:    Namespace SIGNAL, Code 0x4
Terminating Process:   exc handler [0]

Application Specific Information:
%n used in a non-immutable format string

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_c.dylib             	0x00007fff6e1f3a0b __vfprintf + 16437
1   libsystem_c.dylib             	0x00007fff6e218091 __v2printf + 473
2   libsystem_c.dylib             	0x00007fff6e1fd4af _vsnprintf + 415
3   libsystem_c.dylib             	0x00007fff6e1fd50c vsnprintf_l + 41
4   libsystem_c.dylib             	0x00007fff6e1ee4d4 snprintf + 180
5   lbzip2                        	0x000000010dc5a966 vasnprintf + 3094
6   lbzip2                        	0x000000010dc58d55 rpl_fprintf + 185
7   lbzip2                        	0x000000010dc551f4 log_generic + 61
8   lbzip2                        	0x000000010dc5518c info + 170
9   lbzip2                        	0x000000010dc576a8 work + 97
10  lbzip2                        	0x000000010dc56642 main + 2767
11  libdyld.dylib                 	0x00007fff6e15c115 start + 1

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000000  rbx: 0x0000000000000000  rcx: 0x0000050000000000  rdx: 0x0000040000000503
  rdi: 0x0000000000012068  rsi: 0x0000000000000500  rbp: 0x00007ffee1fb59f0  rsp: 0x00007ffee1fb5670
   r8: 0x00007fffa7245da0   r9: 0x0000000000000040  r10: 0x00007fffa7245d98  r11: 0xffffffffffffffff
  r12: 0x00007ffee1fb5dc4  r13: 0x0000000000000000  r14: 0x0000000000000006  r15: 0x00007ffee1fb61f8
  rip: 0x00007fff6e1f3a0b  rfl: 0x0000000000010202  cr2: 0x000000010dc7c000
  
Logical CPU:     4
Error Code:      0x00000000
Trap Number:     6


Binary Images:
       0x10dc48000 -        0x10dc61fff +lbzip2 (0) <A2637AA1-9432-3466-B680-E2313C060313> /usr/local/bin/lbzip2
       0x1120ed000 -        0x11213798f  dyld (519.2.2) <6695F30B-4E88-3C0B-9867-7D738C44A3E6> /usr/lib/dyld
    0x7fff6ba45000 -     0x7fff6ba78fff  libclosured.dylib (519.2.2) <48051216-5647-3643-B979-B77D0FD20011> /usr/lib/closure/libclosured.dylib
    0x7fff6bf57000 -     0x7fff6bf58ff3  libSystem.B.dylib (1252) <5EDEEA85-0387-31EE-9D9B-DA1B75E7AEF3> /usr/lib/libSystem.B.dylib
    0x7fff6c18b000 -     0x7fff6c1e1fff  libc++.1.dylib (400.9) <FCF5E1F6-2B04-3545-8004-F3AB32FED172> /usr/lib/libc++.1.dylib
    0x7fff6c1e2000 -     0x7fff6c206ff7  libc++abi.dylib (400.7) <217656D5-BC40-37FF-B322-91CB2AAD4F34> /usr/lib/libc++abi.dylib
    0x7fff6d558000 -     0x7fff6d9467e7  libobjc.A.dylib (723) <93A92316-DE1E-378C-8891-99720B50D075> /usr/lib/libobjc.A.dylib
    0x7fff6dff3000 -     0x7fff6dff7ff7  libcache.dylib (80) <354F3B7D-404E-3398-9EBF-65CA2CE65211> /usr/lib/system/libcache.dylib
    0x7fff6dff8000 -     0x7fff6e002ff3  libcommonCrypto.dylib (60118.30.2) <674286D3-7744-36A3-9AAA-49DFCD97A986> /usr/lib/system/libcommonCrypto.dylib
    0x7fff6e003000 -     0x7fff6e00afff  libcompiler_rt.dylib (62) <4487CFBA-A5D7-3282-9E6B-94CAD7BE507E> /usr/lib/system/libcompiler_rt.dylib
    0x7fff6e00b000 -     0x7fff6e013ffb  libcopyfile.dylib (146.30.2) <2C7C67D7-562B-3FFA-973D-BACF4C10E1EC> /usr/lib/system/libcopyfile.dylib
    0x7fff6e014000 -     0x7fff6e099fff  libcorecrypto.dylib (562.30.10) <8A53EFE1-AFCA-3676-BEE1-FA5ED9F0E222> /usr/lib/system/libcorecrypto.dylib
    0x7fff6e121000 -     0x7fff6e15aff7  libdispatch.dylib (913.30.4) <7D0E3183-282B-3FEE-A734-2C0ADC092084> /usr/lib/system/libdispatch.dylib
    0x7fff6e15b000 -     0x7fff6e178ff7  libdyld.dylib (519.2.2) <C50D02BC-A333-3313-B787-02F255A6783F> /usr/lib/system/libdyld.dylib
    0x7fff6e179000 -     0x7fff6e179ffb  libkeymgr.dylib (28) <6D84A96F-C65B-38EC-BDB5-21FD2C97E7B2> /usr/lib/system/libkeymgr.dylib
    0x7fff6e187000 -     0x7fff6e187ff7  liblaunch.dylib (1205.30.29) <E66F58ED-C15E-3DFB-BC22-A861E13918C6> /usr/lib/system/liblaunch.dylib
    0x7fff6e188000 -     0x7fff6e18cffb  libmacho.dylib (900.0.1) <756F2553-07B6-3B42-ACEA-2F0F1A5E8D0F> /usr/lib/system/libmacho.dylib
    0x7fff6e18d000 -     0x7fff6e18fff3  libquarantine.dylib (86) <6AC8773F-3817-3D82-99C2-01BABB9C3CBB> /usr/lib/system/libquarantine.dylib
    0x7fff6e190000 -     0x7fff6e191ff3  libremovefile.dylib (45) <912FA211-DD8C-3C92-8424-21B89F8B10FD> /usr/lib/system/libremovefile.dylib
    0x7fff6e192000 -     0x7fff6e1a9fff  libsystem_asl.dylib (356.1.1) <94972913-9DF0-3C78-847C-43E58919E3DA> /usr/lib/system/libsystem_asl.dylib
    0x7fff6e1aa000 -     0x7fff6e1aafff  libsystem_blocks.dylib (67) <F2493BB5-B1C6-3C4D-9F1F-1B402E0F1DB7> /usr/lib/system/libsystem_blocks.dylib
    0x7fff6e1ab000 -     0x7fff6e234ff7  libsystem_c.dylib (1244.30.3) <E0136C71-0648-36F0-9F84-82EA2748A8D7> /usr/lib/system/libsystem_c.dylib
    0x7fff6e235000 -     0x7fff6e238ffb  libsystem_configuration.dylib (963.30.1) <0F8D0B76-4F7D-34EC-AB6C-50F9465809DA> /usr/lib/system/libsystem_configuration.dylib
    0x7fff6e239000 -     0x7fff6e23cffb  libsystem_coreservices.dylib (51) <21A488D0-2D07-344E-8631-CC8B2A246F35> /usr/lib/system/libsystem_coreservices.dylib
    0x7fff6e23d000 -     0x7fff6e23efff  libsystem_darwin.dylib (1244.30.3) <2F750CB1-BC26-3FA3-AE59-553EE30D451B> /usr/lib/system/libsystem_darwin.dylib
    0x7fff6e23f000 -     0x7fff6e245ff7  libsystem_dnssd.dylib (878.30.4) <EB9BB165-45A4-367C-B33A-688D4F383A95> /usr/lib/system/libsystem_dnssd.dylib
    0x7fff6e246000 -     0x7fff6e28fff7  libsystem_info.dylib (517.30.1) <7D79E167-4B5C-3833-81EE-3AF3FB53616D> /usr/lib/system/libsystem_info.dylib
    0x7fff6e290000 -     0x7fff6e2b5ff7  libsystem_kernel.dylib (4570.31.3) <D2E842AA-3B8D-31BF-8234-8C1BE11CFE32> /usr/lib/system/libsystem_kernel.dylib
    0x7fff6e2b6000 -     0x7fff6e301fcb  libsystem_m.dylib (3146) <ABB1B85F-9FFE-31B8-AD4F-E39A30794A93> /usr/lib/system/libsystem_m.dylib
    0x7fff6e302000 -     0x7fff6e321fff  libsystem_malloc.dylib (140.1.1) <9F0745FF-B92F-330D-8812-BB74001D1D33> /usr/lib/system/libsystem_malloc.dylib
    0x7fff6e322000 -     0x7fff6e3c6ff3  libsystem_network.dylib (1229.30.11) <40BAD301-8744-3AD8-A688-E7925C587B00> /usr/lib/system/libsystem_network.dylib
    0x7fff6e3c7000 -     0x7fff6e3d1ffb  libsystem_networkextension.dylib (767.30.7) <C5ED4BAB-2480-369D-8AB6-DFB37C15A1EE> /usr/lib/system/libsystem_networkextension.dylib
    0x7fff6e3d2000 -     0x7fff6e3dbff3  libsystem_notify.dylib (172) <98EA3D62-7C86-30DE-8261-D020D2F1EFF3> /usr/lib/system/libsystem_notify.dylib
    0x7fff6e3dc000 -     0x7fff6e3e3ff7  libsystem_platform.dylib (161.20.1) <C049250F-8C35-314D-810F-4E28AEAED983> /usr/lib/system/libsystem_platform.dylib
    0x7fff6e3e4000 -     0x7fff6e3effff  libsystem_pthread.dylib (301.30.1) <ABA848E1-6978-3B42-A3A7-608B2C36FA93> /usr/lib/system/libsystem_pthread.dylib
    0x7fff6e3f0000 -     0x7fff6e3f3ff3  libsystem_sandbox.dylib (765.30.4) <A51828BA-26D8-3205-AF3A-DF5C37269468> /usr/lib/system/libsystem_sandbox.dylib
    0x7fff6e3f4000 -     0x7fff6e3f5ff3  libsystem_secinit.dylib (30) <F06ADB8F-9E94-34A7-B3C9-2C22FDD14BAD> /usr/lib/system/libsystem_secinit.dylib
    0x7fff6e3f6000 -     0x7fff6e3fdff7  libsystem_symptoms.dylib (820.30.7) <DC3586C2-AA56-3419-88D3-FC0DBF08E3C0> /usr/lib/system/libsystem_symptoms.dylib
    0x7fff6e3fe000 -     0x7fff6e411ff7  libsystem_trace.dylib (829.30.14) <69EBF017-D40F-30D7-9B0B-BFC862D761A5> /usr/lib/system/libsystem_trace.dylib
    0x7fff6e413000 -     0x7fff6e418ff7  libunwind.dylib (35.3) <6D4FCD49-D2A9-3233-95C7-A7635CE265F2> /usr/lib/system/libunwind.dylib
    0x7fff6e419000 -     0x7fff6e445ff7  libxpc.dylib (1205.30.29) <F7E5F1BC-614B-39CB-B6CE-92A9C7B7EC0B> /usr/lib/system/libxpc.dylib

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 19972451
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=197.0M resident=0K(0%) swapped_out_or_unallocated=197.0M(100%)
Writable regions: Total=17.3M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=17.3M(100%)
 
                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
Kernel Alloc Once                    8K        2 
MALLOC                            9276K        8 
MALLOC guard page                   16K        5 
STACK GUARD                       56.0M        2 
Stack                             8192K        2 
__DATA                            2324K       43 
__LINKEDIT                       188.3M        4 
__TEXT                            8976K       42 
shared memory                        8K        3 
===========                     =======  ======= 
TOTAL                            272.4M      102

divbwt() arguments

Can someone explain what are the arguments to divbwt in encode.c?

//-- encode.c --

  uint8_t *block = (void *)(s->SA + s->max_block_size + GROUP_SIZE);
  (...)
  s->bwt_idx = divbwt(block, s->SA, s->u.bucket, s->nblock);

I tried to test it with sais or libsais and it didn't work according to a plan. First i read a bit about them, found en example, and prepared tenporary buffer (Atmp) of 4 blocks size then tried to call it as shown below but it didn't work.

//-- encode.c-sais --

  int* Atmp = malloc(bs100k*100000u*sizeof(int));
  (...)
  /* s->bwt_idx = divbwt(block, s->SA, s->u.bucket, s->nblock); */
  /*// sais_u8_bwt(T, T, SA, (sa_int32_t)m, 256)) //*/
  s->bwt_idx = sais_u8_bwt(block, block, s->SA, s->nblock, 256); /// somewhat corrupted
  s->bwt_idx = sais_u8_bwt(block, s->SA, s->u.bucket, s->nblock, 256); /// Segmentation fault
  s->bwt_idx = sais_u8_bwt(block, s->SA, Atmp, s->nblock, 256); /// Segmentation fault

Here, the first example (somewhat corrupted) is the only that didn't segfault but still it's corrupted (repeats short piece of encoded message; I guess it's too small buffer but why the others with Atmp fail so miserably?

-- encode.c-libsais

  int* Atmp = malloc(bs100k*100000u*sizeof(int));
  (...)
  /* s->bwt_idx = divbwt(block, s->SA, s->u.bucket, s->nblock); */
  s->bwt_idx = libsais_bwt(block, s->SA, s->u.bucket, s->nblock); ///const int idx=libsais_bwt(buf, buf, ptr, n);
  s->bwt_idx = libsais_bwt(block, s->SA, Atmp, s->nblock); ///const int idx=libsais_bwt(buf, buf, ptr, n);

Ed: I realised that there is something like discussion on github and this shall be moved there. The only thing is it's not enabled in this repo. If some maintainer could enable out and move the thread to discussion, that would be better, I think.

"lbzcat a.bz2 > a" is much slower than "lbzcat a.bz2 >! a"

I found an interesting thing,

if a file a does already exist,
then

lbzcat a.bz2 > a
is much much slower than
lbzcat a.bz2 >! a
and
rm -f a; lbzcat a.bz2 > a

Why?

porting to windows

I had tried to compile it on MinGW, but it didn't work out because of signals

MinGW has only a small amount of signals

those are missing:
I had defined them at the bottom of common.h , not sure what is the effect of this.

define SIGUSR1 26

define SIGUSR2 27

define SIGXFSZ 28

ideas for signals alternative:
http://www.perlmonks.org/?node_id=1022364
https://social.msdn.microsoft.com/Forums/vstudio/en-US/8d128695-1727-434b-9fd1-7b74b2bf25ea/need-a-better-way-for-inter-thread-communication-?forum=vcgeneral

also had an error:

signals.c:78:20: error: storage size of 'act' isn't known
struct sigaction act;

migw's signal.h

#ifndef _INC_SIGNAL
#define _INC_SIGNAL

#include <crtdefs.h>
#include <pthread_signal.h>

#ifdef __cplusplus
extern "C" {
#endif

#ifndef _SIG_ATOMIC_T_DEFINED
#define _SIG_ATOMIC_T_DEFINED
  typedef int sig_atomic_t;
#endif

#define NSIG 23

#define SIGINT 2
#define SIGILL 4
#define SIGABRT_COMPAT 6
#define SIGFPE 8
#define SIGSEGV 11
#define SIGTERM 15
#define SIGBREAK 21
#define SIGABRT 22       /* used by abort, replace SIGIOT in the future */
#define SIGABRT2 22

#ifdef _POSIX
#define SIGHUP  1   /* hangup */
#define SIGQUIT 3   /* quit */
#define SIGTRAP 5   /* trace trap (not reset when caught) */
#define SIGIOT  6       /* IOT instruction */
#define SIGEMT  7   /* EMT instruction */
#define SIGKILL 9   /* kill (cannot be caught or ignored) */
#define SIGBUS  10  /* bus error */
#define SIGSYS  12  /* bad argument to system call */
#define SIGPIPE 13  /* write on a pipe with no one to read it */
#ifdef __USE_MINGW_ALARM
#define SIGALRM 14  /* alarm clock */
#endif
#endif

  typedef   void (*__p_sig_fn_t)(int);

#define SIG_DFL (__p_sig_fn_t)0
#define SIG_IGN (__p_sig_fn_t)1
#define SIG_GET (__p_sig_fn_t)2
#define SIG_SGE (__p_sig_fn_t)3
#define SIG_ACK (__p_sig_fn_t)4
#define SIG_ERR (__p_sig_fn_t)-1

  extern void **__cdecl __pxcptinfoptrs(void);
#define _pxcptinfoptrs (*__pxcptinfoptrs())

  __p_sig_fn_t __cdecl signal(int _SigNum,__p_sig_fn_t _Func);
  int __cdecl raise(int _SigNum);

#ifdef __cplusplus
}
#endif
#endif

pthread_signal.h

#ifndef WIN_PTHREADS_SIGNAL_H
#define WIN_PTHREADS_SIGNAL_H

/* Windows has rudimentary signals support.  */
#define pthread_sigmask(H, S1, S2) 0

#endif /* WIN_PTHREADS_SIGNAL_H */

just to give some sense about the errors:
the last one i did not knew how to solve
.


$ make
Making all in lib
make[1]: Entering directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
  GEN      alloca.h
  GEN      errno.h
  GEN      c++defs.h
  GEN      arg-nonnull.h
  GEN      warn-on-use.h
  GEN      fcntl.h
  GEN      inttypes.h
  GEN      math.h
  GEN      stdio.h
  GEN      stdlib.h
  GEN      string.h
  GEN      sys/stat.h
  GEN      sys/time.h
  GEN      sys/types.h
  GEN      time.h
  GEN      unistd.h
  GEN      wchar.h
make  all-recursive
make[2]: Entering directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
make[3]: Entering directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
  CC       dtotimespec.o
  CC       gettime.o
  CC       math.o
  CC       printf-frexp.o
  CC       printf-frexpl.o
  CC       stat-time.o
  CC       timespec.o
  CC       timespec-add.o
  CC       timespec-sub.o
  CC       unistd.o
  CC       utimens.o
  CC       xmalloc.o
  CC       xsize.o
  CC       asnprintf.o
  CC       fprintf.o
  CC       frexp.o
  CC       frexpl.o
  CC       fseterr.o
  CC       fstat.o
  CC       gettimeofday.o
  CC       isnanl.o
  CC       malloc.o
  CC       msvc-inval.o
  CC       printf-args.o
  CC       printf-parse.o
  CC       stat.o
  CC       vasnprintf.o
  AR       libgnu.a
make[3]: Leaving directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
make[2]: Leaving directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
make[1]: Leaving directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/lib'
Making all in src
make[1]: Entering directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/src'
  CC       compress.o
  CC       crctab.o
  CC       decode.o
  CC       divbwt.o
  CC       encode.o
encode.c: In function 'encoder_init':
encode.c:121:3: warning: implicit declaration of function 'bzero' [-Wimplicit-function-declaration]
   bzero(s->cmap, 256u * sizeof(bool));
   ^~~~~
encode.c:121:3: warning: incompatible implicit declaration of built-in function 'bzero'
encode.c: In function 'generate_initial_trees':
encode.c:829:5: warning: incompatible implicit declaration of built-in function 'bzero'
     bzero(&s->length[t][a], b - a);
     ^~~~~
encode.c: In function 'assign_codes':
encode.c:901:3: warning: incompatible implicit declaration of built-in function 'bzero'
   bzero(tree, sizeof(tree));
   ^~~~~
encode.c: In function 'generate_prefix_code':
encode.c:1060:5: warning: incompatible implicit declaration of built-in function 'bzero'
     bzero(s->frequency, nt * sizeof(*s->frequency));
     ^~~~~
  CC       expand.o
expand.c: In function 'on_input_avail':
expand.c:842:3: warning: implicit declaration of function 'bzero' [-Wimplicit-function-declaration]
   bzero((char *)buffer + size, missing);
   ^~~~~
expand.c:842:3: warning: incompatible implicit declaration of built-in function 'bzero'
  CC       main.o
main.c: In function 'info':
main.c:107:5: warning: implicit declaration of function 'flockfile' [-Wimplicit-function-declaration]
     flockfile(stderr);                          \
     ^
main.c:123:1: note: in expansion of macro 'DEF'
 DEF(info   (                                 const char *fmt, ...), 0,0,0,0,1)
 ^~~
main.c:117:7: warning: implicit declaration of function 'funlockfile' [-Wimplicit-function-declaration]
       funlockfile(stderr);                      \
       ^
main.c:123:1: note: in expansion of macro 'DEF'
 DEF(info   (                                 const char *fmt, ...), 0,0,0,0,1)
 ^~~
main.c: In function 'xstrtol':
main.c:173:12: warning: implicit declaration of function 'index' [-Wimplicit-function-declaration]
   endptr = index(suffix, *endptr);
            ^~~~~
main.c:173:12: warning: incompatible implicit declaration of built-in function 'index'
main.c: In function 'output_regf_uninit':
main.c:867:13: warning: implicit declaration of function 'fchown' [-Wimplicit-function-declaration]
   if (-1 == fchown(outfd, sbuf->st_uid, sbuf->st_gid)) {
             ^~~~~~
main.c:877:15: warning: implicit declaration of function 'fchmod' [-Wimplicit-function-declaration]
     if (-1 == fchmod(outfd, sbuf->st_mode & (S_IRWXU | S_IRWXG | S_IRWXO))) {
               ^~~~~~
  CC       parse.o
  CC       process.o
  CC       signals.o
signals.c: In function 'xempty':
signals.c:37:7: warning: implicit declaration of function 'sigemptyset' [-Wimplicit-function-declaration]
   if (sigemptyset(set) != 0)
       ^~~~~~~~~~~
signals.c: In function 'xadd':
signals.c:44:7: warning: implicit declaration of function 'sigaddset' [-Wimplicit-function-declaration]
   if (sigaddset(set, sig) != 0)
       ^~~~~~~~~
signals.c: In function 'xpending':
signals.c:58:7: warning: implicit declaration of function 'sigpending' [-Wimplicit-function-declaration]
   if (sigpending(set) != 0)
       ^~~~~~~~~~
signals.c: In function 'xmember':
signals.c:68:8: warning: implicit declaration of function 'sigismember' [-Wimplicit-function-declaration]
   rv = sigismember(set, sig);
        ^~~~~~~~~~~
signals.c: In function 'xaction':
signals.c:78:20: error: storage size of 'act' isn't known
   struct sigaction act;
                    ^~~
signals.c:84:7: warning: implicit declaration of function 'sigaction' [-Wimplicit-function-declaration]
   if (sigaction(sig, &act, NULL) != 0)
       ^~~~~~~~~
signals.c: In function 'halt':
signals.c:232:9: warning: implicit declaration of function 'sigsuspend' [-Wimplicit-function-declaration]
   ret = sigsuspend(&saved);
         ^~~~~~~~~~
signals.c: In function 'xraise':
signals.c:257:7: warning: implicit declaration of function 'kill' [-Wimplicit-function-declaration]
   if (kill(pid, sig) != 0)
       ^~~~
signals.c: In function 'bailout':
signals.c:308:1: warning: 'noreturn' function does return
 }
 ^
make[1]: *** [Makefile:1342: signals.o] Error 1
make[1]: Leaving directory '/c/Users/user/Downloads/lbzip2-2.5/lbzip2-2.5/src'
make: *** [Makefile:1308: all-recursive] Error 1

Delete bz2 file when compression fails

Using 2.3, when #14 happens, the bz2 file is not deleted:

$ lbzip2 beic-010.10.tar
lbzip2: skipping "beic-010.10.tar": open("beic-010.10.tar.bz2"): File exists
$ ls -lh beic-010.10*
-rw-rw-r-- 1 fondazione fondazione 4,1G giu 20 11:22 beic-010.10.tar
-rw------- 1 fondazione fondazione  25M giu 20 11:38 beic-010.10.tar.bz2

fails to decompress some valid bzip2 files

Bug reported under Debian BTS:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=582476

lib directory missing in HEAD

The lib directory that contains Gnulib C files seems missing from this repository, despite that it's there inside release versions:http://lbzip2.org/download.

Spammy post at lbzip2.org

The top post on the lbzip2.org website is The four most popular Colombian women on Onlyfans: one of them charges over $100,000 for racy photos and videos. All the other posts are technical content about lbzip2 and this one is out of place.

lbzip2-2* fails to completly unpack pytz tarball

And again, here the downstream report
https://bugs.gentoo.org/show_bug.cgi?id=393155

Simply download the latest version from here
http://pytz.sourceforge.net/
and extract the archive. It fails to extract pytz/init.py

lbzip2 thread crashes with signal SIGSEGV

Hello,

I have a bug report here, any clues?

https://bugzilla.redhat.com/show_bug.cgi?id=1017957

Speedup opportunity via vector shuffle

@bonzini forwarded me a request to contribute this here. Thanks Paolo, and thank you Mikolaj for lbzip2.

There's an opportunity to significantly speed up the common case in the inverse MTF transform where the reordered index is in the first sliding window (i.e. 0..15). Instead of taking a poorly predictable branch, we can load the entire window, shuffle it on registers, and write it back. With SSSE3 this boils down to a load-pshufb-store sequence, but could be done as well by shifting/masking on general registers (obviously more instructions, but still should be an improvement over the current branchy code).

I coded the following as a personal exercise. Probably not upstreamable, would like to hear what the requirements for a proper patch might be. If you try this patch, keep in mind that CFLAGS need to enable SSSE3 (-mssse3, or something like -march=native).

--- ../lbzip2/src/decode.c	2022-07-06 00:57:22.365988349 +0300
+++ src/decode.c	2022-07-06 09:54:45.420303970 +0300
@@ -452,6 +452,34 @@ mtf_one(uint8_t **imtf_row, uint8_t *imt
        version is given in #else clause.
      */
 #if ROW_WIDTH == 16
+#if defined(__SSSE3__)
+    {
+      typedef char c8v16 __attribute__((vector_size(16)));
+      static const c8v16 tab[] = {
+        { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 },
+        { 1, 0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 },
+        { 2, 0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 },
+        { 3, 0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 },
+        { 4, 0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 },
+        { 5, 0, 1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 },
+        { 6, 0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 12, 13, 14, 15 },
+        { 7, 0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15 },
+        { 8, 0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15 },
+        { 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15 },
+        { 10, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15 },
+        { 11, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15 },
+        { 12, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15 },
+        { 13, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 15 },
+        { 14, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 15 },
+        { 15, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 },
+      };
+      c8v16 t;
+      memcpy(&t, pp, sizeof t);
+      t = __builtin_ia32_pshufb128(t, tab[nn]);
+      memcpy(pp, &t, sizeof t);
+      return c;
+    }
+#endif
     switch (nn) {
     default:
       abort();

A maintained recipe in Homebrew would be great

Currently, the only parallel bzip2 implemention in Homebrew is pbzip2. Having one for lbzip2 would be great. I could add one, but the author may want first crack at it.

More info:

http://mxcl.github.com/homebrew/

https://github.com/mxcl/homebrew/wiki

Build directions are missing/invalid/incomplete

On the website, in DOWNLOAD section, the SOURCE CODE subsection says:

For instructions how to build lbzip2 from source code unpack sources and read the INSTALL file included there. lbzip2 uses GNU build system, so compilation and installation usually boils down to the following sequence:

The source code contains no such INSTALL file.
After "the following sequence:", no sequence is provided
In the source code tarball, the bootstrap file mentions how to boostrap the application, but it's not made clear what needs to be done once everything is boostrapped to finish building/installing the package.

As well, directly above that, the site links to the source code are missing.
"You can download source tarball of the latest released version here: lbzip2-2.5.tar.gz, or in bz2 format: lbzip2-2.5.tar.bz2. Downloadable tarballs of older releases are listed at the botton of this page."

Thanks for the hard work.

make_code_lengths: Assertion `i == n' failed.

CentOs 6.4 lbzip2 2.3

Getting crash:
make_code_lengths: Assertion `i == n' failed.

Hard to reproduce, we are using it as:
pv uncomressed | lbzip2 > compressed.
And after 1-2 TB - it crashing.

minbzcat fails to decompress some files with trailing garbage

minbzcat fails to decompress some files with trailing garbage, in particular f82c7be77eec69287f7924618924b4c8.bz2 from PROTOS Genome.

$ ./minbzcat <bz2/f82c7be77eec69287f7924618924b4c8.bz2
****
minbzcat: data error in bz2 file
$ echo $?
1
$ bzcat <bz2/f82c7be77eec69287f7924618924b4c8.bz2
****

bzcat: (stdin): trailing garbage after EOF ignored
$ echo $?
0

release tarball 2.2 is not present

Could you please add the release tarball?

-n flag not working in combination with tar

First, I don't know if this is a lbzip2 problem, or a problem with tar. I posted this question on stack exchange (here). Here is the short description: tar allows you to specify the compression algorithm and pass flag/options to that algorithm. So, this should tar a directory and pass that on to lbzip2 for compression using two processors in verbose mode:

tar cf dst.tar.bz2 src --use-compress-program=lbzip2 -v -n 2

The verbose flag works, but lbzip always uses the max number of processors.

RHEL 7.6
lbzip2 2.5

lbzip2-2.2: Assertion failure while compressing

I am just copying over from downstream report:

https://bugs.gentoo.org/show_bug.cgi?id=436382

Something in a certain QEMU raw image seems to upset lbzip2-2.2. Other images are okay but I could not test another NTFS file system image. The error happens with a small part of the image so I will attach it.

Earlier versions of lbzip2 seem to be unaffected.

Reproducible: Always

Steps to Reproduce:

lbzip2 -c hpwin.img >/dev/null
Actual Results:
lbzip2: /var/tmp/portage/app-arch/lbzip2-2.2/work/lbzip2-2.2/src/encode.c:794: generate_initial_trees: Assertion `a < b' failed.

fails to check stream CRC

lbzip2 fails to check stream CRC when decompressing bzip2 some files with blocks larger than 1 MiB (highly compressible blocks). As the result these files aren't decompressed correctly. Note than lbzip2 itself never creates such files, so this bug doesn't affect decompressing files created by any version of lbzip2.

Ways to reproduce it:

$ echo QlpoOTFBWSZTWTcImb8AAAAAAIAAoACAJpQKpCETF3JFOFCQNwiZvw== | base64 -d | lbzcat
lbzcat: stdin: stream CRC mismatch

Justin Lecher reported the following case to also trigger the bug:

$ lbzip2 -d patchutils-0.3.2.tar.bz2
lbzip2: "patchutils-0.3.2.tar.bz2": stream CRC mismatch

Work to fix this problem is in progress. Estimated time of arrival: January 2012. For impatient there is a workaround available:

diff --git a/src/lbunzip2.c b/src/lbunzip2.c
index 113be47..67579ea 100644
--- a/src/lbunzip2.c
+++ b/src/lbunzip2.c
@@ -1453,7 +1453,7 @@ mux(struct w2m_q *w2m_q, struct m2s_q *m2s_q, struct files
         if (w2m_blk->bs100k) {
           bs100k = w2m_blk->bs100k;
           any |= (9u >= bs100k);
-          if (crc != w2m_blk->crc)
+          if (0 && crc != w2m_blk->crc)
             log_fatal("%s: %s%s%s: stream CRC mismatch\n", pname, ispec->sep,
                 ispec->fmt, ispec->sep);
           crc = 0u;

uses excessive amounts of memory

Bug reported under Debian BTS.
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=645999

Don't exceed 18002 selectors

It would be better not to abuse bzip2 bug that makes it accept more than 18002 selectors. Extra padding bits (max 7) can be added eg. with suboptimal tree descriptors (even number of bits) and up to one dummy selector.