gozora / ebiso Goto Github PK

Hello @gozora

I was recently informed via a SUSE customer issue report
that ebiso silently corrupts files greater or equal 2GiB.

As far as I understand it the "silently" is the actual issue in ebiso because
the actual root cause is a limit of the ISO 9660 file system according to
https://en.wikipedia.org/wiki/ISO_9660#The_2/4_GiB_file_size_limit

How it can happen that ebiso has to deal with files greater or equal 2GiB:

In ReaR it is possible to include the backup in the ISO
with OUTPUT=ISO and BACKUP_URL=iso://, cf.
https://github.com/rear/rear/blob/master/doc/rear.8.adoc

When the backup archive file (e.g. backup.tar.gz) is greater or equal 2GiB
ebiso has to deal with files greater or equal 2GiB.

Because of the 2GiB file size limit in the ISO 9660 file system
the backup archive file gets corrupted in the ISO.

The issue is that ebiso does not detect that problem
so for the user it appears af if everything went OK during "rear mkbackup"
until he finds out later during "rear recover" that his backup got corrupted.

So it would be good if ebiso could detect files greater or equal 2GiB
and show an error message in such cases and finish with non-zero exit code.

If ebiso could not detect (with reasonable effort) files greater or equal 2GiB
we could perhaps do something in ReaR to detect such a problem?

Perhaps in general it would be best to have something in ReaR
when making the ISO (regardless what ISO-making tool is used)
that verifies there is no file greater or equal 2GiB?

Also the ReaR recovery system initrd could become greater or equal 2GiB
(e.g. because of accidentally too much in COPY_AS_IS).

Reproducer script ebiso-test.sh:

#!/bin/bash

#set -x

mkdir -p ebiso-test
cd ebiso-test

echo "2 GiB =  2 * 1024 * 1024 * 1024 = 2147483648"
mkdir -p below2g exact2g above2g
test -s below2g/testfile || dd if=/dev/urandom of=below2g/testfile bs=1M count=2147483647 iflag=count_bytes status=progress
test -s exact2g/testfile || { cp below2g/testfile exact2g/testfile ; echo -n 'x' >>exact2g/testfile ; }
test -s above2g/testfile || { cp below2g/testfile above2g/testfile ; echo -n 'yy' >>above2g/testfile ; }

for subdir in below2g exact2g above2g ; do
    pushd $subdir
    echo "in $subdir"
    ls -l testfile
    md5sum testfile
    echo 'dummy' >dummy.efi
    ebiso -e dummy.efi -o test.iso .
    mkdir -p /tmp/ebiso-test
    mount test.iso /tmp/ebiso-test
    ls -l /tmp/ebiso-test/testfile
    md5sum /tmp/ebiso-test/testfile
    umount /tmp/ebiso-test
    popd
done

Result of the reproducer script:

# ./ebiso-test.sh
2 GiB =  2 * 1024 * 1024 * 1024 = 2147483648
2096103424 bytes (2.1 GB, 2.0 GiB) copied, 37 s, 56.6 MB/s 
2047+1 records in
2047+1 records out
2147483647 bytes (2.1 GB, 2.0 GiB) copied, 37.8585 s, 56.7 MB/s
~/ebiso-test/below2g ~/ebiso-test
in below2g
-rw-r--r-- 1 root root 2147483647 Nov 20 09:53 testfile
e964b39bce5e7a3aafb2656b1e357f69  testfile
mount: /tmp/ebiso-test: WARNING: device write-protected, mounted read-only.
-r-xr-xr-x 1 root root 2147483647 Nov 20 09:53 /tmp/ebiso-test/testfile
e964b39bce5e7a3aafb2656b1e357f69  /tmp/ebiso-test/testfile
~/ebiso-test
~/ebiso-test/exact2g ~/ebiso-test
in exact2g
-rw-r--r-- 1 root root 2147483648 Nov 20 09:54 testfile
9ae5c212a37bd45cf5a6b0a8b0c0d246  testfile
mount: /tmp/ebiso-test: WARNING: device write-protected, mounted read-only.
-r-xr-xr-x 1 root root 2147483647 Nov 20 09:54 /tmp/ebiso-test/testfile
e964b39bce5e7a3aafb2656b1e357f69  /tmp/ebiso-test/testfile
~/ebiso-test
~/ebiso-test/above2g ~/ebiso-test
in above2g
-rw-r--r-- 1 root root 2147483649 Nov 20 09:54 testfile
fde433ea7108fbe31fe68fb39bba13f1  testfile
mount: /tmp/ebiso-test: WARNING: device write-protected, mounted read-only.
-r-xr-xr-x 1 root root 2147483648 Nov 20 09:54 /tmp/ebiso-test/testfile
789f0fb931c30af13be3f68bedf0ebba  /tmp/ebiso-test/testfile
~/ebiso-test

So in my case it seems to work up to a file size of 2 GiB - 1 (2147483647 bytes).
But problems start with a file size of 2 GiB (2147483648 bytes) because
then the file in the ISO is listed with only 2147483647 bytes and different md5sum
and same for a a file size of 2 GiB + 1 (2147483649 bytes) where
the file in the ISO is listed with only 2147483648 bytes and different md5sum.

This issue was created for reference to ReaR issue rear/rear#1208.

When building ebiso for openSUSE, following issues are observed by automatic checks of build log:
I: Statement is overflowing a buffer
E: ebiso bufferoverflow lib/filename.c:44:13
E: ebiso bufferoverflow lib/iso9660.c:594:16

See also openSUSE bug 974360

ReaR/GRUB had trouble to automatically boot with 8.3 file format, so it was disabled until RRIP will be implemented (v0.1.1).
ISO is now not fully ISO9660 compatible, but shout work anyhow...

please try to create rpm builds (you can use OpenSuse Build Service for this). Would be easier to download binary rpms to use immediately on SLES architectures.
Thanks for EFI Boot ISO maker.... glad we have something working now on SLES/UEFI systems.

Is there any chance your could create a package for CentOS 7?