Investigate memory consumption when compressing large files about dwarfs

[Phantop]

@Phantop, just to make sure I understand this correctly: you're trying to build DwarFS image from a tree where single files have a size of 3.1GB; this single file by itself doesn't compress well, but it appears to be the root cause of mkdwarfs using an excessive amount of memory?

Yeah. The issue boils down to memory usage increasing to ludicrous levels with larger single file sizes, regardless of the size of the entire folder being compressed. I've had it happen in an 11GB folder with the largest file being ~650MB (which I managed to compress using lower memory and job options) and in a 7GB folder with a 1.1GB file (which required me to recompress). It's entirely due to single files.

This mostly makes sense. You can't change the block size (-S) during recompression, so that explains why it compresses much better with -S24. Nothing is stopping you from using -l0 -S24 initially if that helps. At -l0, the block size is 1MiB, just as with mksquashfs -b 1M, so again, no real surprise.

Yeah I'm aware of all this. just trying to make it clear that I have been able to compress images with these files, just after some workarounds. Really glad you're working on this!

[mhx]

@Phantop, just to make sure I understand this correctly: you're trying to build DwarFS image from a tree where single files have a size of 3.1GB; this single file by itself doesn't compress well, but it appears to be the root cause of mkdwarfs using an excessive amount of memory?

[mhx]

What I did find instead was that using the -l0 option and then recompressing the image works in these cases without issue. Creating the initial image with -S24 results in very well recompressed files in these instances, the 3.1 GB file compressing down to 2.3 GB whereas the default block size for -l0 resulted in a 2.6 GB file (which is approx what mksquashfs -comp zstd -b 1M -Xcompression-level 22 also gave me).

This mostly makes sense. You can't change the block size (-S) during recompression, so that explains why it compresses much better with -S24. Nothing is stopping you from using -l0 -S24 initially if that helps. At -l0, the block size is 1MiB, just as with mksquashfs -b 1M, so again, no real surprise.

[mhx]

@Phantop, I know exactly what the problem is. I just tried compressing my directory of virtualbox images and RSS memory consumption went through the roof (I stopped the process when RSS hit 50 GiB).

It's definitely fixable, but it's not a trivial fix, so it might take a bit of time. In the meantime, I've got a workaround: pass --blockhash-window-sizes=-. This may impact compression ratio a bit, but likely not as much as changing the block size. It should definitely fix your problem, though.

[mhx]

FYI, I've got a fix. (In fact, it's much more than a fix, it's a massive improvement that fixes this issue as a side-effect.)

[mhx]

Fixed in v0.4.0.