Unable to do anything with image; truncated section header about dwarfs

[mhx]

Definitely not a known issue. I'll get back to you later when I have access to my laptop.

[mibmo]

That's alright, I'm just happy to have an answer to what happened. Thanks for the help :)

I think it's just been one of those days - today two drives died (why is why I did the backup; to rebuild the array), somehow now the server wont start at all (no matter the image I flash on any usb, it has a unique error each time - even after clearing bios & taking all drives out), seems that two USBs are dead (one of them being the one I had the backup on), and somehow my headphones died too (thankfully just the cable though). ^{(apologies for the rant; it's been a long day.)}

On the positive side though, not a dwarfs issue! :D

I wonder if this could be due to removing the USB disk before all data has been written? I usually verify the checksum of important files copied to USB media after a full umount / sync / remove / insert / mount cycle.

I did wait for the umount to finish, but I think the hardware is just faulty. Wiggling the USB even the tiniest bit results in a disconnect and I might've also been a bit hasty in trying to minimize downtime.

Thanks again for the help :)

[mhx]

So a DwarFS image contains a variable number of sections, each of which is prefixed with a 64-byte section header. If a section has been fully read and there is further data after the section, it is assumed that there is another section header. If the amount of data is less than 64 bytes, though, this will trigger the error you're seeing.

So there's a few possibilities:

1. The DwarFS image file was truncated.
2. Some data was appended to the DwarFS image.
3. Some corruption happened and the section parser got out of sync.

Right now, there's no automatic way to recover from any kind of corruption to a DwarFS image.

As a first step, could you please run:

$ dwarfsck --log-level=trace name-of-your-image.dwarfs

This should at least show you a list of all sections than can be read. Also, could you run:

$ grep --byte-offset --only-matching --text DWARFS name-of-your-image.dwarfs

That should hopefully give us an initial idea of the state of your DwarFS image.

[mibmo]

Sure, I'll give it a go. :) Thanks for helping btw.

dwarfsck --log-level=trace name-of-your-image.dwarfs

Ah, yeah okay - looks like something went wrong after section 137? https://0x0.st/HViv.log

grep --byte-offset --only-matching --text DWARFS name-of-your-image.dwarfs

I don't know how this is supposed to look. https://0x0.st/HViw.log

Of the 3 options you mentioned, which do you think this is? looks to me like option 3...?

To be clear, this is essentially everything from my perspective.

mkdwarfs -i /persist -o persist.dwarfs -l9
mv persist.dwarfs /mnt # usb
umount /mnt

and then the file on the USB (5G on 16G stick) had this issue. Could it perhaps be faulty hardware too? The archive on the machine might be good, I didn't get to check it; I did the classic mistake of wiping before checking the backup, haha.

[mhx]

Looks like the file is all zeros after section 137. Which means the metadata section is missing. Which unfortunately also means there's zero chance of recovery.

I wonder if this could be due to removing the USB disk before all data has been written? I usually verify the checksum of important files copied to USB media after a full umount / sync / remove / insert / mount cycle.

So this is case 3 (corruption) and indeed the section parser ends up out of sync (that's definitely something that can be improved).

Sorry for the bad news, but it's impossible to recover the data due to the missing metadata section.