gregwar / fatcat Goto Github PK

Build Info

Platform: Windows 10 / Cygwin
Compiler(s): Clang

Issue

When trying to get the info of a fat12 image, it threw a heap corruption error

So I went to debug it ...aaannnnd

(There's nothing wrong with the image file itself, even the root directory is correctly recognised by the program)

Quite a lot of fixes since last release in 2017, are you planning a new tag/release soon or should I package a snapshot?

Thanks

When extracting, the program crashes when the target directory specified does not exist

fatcat /tmp/sdh.img -O 1048576 -x /tmp/NoDir
Extracting /LOST.DIR/132 to NoDir/LOST.DIR/132
.        �dH�yNyNH�yN�..       �dH�yNyNH�yN��Segmentation fault (core dumped)

In normal cases the target directory exists and the recovery process starts working.
However, we may run out of disk space, in which case the program silently creates the output files with zero length.

it would be useful to have an option to specify offset for fat32 bootable images that could be inspected right a way without a need to create an intermediate file using dd like

dd if=boot.img of=boot_shifted.img skip=2048

exfat is now used for large removable media. vfat is also widely in use for removable media.

please support these FAT extensions.

I'm trying to use fatcat on my macos laptop to forensics a set of old floppies that I found, but am failing at the first hurdle: No matter what arguments I pass the fatcat binary, all it does is show the usage screen.

I've repro'd this in the docs/images directory:

$ pwd
/Users/asf/Hacks/fatcat/docs/images
$ gunzip empty.img.gz
$ fatcat empty.img -l /
fatcat v1.0.6, Gregwar <[email protected]>

Usage: fatcat disk.img [options]
  -i: display information about disk
  -O [offset]: global offset (may be partition place)

Browsing & extracting:
  -l [dir]: list files and directories in the given path
  -L [cluster]: list files and directories in the given cluster
  -r [path]: reads the file given by the path
  -R [cluster]: reads the data from given cluster
  -s [size]: specify the size of data to read from the cluster
  -d: enable listing of deleted files
  -x [directory]: extract all files to a directory, deleted files included if -d
                  will start with rootDirectory, unless -c is provided
* -S: write scamble data in unallocated sectors
* -z: write scamble data in unallocated sectors

FAT Hacking
  -@ [cluster]: Get the cluster address and information
  -2: analysis & compare the 2 FATs
  -b [file]: backup the FATs (see -t)
* -p [file]: restore (patch) the FATs (see -t)
* -w [cluster] -v [value]: write next cluster (see -t)
  -t [table]: specify which table to write (0:both, 1:first, 2:second)
* -m: merge the FATs
  -o: search for orphan files and directories
* -f: try to fix reachable directories

Entries hacking
  -e [path]: sets the entry to hack, combined with:
* -c [cluster]: sets the entry cluster
* -s [size]: sets the entry size
* -a [attributes]: sets the entry attributes
  -k [cluster]: try to find an entry that point to that cluster

*: These flags writes on the disk, and may damage it, be careful
$ fatcat empty.img -i
fatcat v1.0.6, Gregwar <[email protected]>

Usage: fatcat disk.img [options]
  -i: display information about disk
  -O [offset]: global offset (may be partition place)

Browsing & extracting:
  -l [dir]: list files and directories in the given path
  -L [cluster]: list files and directories in the given cluster
  -r [path]: reads the file given by the path
  -R [cluster]: reads the data from given cluster
  -s [size]: specify the size of data to read from the cluster
  -d: enable listing of deleted files
  -x [directory]: extract all files to a directory, deleted files included if -d
                  will start with rootDirectory, unless -c is provided
* -S: write scamble data in unallocated sectors
* -z: write scamble data in unallocated sectors

FAT Hacking
  -@ [cluster]: Get the cluster address and information
  -2: analysis & compare the 2 FATs
  -b [file]: backup the FATs (see -t)
* -p [file]: restore (patch) the FATs (see -t)
* -w [cluster] -v [value]: write next cluster (see -t)
  -t [table]: specify which table to write (0:both, 1:first, 2:second)
* -m: merge the FATs
  -o: search for orphan files and directories
* -f: try to fix reachable directories

Entries hacking
  -e [path]: sets the entry to hack, combined with:
* -c [cluster]: sets the entry cluster
* -s [size]: sets the entry size
* -a [attributes]: sets the entry attributes
  -k [cluster]: try to find an entry that point to that cluster

*: These flags writes on the disk, and may damage it, be careful

I built the binary using the exact procedure given in the README file, from current git master, 839090b

uname -a says Darwin courage.local 18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64 i386 MacBookPro13,2 Darwin, I'm running macos 10.14.5 (18F132)

# fdisk -l /dev/sde
Festplatte /dev/sde: 1,36 TiB, 1500301910016 Bytes, 2930277168 Sektoren
Festplattenmodell:                 
Einheiten: Sektoren von 1 * 512 = 512 Bytes
Sektorgröße (logisch/physikalisch): 512 Bytes / 4096 Bytes
E/A-Größe (minimal/optimal): 4096 Bytes / 4096 Bytes
Festplattenbezeichnungstyp: dos
Festplattenbezeichner: 0xeb8e465e

Gerät      Boot Anfang       Ende   Sektoren Größe Kn Typ
/dev/sde1        16065 2930272064 2930256000  1,4T  f W95 Erw. (LBA)
/dev/sde5        16128 2930272064 2930255937  1,4T  6 FAT16

Partition 1 beginnt nicht an einer physikalischen Sektorgrenze.


[root@eve ~]# fatcat -i /dev/sde5
WARNING: Bytes per sector is not 512 (38637)
WARNING: Sectors per cluster high (198)
! Failed to init the FAT filesystem

no matter what, if used with different calculated offsets via -O in any combination of the devices /dev/sde* , it was the same error message.

When using fatcat on a image I have that is dumped from a micro sd card I get a warning about missing fat table, WARNING: Fats number different of 2 (1). How this happened I don't know but there have been issues with that specific card.

The card is mountable in both Windows and Linux. I've tried to dump with dd and ddrescue but same issue occurs. The card is partitioned starting at sector 63 so I need an offset of 32256.

When commenting out strange++ in FatSystem.cpp for the fats check I can backup both tables and do other stuff.
When using -t 1 I get a file that is essentially empty but -t 2 looks like a valid one.
As a test I patched the first table with data from the second but it still complains about missing fat table.

Info shows (after commenting out the check)

WARNING: Fats number different of 2 (1)
FAT Filesystem information

Filesystem type: FAT
OEM name:
Total sectors: 3868609
Total data clusters: 106496
Data size: 3489660928 (3.25G)
Disk size: 1980727808 (1.8447G)
Bytes per sector: 512
Sectors per cluster: 64
Bytes per cluster: 32768
Reserved sectors: 1
Root entries: 512
Root clusters: 1
Sectors per FAT: 416
Fat size: 212992 (208K)
FAT1 start address: 0000000000000200
FAT2 start address: 0000000000034200
Data start address: 0000000000034200
Root directory cluster: 0
Disk label: NO NAME

Free clusters: 36219/106496 (34.0097%)
Free space: 1186824192 (1.10532G)
Used space: 2302836736 (2.14468G)

Is it fixable?
How to fix it?
How dangerous is it to do stuff without fixing it?

fatcat's -z mode is great for backups of FAT filesystems, in order to make them more compressible.

Would there be any interest in adding fallocate support for making holes in the file for similar space-saving purposes? I've hacked around a proof of concept which seems to work.

This would be functionally similar to defragging / compacting then shrinking the filesystem, except when done right, no undeleted data is touched.

(Why do this? I have backups of filesystems with low %age use, but prefer to keep them in the exact backup file which dd copies. It would save gigs of space to be able to make the files sparse.)

partitions.csv

# Notes: the offset of the partition table itself is set in
# $IDF_PATH/components/partition_table/Kconfig.projbuild.
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x1F0000,
vfs,      data, fat,     0x200000, 0x200000,

read_flash to bin
esptool --chip esp32c3 -p COM24 -b 921600 --before=default_reset --after=hard_reset read_flash 0x200000 0x200000 C:/test/ESP32/esp32c3/data_2M.bin

qgb@ubuntu:~/github/micropython/ports/esp32$ fatcat /mnt/test/esp32/esp32c3/data_2M.bin -l /
WARNING: Bytes per sector is not 512 (27764)
WARNING: Fats number different of 2 (47)
! Failed to init the FAT filesystem

Besides -x mode (-x <directory>: extract all files to a directory) I suggest implementing a very similar scan functionality (the switch might be -X or something).
In this mode the program would walk through the entries similarly, but instead of extracting all the files, it would

• print only the pure directory tree (names, cluster numbers)
• create them as well, if the optional targetdir argument would exist (-X [directory]).

I think recovering the full directory-tree can be very useful in cases when only some of the directories contain important files. Also, with my damaged image, the extracting crashed in some folders, which prevented me to get the whole directory-tree in one stage.

After doing some fuzzing, I found a file that produces a floating point exception in fatcat. It seems to be a divide by zero in FatSystem::parseHeader() on line 160.

The image file and a README detailing the crash can be found Here

but should say this instead
-S: write scramble data in unallocated sectors
-z: write zero data in unallocated sectors

This is probably going to sound dumb in my part, but it seems like there's no makefile.
The CMake Code? Yup, it exists
The makefile used to make the build? Nope, not even a trace of it

platform: Windows (cygwin)
How to replicate:

1. Build the files as shown from readme.md
2. That's it really, make will complain about no makefile

I want to create a file system with a file at a specific point, so I was using fcat -@ 0 to find the base address of the clusters then some simple maths (data_addr - base_addr)/cluster_size to find the cluster that points to the data.

I noticed that it worked in 1.05 and not in v1.1.0 it looks like change 530b663 to support larger disks broke it.

1.0.5:

Cluster 0 address:
47104 (000000000000b800)
Next cluster:
FAT1: 4294967295 (ffffffff)
FAT2: 4294967295 (ffffffff)
Chain size: 1 (2048 / 2K)
Chain is contiguous

1.1.0:

Cluster 0 address:
8796093069312 (000008000000b800)
Next cluster:
FAT1: 4294967295 (ffffffff)
FAT2: 4294967295 (ffffffff)
Chain size: 1 (2048 / 2K)
Chain is contiguous

Per the Microsoft Extensible Firmware Initiative FAT32 File System Specification (https://msdn.microsoft.com/en-us/windows/hardware/gg463080.aspx), BPB_RootClus, which stores the cluster number of the first cluster of the root directory, is usually 2, but not required to be 2. This can be set to values others than 2 if cluster 2 is marked bad.

The check for this occurs at https://github.com/Gregwar/fatcat/blob/master/src/core/FatSystem.cpp#L190.

We found an instance where a superfloppy volume formatted on Windows 10 placed the root directory at cluster 4. fatcat was unable to load the filesystem, yielding the following error message:

sudo fatcat /dev/loop0 -i -l /
WARNING: Root directory is not 2 (4)
! Failed to init the FAT filesystem

I have a disk where FAT1 and FAT2 are missing. It's a very old drive, so heads 1 and 2 are no longer over the tracks for the first several tracks (I'm using an advanced imaging tool that tells me things like this about the drive). So, the drive cannot read those areas and so my image has 0's in them. It would be nice to be able to start somewhere and initialize a FAT whose chains are all 1 long, based on different parameters I give on the command line. This may also mean the first N sectors of the root directory are zero'd too, I don't know yet. The first entry is weird, so I strongly suspect that's the case:

00002400  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00006c00  57 53 4d 53 47 53 20 20  4f 56 52 20 00 00 00 00  |WSMSGS  OVR ....|
00006c10  00 00 00 00 00 00 9a 06  21 09 21 00 80 71 00 00  |........!.!..q..|
00006c20  4c 4f 54 55 53 32 20 20  20 20 20 10 00 00 00 00  |LOTUS2     .....|
00006c30  00 00 00 00 00 00 34 6e  07 11 dd 04 00 00 00 00  |......4n........|

It's a 10MB file from a non PC-DOS compatible computer (it ran MS-DOS 2.11), so there's also some weird stuff at front for the non-standard partitioning scheme that it uses. It's clearly more damaged than you're used to dealing with in your fatcat program...

Hi there,

Thank you for this library! We are using it in our project to repair errors on the file system.

I was wondering, is there a way to repair invalid directory entries? For example, if the same filename appears multiple times in a directory's entries or if the filename is garbage (i.e. non-printable) then some sort of corruption has occurred.

Is it possible to enhance the library to support this?

Thanks.

A -v switch for -z or -S would be a nice addition to see whats goind on while scrubbing the fs.
I tried -z switch on two 16GB SD-Cards with FAT32 and both were after this unreadable.

A fsck after this says: Logical sector size is zero
and stops.