nlitsme / extfstools

Tools for extracting files from ext2,3,4 filesystem images

This my my profile README.

A short overview of all my repositores:

Risc-V / Titan-M2

repo	desc
riscv-isa-sim-titanM2	Patches such that the SPIKE Risc-V simulator can run the Titan-M2 firmware.
python-riscv-sim	A pure python RISC-V simulator

whatsapp

repo	desc
whatsapp-apk-proto	Changes in the whatsapp protocol as extracted from apk files
whatsapptools	A collection of python scripts i use for managing whatsapp chats from the commandline.
whatsmeow	My changes to the whatsmeow project
codehistory	Archive of all versions of web.whatsapp.com

bitcoin

repo	desc
bitcoinexplainer	Interactive examples explaining the details of how bitcoin calculations work.
python-bcutils	bitcoin utils

firmware and filesystem unpackers

repo	desc
CelbEprDecode	Decode Cellebrite bootloaders from ufedsamsungpack_v21.epr
CompressUtils	(old) perl extension for decompressing WinCE roms.
dllloader	(old) library for loading functions from win32 binaries on macos or linux.
eimgfs	Tool for editting Windows CE/Mobile firmware images.
extfstools	Tools for extracting files from ext2,3,4 filesystem images
fatutils	tools for reading from FAT filesytem images
hfstools	Tools for reading or recovering files from an apple HFS+ filesystem
hvtool	Create or View Windows CE registry .hv hive files.
iphonetools	Tools for inspecting iOS firmware images
ubidump	Tool for viewing and extracting files from an UBIFS image
SquashFSDumper	tool for listing and extracting files from SQUASHFS images
encrypteddmg	Tool for analyzing and decrypting apple encrypted disk images
cronodump	A Cronos database converter
ntfs_research_tool	Tool for investigating broken ntfs partitions.
qualcomm-q6zip	decompressor for qualcomm q6zip and delta compressed firmware sections

c++

repo	desc
cpputils	various c++ utility classes
hexdumper	hexdumper tool i use for just about anything.
legacy-itsutils-library	Part of the old itsutils library

commandline toolbox

repo	desc
findstr	A tool for searching text or byte patterns in binary files.
zipdump	Analyze zipfile, either local, or from url
transpose	Command line tool for transforming matrices of textual data

perl

repo	desc
Crypt-CCM	Release history of Crypt-CCM

math

repo	desc
expressionfinder	Tool for solving math problems involving finding an expression for a given value. (like numberphile's 10958 problem)
GeometricShapes	Python module for generating the coordinates of the platonic solids in n dimensions
mpmp19_solver	None
mpmp7_solver	Solving Matt Parker's Unique Distancing Puzzle
pointenumerator	Several algorithms for enumerating points in an unbounded plane

arduino / avr

repo	desc
arduino-simulator	Minimal Arduino simulator, debug your project on your laptop.
AVRInstructionSet	Investigating the AVR / Arduino instruction set

site api scraping tools

repo	desc
githubtool	Commandline tool for searching github
youtube_tool	Tool for extracting comments or subtitles from youtube video's
wikiexport	Tool for downloading the contents of a mediawiki site
pleskfilemanager	python script for managing files on a plesk server from the commandline
magister-tool	Tool for getting information from the magister school server
4chan-tool	Tool for scraping / searching 4chan forums
twitter-tool	Tool for scraping tweets

ida

repo	desc
idaperl	perl scripting support for IDApro
idascripts	IDApro idc and idapython script collection
idbutil	Library and tool for reading IDApro databases.
pyidbutil	A python library for reading IDA pro databases.
idcinternals	IDA plugin investigating the internal representation of IDC scripts

ancient

repo	desc
HACKTIC_demon_dialer	Original sourcecode for the HACKTIC demon dialer
MSDOS_demon_dialer_tool	A MSDOS Demon Dialer phone scanning tool - for the museum
MSDOS_game_of_life	For the museum: a very old implementation of Conway's game of LIFE
MSDOS_interrupt_catcher	A very old msdos interrupt tracing tool - for the museum
ancient	various ancient documents

the rest

repo	desc
gnubc	Gnu bc calculator with some improvements
covidcalculations	Several covid calculations in javascript using live datasets.
pyCrcExperiment	Experiment, making the relation of CRC's and polynomials explicit
pyCryptoAdapter	Extend pyCrypto with your own ciphers.
pyCryptoBenchmarking	benchmarking the python pyCrypto and cryptography moduls
pyPdfCrack	Investigation in PDF encryption
PythonMonkey	Drop-in replacement for the android Jython monkeyrunner library
pytorify	module which makes sure all sockets use the TOR proxy
tsthtml2	None
uselection2020	None
vimdecrypt	Python tool for decrypting vim encrypted files.
xpcap	tool for analyzing packet capture dumps
xpcapperl	perl script for making the output of tcpdump -x more readable
nlitsme	this page
nlitsme.github.io	my github.io page
EnergieLeveranciers	Tools for getting data from some dutch energy providers
b3603	Updates for the b3603 firmware
bounchingdots	Experiment with random bouncing dots in c++ and python, using Qt
findlinks	tool for extracting links for a set of html files
oldmstools	win32 tools, originally from the itsutils package.
vandale2008reader	Decodes files from the 2008 edition of the vandale dictionary

extfstools's Issues

‘size’(‘empty’) is not a member of ‘std’

Hi, I cloned the repo and try to build the project in linux visual studio code 1.47.1

I use the Code Runner plugin the compile this project, but get errors ‘size’ is not a member of ‘std’ and ‘empty’ is not a member of ‘std’:

[Running] cd "work/roms/extfstools-origin/" && make ext2rd && ./ext2rd && make clean
g++ -g -Wall -c -O3 -I itslib -D_UNIX -D_NO_RAPI -I /usr/local/include -I . -std=c++17  -MD ext2rd.cpp -o ext2rd.o 
In file included from ext2rd.cpp:14:0:
itslib/util/rw/OffsetReader.h: In constructor ‘OffsetReader::OffsetReader(ReadWriter_ptr, uint64_t, uint64_t)’:
itslib/util/rw/OffsetReader.h:16:84: warning: format ‘%llx’ expects argument of type ‘long long unsigned int’, but argument 2 has type ‘uint64_t {aka long unsigned int}’ [-Wformat=]
             printf("off=%08llx, size=%08llx,  src: %08llx\n", off, size, _r->size());
                                                                                    ^
itslib/util/rw/OffsetReader.h:16:84: warning: format ‘%llx’ expects argument of type ‘long long unsigned int’, but argument 3 has type ‘uint64_t {aka long unsigned int}’ [-Wformat=]
itslib/util/rw/OffsetReader.h:16:84: warning: format ‘%llx’ expects argument of type ‘long long unsigned int’, but argument 4 has type ‘uint64_t {aka long unsigned int}’ [-Wformat=]
In file included from itslib/args.h:27:0,
                 from ext2rd.cpp:15:
itslib/stringutils.h: In function ‘std::__cxx11::string rhexdump(R)’:
itslib/stringutils.h:474:39: error: ‘size’ is not a member of ‘std’
     return hexdump(std::begin(range), std::size(range), sizeof(*std::begin(range)));
                                       ^
itslib/stringutils.h: In function ‘std::__cxx11::string base64_encode(const R&)’:
itslib/stringutils.h:554:9: error: ‘empty’ is not a member of ‘std’
     if (std::empty(data))
         ^

I have little knowledge about c++, could you give me some advice on how to fix the errors?

build faield

vcpkg install vcpkg install boost:x64-windows

"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Linux\GNU MCU Eclipse\Build Tools\2.9-20170629-1013\bin\make" vc
"C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/bin/cmake.exe" -G"Visual Studio 16 2019" -B build .
-- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19042.
-- The C compiler identification is MSVC 19.28.29336.0
-- The CXX compiler identification is MSVC 19.28.29336.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.28.29333/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message):
Could NOT find Boost (missing: Boost_INCLUDE_DIR date_time)
Call Stack (most recent call first):
C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:458 (_FPHSA_FAILURE_MESSAGE)
C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/Common7/IDE/CommonExtensions/Microsoft/CMake/CMake/share/cmake-3.18/Modules/FindBoost.cmake:2177 (find_package_handle_standard_args)
CMakeLists.txt:43 (find_package)

-- Configuring incomplete, errors occurred!
See also "D:/extfstools-master/build/CMakeFiles/CMakeOutput.log".
Makefile:7: recipe for target 'vc' failed
make: *** [vc] Error 1

ext2rd.cpp:5:10: fatal error: 'map' file not found

[root@lie-recovery extfstools]# make
clang++ -g -Wall -c -O3 -I itslib -D_UNIX -D_NO_RAPI -I /usr/local/include -I . -std=c++11 -stdlib=libc++ -MD ext2rd.cpp -o ext2rd.o
ext2rd.cpp:5:10: fatal error: 'map' file not found
#include
^
1 error generated.
make: *** [ext2rd.o] Error 1
[root@lie-recovery extfstools]#

Where is that from? A required package?

ext2rd.cpp:5:10: fatal error: 'map' file not found

clang++ -g -Wall -c -O3 -I itslib -D_UNIX -D_NO_RAPI -I /usr/local/include -I . -std=c++11 -stdlib=libc++ -MD ext2rd.cpp -o ext2rd.o
ext2rd.cpp:5:10: fatal error: 'map' file not found
#include
^
1 error generated.
make: *** [ext2rd.o] Error 1

gcc version compatibility

Dear author,the current code needs C++17,which means at least gcc(7.1+)/ubuntu 18.04+,the gcc upgrade is complex in lower version systems

itslib build failure

At the moment the src fails to build despite CMake being able to locate and add the itslib submodule:

[ 77%] Completed 'itslib-populate'
[100%] Built target itslib-populate
Found boost at /usr/include
-- Configuring done
CMake Error at cmake_find/Finditslib.cmake:14 (add_library):
  Cannot find source file:

   .../extfstools/build/itslib-src/src/FileFunctions.cpp

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h
  .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc
Call Stack (most recent call first):
  CMakeLists.txt:38 (find_package)


CMake Error at cmake_find/Finditslib.cmake:14 (add_library):
  No SOURCES given to target: itslib
Call Stack (most recent call first):
  CMakeLists.txt:38 (find_package)

Add support for wildcards

I would like to be able to extract only directories inside a specific directory

e.g. directories

modem-config/S1234.1
modem-config/S5678.1

but exclude file
modem-config/modem.conf

ext2dump.exe failed to extract system.img.ext4

https://samfrew.com/download/GALAXY__Note3__/nO5r/TPL/N9005XXSGBRI2/N9005TPLGBRJ1/

cat it compiled by cygwin?

cat it compiled by cygwin?

Symlinks are not exported

Thanks for the great tool.

E.g. if I run:
mkdir outdir
ext2rd <my_ext4.img> ./:outdir

Symlink (valid or broken) are not exported. I would expect symlinks (whether valid or broken) to be exported.

Seems like export ignores the symlinks. Only directories and regular files are processed.
In ext2rd.cpp, near line 1063:

    recursedirs(fs, ino, ".", [&](const DirectoryEntry& e, const std::string& path) {
        if (e.filetype==EXT4_FT_DIR) {
            if (-1==mkdir((savepath+"/"+path+"/"+e.name).c_str(), 0777)) {
                perror("mkdir(savepath)");
            }
        }
        else if (e.filetype==EXT4_FT_REG_FILE) {
            exportinode byino(e.inode, savepath+"/"+path+"/"+e.name);
            byino.perform(fs);
        }
    });

ext2rd seems make apex in system.img fail to unzip

Hi,
Thanks for sharing this great tool for rom analysing !
I meet one thing that, after I run the ext2rd system_ext.img ./:savedir there is no error show and the jar file in system image is good to unzip.

the system.img in the test below is from https://dl.google.com/developers/android/sc/images/factory/coral-spb3.210618.016-factory-b1598d08.zip

file system.img
system.img: Android sparse image, version: 1.0, Total of 204832 4096-byte output blocks in 21 input chunks.

file system_ext.img
system_ext.img: Linux rev 1.0 ext2 filesystem data, UUID=ac16ea36-9e78-5adc-92ff-3e6fd0ccedcc (extents) (large files) (huge files)

When it comes to file under system/apex/ , the unzip is not success，seems all the apex file are broken

savedir/system/apex$ unzip -l com.android.appsearch.apex
Archive:  com.android.appsearch.apex
error [com.android.appsearch.apex]:  missing 32768 bytes in zipfile
  (attempting to process anyway)
  Length      Date    Time    Name
---------  ---------- -----   ----
      761  2009-01-01 00:00   apex_build_info.pb
       29  2009-01-01 00:00   apex_manifest.pb
  1355776  2009-01-01 00:00   apex_payload.img
     1032  2009-01-01 00:00   apex_pubkey
     1359  2009-01-01 00:00   assets/NOTICE.html.gz
       40  2009-01-01 00:00   resources.arsc
     1340  2009-01-01 00:00   AndroidManifest.xml
       32  2009-01-01 00:00   stamp-cert-sha256
      868  2009-01-01 00:00   META-INF/CERT.SF
     2244  2009-01-01 00:00   META-INF/CERT.RSA
      733  2009-01-01 00:00   META-INF/MANIFEST.MF
---------                     -------
  1364214                     11 files

while the jar xvf com.android.appsearch.apex returns

savedir/system/apex$ jar xvf com.android.appsearch.apex
extracted: apex_build_info.pb
extracted: apex_manifest.pb
java.util.zip.ZipException: invalid entry CRC (expected 0xdb6c7410 but got 0x2c51a2fa)
        at java.base/java.util.zip.ZipInputStream.read(ZipInputStream.java:222)
        at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
        at jdk.jartool/sun.tools.jar.Main.copy(Main.java:1249)
        at jdk.jartool/sun.tools.jar.Main.copy(Main.java:1277)
        at jdk.jartool/sun.tools.jar.Main.extractFile(Main.java:1454)
        at jdk.jartool/sun.tools.jar.Main.extract(Main.java:1364)
        at jdk.jartool/sun.tools.jar.Main.run(Main.java:409)
        at jdk.jartool/sun.tools.jar.Main.main(Main.java:1681)

I tried several system image including the newest android-s-beta 3.1 ， the result seems same， all apex file is croupted while the other jar file is good to unzip.

Any idea if some log I could help to catch to debug this ? the tool build is using the HEAD version.

segfault on Pixel 3a Android 9 image

Firmware: https://dl.google.com/dl/android/aosp/sargo-pq3b.190801.002-factory-56817dc2.zip

Running ext2rd system.img system/build.prop:build.prop causes a segfault:

➜  sargo-pq3b.190801.002  ext2rd system.img system/build.prop:build.prop
sparse: v00000001 h:001c c:000c b:00001000 nblk:000c2c00 ncnk:00000017,  cksum:00000000
end of sparse: c2c00000
 23 entries in sparse map
first: 0-8002000: copy from 28
last:  c2bff000-c2c00000: copy from 8e78c130
ehmagic=6818 -  18 68 0c 38 f7 f2 a8 e0 20 46 ea f2
[1]    145861 segmentation fault (core dumped)  ext2rd system.img system/build.prop:build.prop

When using ext2rd on a not yet existing output directory it fails with a "No such file or directory" error

The error message it quite fuzzy:

mkdir(savepath): No such file or directory
mkdir(savepath): No such file or directory
mkdir(savepath): No such file or directory
mkdir(savepath): No such file or directory
mkdir(savepath): No such file or directory
mkdir(savepath): No such file or directory
EXCEPTION - did you specify -B for a block device?

If you create the directory before running ext2rd it works okay.

Expected behaviour: ext2rd should create the directory if it does not exist or give a clear message that the output directory should be created before using the tool.

Incorrect offset for block group descriptors when the block size is 1k

On parsing one ext2 dump with a block size of 1024, this tool outputs "#2 is not a dir" and exits without generating any results. The offset for the block group descriptors is actually 0x800 but this tool thinks it's on 0x400. I found this comment on the source code.

extfstools/ext2rd.cpp

Lines 809 to 811 in 2a7e65b

	// superblock is always at fixed position 0x400
	// ... does that indicate a min blocksize of 0x800 ?
	// with smaller blocksize, the groupdescs would overlap with the first superblock

So, I did some searching and found this.

https://www.kernel.org/doc/html/latest/filesystems/ext4/overview.html#layout

For the special case of block group 0, the first 1024 bytes are unused, to allow for the installation of x86 boot sectors and other oddities. The superblock will start at offset 1024 bytes, whichever block that happens to be (usually 0). However, if for some reason the block size = 1024, then block 0 is marked in use and the superblock goes in block 1.

My understanding is that, if the block size is equal or greater than 2k, this tool should work fine as the first block is large enough to hold the first 2k data, and the second block can be used to hold block group descriptors. When the block size is 1k, it needs two blocks; the first to hold the first 1k data, and the second to hold the super block. As the block group descriptors follow the super block, in this case the third block is used to store block group descriptors.

If my understanding is correct, the offset for the block group descriptors should be determined by where the super block locates; the block for block group descriptors follows the super block and may not be the second block.

I'm not really familiar with ext. Don't be hesitate to tell me if there's any mistake :)

exportdir: path not found

Hi,
would it be possible to improve the error message so taht the path is shown? I used the a838c9b commit version.

It feels that some_dir/:'pwd' is wrong and some_dir/:. too.

# ext2rd raw_ext4_image.bin some_dir/:'pwd' 
ehmagic=080e -  0e 08 00 00 00 00 00 00 1c 00 00 00
exportdir: path not found
#

Are you aware of https://sourceforge.net/p/ext4magic/tickets/ being unmaintained and that its re-write ext4magic also has issues? I think helping these could be more helpful to the people. :-)