androiddumps / firmware_extractor

Requirements

• protobuf
• LZMA
• 7z
• lz4

Linux

apt install unace unrar zip unzip p7zip-full p7zip-rar sharutils rar uudeview mpack arj cabextract rename liblzma-dev python-pip brotli lz4 protobuf-compiler git gawk
pip install backports.lzma protobuf pycrypto twrpdtgen extract-dtb pycryptodome

Mac

brew install protobuf liblzma-dev brotli lz4
pip install backports.lzma protobuf pycrypto twrpdtgen extract-dtb

Also install mono

Windows

Install cygwin, and select

Latest python and pip packages, arj, brotli, cabextract, dos2unix, lz4, p7zip, renameutils, sharutils, unace, unzip and zip

If you get syntax errors run dos2unix on extractor.sh

Misc

• Install rust using https://rustup.rs/
• Run git clone https://github.com/crazystylus/otadump , cd otadump/, cargo build --release.
• Navigate to /target/release and add the generated otadump executable to your PATH.

How to use

Download

git clone --recurse-submodules https://github.com/AndroidDumps/Firmware_extractor.git

Extract images from firmware URL

Example: Extracting images from pixel 2 factory image:

cd Firmware_extractor
wget https://dl.google.com/dl/android/aosp/walleye-pq3a.190705.001-factory-cc471c8c.zip -o firmware.zip
./extractor.sh firmware.zip

output will be on "Firmware_extractor/out"

firmware_extractor's Issues

BRUH: This type of firmwares not supported(pac file)

I tried to use the tool on a pac and failed unpacked it using toll in tools directory and try tool fail used super.img and fail what do I do (btw I use termux and I didn't install some dependencies )

./extractor.sh outdir/super.img Already up to date. Already up to date. Create Temp and out dir BRUH: This type of firmwares not supported

Before you ask for files my net is slow (really really slow)
./extractor.sh outdir/super.img
Already up to date.
Already up to date.
Create Temp and out dir
BRUH: This type of firmwares not supported
I tried to use termux for pac and failed then tried after unpacking from termux and failed now I have Ubuntu 20.04 and failed for BOTH

Suggestion: consider using `otadump` in place of `payload-dumper-go`

Hi! I'm the author of otadump. It works similar to payload-dumper-go, but is 2-5x faster and verifies extracted partitions to be extra safe.

I recently posted on Reddit and a commenter mentioned that this tool might be useful to you. Would you accept a PR to integrate otadump into your codebase?

Thanks!

CC: @crazystylus

feat: Add CI tests to prevent regressions

extractor.sh code is complicated, and preventing regressions whenever code is modified or added takes time and effort.

An idea to prevent this would be to add tests so that commits wouldn't be able to be merged unless a sample set of each supported firmware successfully outputs IMGs, and that their hashes match those previously version controlled in this repository.

For A-only OTA, https://otafsg1.h2os.com/patch/amazone2/GLO/OnePlus3TOxygen/OnePlus3TOxygen_28.A.86_GLO_086_1911042121/OnePlus3TOxygen_28_OTA_086_all_1911042121_9156030ead54e.zip can be used.
For Oplus A-only OTA, https://oxygenos.oneplus.net/system_vendor.zip can be used
For A/B OTAs, https://gauss-componentotacostmanual-sg.allawnofs.com/remove-60818dddbcc4ff1e05001dbecdd5a9b3/component-ota/22/03/27/d7a3625cd77440418671588fe07b4a6b.zip can be used.

I will edit content of this issue later to add MD5s of the extracted IMGs.

A problem though is I am unsure how to implement this idea in code, also maybe some of the supported firmware isn't available to download via an anonymous link. There would likely be a need to store it on the machine on which one tests will run to prevent issues such as firmware being unavailable, or downloading extremely slowly, thereby making CI tests run for a long amount of time or fail.

extraction fails

Hello,

i am trying to extract Samsung Galaxy A15 rom but it looks like there is a problem with super partition extraction

I am using an ARM64 Linux server to do this...please tell if thats the problem.

ubuntu@instance-20231104-0510:~/Firmware_extractor$ neofetch
.-/+oossssoo+/-. ubuntu@instance-20231104-0510
:+ssssssssssssssssss+: -----------------------------
-+ssssssssssssssssssyyssss+- OS: Ubuntu 22.04.4 LTS aarch64
.ossssssssssssssssssdMMMNysssso. Host: KVM Virtual Machine virt-4.
/ssssssssssshdmmNNmmyNMMMMhssssss/ Kernel: 6.5.0-1023-oracle
+ssssssssshmydMMMMMMMNddddyssssssss+ Uptime: 3 days, 11 hours, 27 mins
/sssssssshNMMMyhhyyyyhmNMMMNhssssssss/ Packages: 1092 (dpkg), 6 (snap)
.ssssssssdMMMNhsssssssssshNMMMdssssssss. Shell: bash 5.1.16
+sssshhhyNMMNyssssssssssssyNMMMysssssss+ Resolution: 1024x768
ossyNMMMNyMMhsssssssssssssshmmmhssssssso Terminal: /dev/pts/2
ossyNMMMNyMMhsssssssssssssshmmmhssssssso CPU: (4)
+sssshhhyNMMNyssssssssssssyNMMMysssssss+ GPU: 00:01.0 Red Hat, Inc. Virtio
.ssssssssdMMMNhsssssssssshNMMMdssssssss. Memory: 272MiB / 23981MiB
/sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
+sssssssssdmydMMMMMMMMddddyssssssss+
/ssssssssssshdmNNNNmyNMMMMhssssss/
.ossssssssssssssssssdMMMNysssso.
-+sssssssssssssssssyyyssss+-
:+ssssssssssssssssss+:
.-/+oossssoo+/-.

ubuntu@instance-20231104-0510:~/Firmware_extractor$ wget https://07-samfw.cloud/v2/IxJCDiMnLgE1MRoULy8HODQvMxwPCB4lFzssIDs2ByAzMUEgOzEUQDMQMCMBOyw/LjEzNTxAKQIeJSkGMjYUAh4bBy0PCDMaARYhNh4sLhoyOwI/MgAvPDgxBhoDLy88LghBIyMbHhI1EQgkMiUeBTILBxI8LwgfAQQxMCMlFTE7Jy8dOwc+CS84BkIeCzwGHgM8FCMkNDEjCxVACTghHzInDQYuFx8rIwAvJCFAMxE8EQY5NRshDjURPho0ES8kNRYhOR44LwsJEQc5NBsjKyEAPg4hAAYsLjgeLDQRHg4yJTMsMiUNKzwAPh8mGzk5FyUzDTUHQQEuByADISxBIwMnOUANQAokMzYCICMXPjkDFzA1CS8pDTsxHhwzQAZCOzEHHjMsNCsNMT4eODAeHgk7CAYeAy8nODFBHDMwPiAjAzswMywpPC8DQQQXCDsxOAAhKTw/FD01OCBCJicxMB5ANDkPOzACO0AzJA8lMwU4ACwkLiUHHBIvKS07By8eNSRBAB4nIRcjOAZCIwM0BgEDIh0jJCIAAQ0TEw==
--2024-06-22 17:33:41-- https://07-samfw.cloud/v2/IxJCDiMnLgE1MRoULy8HODQvMxwPCB4lFzssIDs2ByAzMUEgOzEUQDMQMCMBOyw/LjEzNTxAKQIeJSkGMjYUAh4bBy0PCDMaARYhNh4sLhoyOwI/MgAvPDgxBhoDLy88LghBIyMbHhI1EQgkMiUeBTILBxI8LwgfAQQxMCMlFTE7Jy8dOwc+CS84BkIeCzwGHgM8FCMkNDEjCxVACTghHzInDQYuFx8rIwAvJCFAMxE8EQY5NRshDjURPho0ES8kNRYhOR44LwsJEQc5NBsjKyEAPg4hAAYsLjgeLDQRHg4yJTMsMiUNKzwAPh8mGzk5FyUzDTUHQQEuByADISxBIwMnOUANQAokMzYCICMXPjkDFzA1CS8pDTsxHhwzQAZCOzEHHjMsNCsNMT4eODAeHgk7CAYeAy8nODFBHDMwPiAjAzswMywpPC8DQQQXCDsxOAAhKTw/FD01OCBCJicxMB5ANDkPOzACO0AzJA8lMwU4ACwkLiUHHBIvKS07By8eNSRBAB4nIRcjOAZCIwM0BgEDIh0jJCIAAQ0TEw==
Resolving 07-samfw.cloud (07-samfw.cloud)... 188.114.97.3, 188.114.96.3, 2a06:98c1:3120::3, ...
Connecting to 07-samfw.cloud (07-samfw.cloud)|188.114.97.3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7203139280 (6.7G) [application/zip]
Saving to: ‘FD01OCBCJicxMB5ANDkPOzACO0AzJA8lMwU4ACwkLiUHHBIvKS07By8eNSRBAB4nIRcjOAZCIwM0BgEDIh0jJCIAAQ0TEw==’

FD01OCBCJicxMB5ANDk 100%[================>] 6.71G 55.5MB/s in 94s

2024-06-22 17:35:16 (72.7 MB/s) - ‘FD01OCBCJicxMB5ANDkPOzACO0AzJA8lMwU4ACwkLiUHHBIvKS07By8eNSRBAB4nIRcjOAZCIwM0BgEDIh0jJCIAAQ0TEw==’ saved [7203139280/7203139280]

ubuntu@instance-20231104-0510:/Firmware_extractor$ mv FD01OCBCJicxMB5ANDkPOzACO0AzJA8lMwU4ACwkLiUHHBIvKS07By8eNSRBAB4nIRcjOAZCIwM0BgEDIh0jJCIAAQ0TEw== SAMFW.COM_SM-A155F_BOG_A155FXXU2BXE4_fac.zip ubuntu@instance-20231104-0510:/Firmware_extractor$ ./extractor.sh SAMFW.COM_SM-A155F_BOG_A155FXXU2BXE4_fac.zip Create Temp and out dir Extracting firmware on: /home/ubuntu/Firmware_extractor/out AP tarmd5 detected Extracting tarmd5
Extracting images...
removed 'AP_A155FXXU2BXE4_A155FXXU2BXE4_MQB81097780_REV00_user_low_ship_MULTI_CERT_meta_OS14.tar.md5'
Extracted AP_A155FXXU2BXE4_A155FXXU2BXE4_MQB81097780_REV00_user_low_ship_MULTI_CERT_meta_OS14.tar.md5
removed 'BL_A155FXXU2BXE4_A155FXXU2BXE4_MQB81097780_REV00_user_low_ship_MULTI_CERT.tar.md5'
Extracted BL_A155FXXU2BXE4_A155FXXU2BXE4_MQB81097780_REV00_user_low_ship_MULTI_CERT.tar.md5
removed 'CP_A155FXXU2BXE2_CP26527504_MQB80529102_REV00_user_low_ship_MULTI_CERT.tar.md5'
Extracted CP_A155FXXU2BXE2_CP26527504_MQB80529102_REV00_user_low_ship_MULTI_CERT.tar.md5
removed 'CSC_OXM_A155FOXM2BXE3_QB80611278_REV00_user_low_ship_MULTI_CERT.tar.md5'
Extracted CSC_OXM_A155FOXM2BXE3_QB80611278_REV00_user_low_ship_MULTI_CERT.tar.md5
removed 'HOME_CSC_OXM_A155FOXM2BXE3_QB80611278_REV00_user_low_ship_MULTI_CERT.tar.md5'
Extracted HOME_CSC_OXM_A155FOXM2BXE3_QB80611278_REV00_user_low_ship_MULTI_CERT.tar.md5
removed 'boot.img.lz4'
Extracted boot.img.lz4
removed 'cache.img.lz4'
Extracted cache.img.lz4
removed 'dpm-verified.img.lz4'
Extracted dpm-verified.img.lz4
removed 'dtbo.img.lz4'
Extracted dtbo.img.lz4
removed 'efuse.img.lz4'
Extracted efuse.img.lz4
removed 'gz-verified.img.lz4'
Extracted gz-verified.img.lz4
removed 'lk-verified.img.lz4'
Extracted lk-verified.img.lz4
removed 'mcupm-verified.img.lz4'
Extracted mcupm-verified.img.lz4
removed 'md1img.img.lz4'
Extracted md1img.img.lz4
removed 'misc.bin.lz4'
Extracted misc.bin.lz4
removed 'omr.img.lz4'
Extracted omr.img.lz4
removed 'optics.img.lz4'
Extracted optics.img.lz4
removed 'param.bin.lz4'
Extracted param.bin.lz4
removed 'pi_img-verified.img.lz4'
Extracted pi_img-verified.img.lz4
removed 'preloader.img.lz4'
Extracted preloader.img.lz4
removed 'prism.img.lz4'
Extracted prism.img.lz4
removed 'recovery.img.lz4'
Extracted recovery.img.lz4
removed 'scp-verified.img.lz4'
Extracted scp-verified.img.lz4
removed 'spmfw-verified.img.lz4'
Extracted spmfw-verified.img.lz4
removed 'sspm-verified.img.lz4'
Extracted sspm-verified.img.lz4
removed 'super.img.lz4'
Extracted super.img.lz4
removed 'tee-verified.img.lz4'
Extracted tee-verified.img.lz4
removed 'tzar.img.lz4'
Extracted tzar.img.lz4
removed 'uh.bin.lz4'
Extracted uh.bin.lz4
removed 'up_param.bin.lz4'
Extracted up_param.bin.lz4
removed 'userdata.img.lz4'
Extracted userdata.img.lz4
removed 'vbmeta.img.lz4'
Extracted vbmeta.img.lz4
removed 'vbmeta_system.img.lz4'
Extracted vbmeta_system.img.lz4
removed 'vendor_boot.img.lz4'
Extracted vendor_boot.img.lz4
Creating super.img.raw ...
Extract failed

Can't dump QFIL image properly

Create Temp and out dir
grep: warning: stray \ before /
Extracting firmware on: /home/yuifunami/dumpyara/working/Archive
QFIL detected
flag needs an argument: -x
Usage of /home/yuifunami/dumpyara/Firmware_extractor/tools/Linux/bin/packsparseimg:
  -o string
        output dir path (default "./")
  -t string
        image to unsparse: system / cache / userdata / persist / vendor (default "system")
  -x string
        xml file to load (default "./rawprogram_unsparse.xml")
mv: cannot stat 'system.raw': No such file or directory
flag needs an argument: -x
Usage of /home/yuifunami/dumpyara/Firmware_extractor/tools/Linux/bin/packsparseimg:
  -o string
        output dir path (default "./")
  -t string
        image to unsparse: system / cache / userdata / persist / vendor (default "system")
  -x string
        xml file to load (default "./rawprogram_unsparse.xml")
mv: cannot stat 'vendor.raw': No such file or directory
flag needs an argument: -x
Usage of /home/yuifunami/dumpyara/Firmware_extractor/tools/Linux/bin/packsparseimg:
  -o string
        output dir path (default "./")
  -t string
        image to unsparse: system / cache / userdata / persist / vendor (default "system")
  -x string
        xml file to load (default "./rawprogram_unsparse.xml")
mv: cannot stat 'modem.raw': No such file or directory
flag needs an argument: -x
Usage of /home/yuifunami/dumpyara/Firmware_extractor/tools/Linux/bin/packsparseimg:
  -o string
        output dir path (default "./")
  -t string
        image to unsparse: system / cache / userdata / persist / vendor (default "system")
  -x string
        xml file to load (default "./rawprogram_unsparse.xml")
mv: cannot stat 'dtbo.raw': No such file or directory
flag needs an argument: -x
Usage of /home/yuifunami/dumpyara/Firmware_extractor/tools/Linux/bin/packsparseimg:
  -o string
        output dir path (default "./")
  -t string
        image to unsparse: system / cache / userdata / persist / vendor (default "system")
  -x string
        xml file to load (default "./rawprogram_unsparse.xml")
mv: cannot stat 'dtb.raw': No such file or directory
flag needs an argument: -x
Usage of /home/yuifunami/dumpyara/Firmware_extractor/tools/Linux/bin/packsparseimg:
  -o string
        output dir path (default "./")
  -t string
        image to unsparse: system / cache / userdata / persist / vendor (default "system")
  -x string
        xml file to load (default "./rawprogram_unsparse.xml")
mv: cannot stat 'boot.raw': No such file or directory
flag needs an argument: -x
Usage of /home/yuifunami/dumpyara/Firmware_extractor/tools/Linux/bin/packsparseimg:
  -o string
        output dir path (default "./")
  -t string
        image to unsparse: system / cache / userdata / persist / vendor (default "system")
  -x string
        xml file to load (default "./rawprogram_unsparse.xml")
mv: cannot stat 'tz.raw': No such file or directory

Firmware dumped myself using EDL tool, it fails to process it properly. It does have all partitions needed in raw image .bin files.

Newer Spreadtrum/Unisoc .pac support

.pac files for newer devices having super partition have size of greater than 4GiB and can't be handled by existing pacextractor.
So what about using https://github.com/jingpad-bsp/vendor_sprd_tools_ota/blob/58ae4a1775854158a48ab30b0d9d006c043950e7/otatool/prebuilts/linux-x86/ota/pac_tools/unpac_perl/unpac.pl @deadman96385 ?

Devices having .pac files greater than 4GiB that I know are Realme C11 2021, Realme C21Y, Realme C35.