cyxx / extract_android_ota_payload Goto Github PK

https://ota-oversea-1256119282.file.myqcloud.com/SKW-H0_EEA/2019-09-19_2_skywalker_mp_os_ota4_20190904-2019-07-04_skywalker_mp_os_ota3_20190618_OTA3_EEA.20190920_154535.zip

Hi,

I'm running Slackware 14.2 x64 and I have python3 3.6.6 installed, and when I run the script I get this at the terminal;

Traceback (most recent call last):
File "extract_android_ota_payload.py", line 12, in
import update_payload
File "/home/foo/Downloads/update_payload/init.py", line 9, in
from checker import CHECKS_TO_DISABLE
ModuleNotFoundError: No module named 'checker'

I've noticed the command showing running it on a zip so I tried it that way too;

python3 extract_android_ota_payload.py CARBON-CR-6.1-NOCT-RELEASE-mata-20180811-0405.zip /home/foo/Downloads/CR

But then I also extracted the zip and ran the command on the payload.bin, I extracted the rom and placed the contents of extract_android_ota_payload in the same directory as below;

META-INF
README.md
care_map.txt
compatibility.zip
extract_android_ota_payload.py
payload.bin
payload_properties.txt
update_payload

But I get the same message;

Traceback (most recent call last):
File "extract_android_ota_payload.py", line 12, in
import update_payload
File "/home/foo/Downloads/CARBON-CR-6.1-NOCT-RELEASE-mata-20180811-0405/update_payload/init.py", line 9, in
from checker import CHECKS_TO_DISABLE
ModuleNotFoundError: No module named 'checker'

Thanks

When trying to run the script using python2 on Termux (all dependencies installed), I get the following error:

Extracting 'boot.img' Traceback (most recent call last): File "./extract_android_ota_payload.py", line 144, in <module> main(sys.argv) File "./extract_android_ota_payload.py", line 138, in main except update_payload.error.PayloadError as e: NameError: global name 'update_payload' is not defined

The example specifies
python extract_android_ota_payload.py marlin-ota-opm4.171019.021.d1-fd6998a5.zip / tmp /
But this caused an error.
I called
python3 extract_android_ota_payload.py marlin-ota-opm4.171019.021.d1-fd6998a5.zip / tmp /
And it worked. Could you correct the description?
In the header of the script too

Hello, first of all, a huge thanks for your work. I have been using your program on Windows for quite a while.

However, I recently tried it on my Macbook Air with macOS 10.15.4. I tried the following command:
python3 extract_android_ota_payload.py payload.bin
This gives me the following error. May I know if I missed something or what went wrong?

Getting this to work on Windows was a pain.
In the end I accomplished it by installing MinGW.
The relevant code change was on line 78 as follows:
p = subprocess.Popen(['C:\\MinGW\\msys\\1.0\\bin\\'+command+'.exe', '-'], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
Where command is bzcat and xzcat respectively.

There's probably a fancier way to accomplish this using Windows Subsystem for Linux, but I'm not fully across that yet.

While trying to extract a LineageOS 17.1 OTA it failed with:

Traceback (most recent call last):
  File "extract_android_ota_payload.py", line 13, in <module>
    import update_metadata_pb2
  File "/media/martin/extLinux/developer/android/tools/extract_android_ota_payload/update_metadata_pb2.py", line 23, in <module>
    serialized_pb=_b('\n\x15update_metadata.proto\x12\x16\x63hromeos_update_engine\"1\n\x06\x45xtent\x12\x13\n\x0bstart_block\x18\x01 \x01(\x04\x12\x12\n\nnum_blocks\x18\x02 \x01(\x04\"z\n\nSignatures\x12@\n\nsignatures\x18\x01 \x03(\x0b\x32,.chromeos_update_engine.Signatures.Signature\x1a*\n\tSignature\x12\x0f\n\x07version\x18\x01 \x01(\r\x12\x0c\n\x04\x64\x61ta\x18\x02 \x01(\x0c\"+\n\rPartitionInfo\x12\x0c\n\x04size\x18\x01 \x01(\x04\x12\x0c\n\x04hash\x18\x02 \x01(\x0c\"w\n\tImageInfo\x12\r\n\x05\x62oard\x18\x01 \x01(\t\x12\x0b\n\x03key\x18\x02 \x01(\t\x12\x0f\n\x07\x63hannel\x18\x03 \x01(\t\x12\x0f\n\x07version\x18\x04 \x01(\t\x12\x15\n\rbuild_channel\x18\x05 \x01(\t\x12\x15\n\rbuild_version\x18\x06 \x01(\t\"\xee\x03\n\x10InstallOperation\x12;\n\x04type\x18\x01 \x02(\x0e\x32-.chromeos_update_engine.InstallOperation.Type\x12\x13\n\x0b\x64\x61ta_offset\x18\x02 \x01(\x04\x12\x13\n\x0b\x64\x61ta_length\x18\x03 \x01(\x04\x12\x33\n\x0bsrc_extents\x18\x04 \x03(\x0b\x32\x1e.chromeos_update_engine.Extent\x12\x12\n\nsrc_length\x18\x05 \x01(\x04\x12\x33\n\x0b\x64st_extents\x18\x06 \x03(\x0b\x32\x1e.chromeos_update_engine.Extent\x12\x12\n\ndst_length\x18\x07 \x01(\x04\x12\x18\n\x10\x64\x61ta_sha256_hash\x18\x08 \x01(\x0c\x12\x17\n\x0fsrc_sha256_hash\x18\t \x01(\x0c\"\xad\x01\n\x04Type\x12\x0b\n\x07REPLACE\x10\x00\x12\x0e\n\nREPLACE_BZ\x10\x01\x12\x0c\n\x04MOVE\x10\x02\x1a\x02\x08\x01\x12\x0e\n\x06\x42SDIFF\x10\x03\x1a\x02\x08\x01\x12\x0f\n\x0bSOURCE_COPY\x10\x04\x12\x11\n\rSOURCE_BSDIFF\x10\x05\x12\x0e\n\nREPLACE_XZ\x10\x08\x12\x08\n\x04ZERO\x10\x06\x12\x0b\n\x07\x44ISCARD\x10\x07\x12\x11\n\rBROTLI_BSDIFF\x10\n\x12\x0c\n\x08PUFFDIFF\x10\t\"\xd7\x05\n\x0fPartitionUpdate\x12\x16\n\x0epartition_name\x18\x01 \x02(\t\x12\x17\n\x0frun_postinstall\x18\x02 \x01(\x08\x12\x18\n\x10postinstall_path\x18\x03 \x01(\t\x12\x17\n\x0f\x66ilesystem_type\x18\x04 \x01(\t\x12M\n\x17new_partition_signature\x18\x05 \x03(\x0b\x32,.chromeos_update_engine.Signatures.Signature\x12\x41\n\x12old_partition_info\x18\x06 \x01(\x0b\x32%.chromeos_update_engine.PartitionInfo\x12\x41\n\x12new_partition_info\x18\x07 \x01(\x0b\x32%.chromeos_update_engine.PartitionInfo\x12<\n\noperations\x18\x08 \x03(\x0b\x32(.chromeos_update_engine.InstallOperation\x12\x1c\n\x14postinstall_optional\x18\t \x01(\x08\x12=\n\x15hash_tree_data_extent\x18\n \x01(\x0b\x32\x1e.chromeos_update_engine.Extent\x12\x38\n\x10hash_tree_extent\x18\x0b \x01(\x0b\x32\x1e.chromeos_update_engine.Extent\x12\x1b\n\x13hash_tree_algorithm\x18\x0c \x01(\t\x12\x16\n\x0ehash_tree_salt\x18\r \x01(\x0c\x12\x37\n\x0f\x66\x65\x63_data_extent\x18\x0e \x01(\x0b\x32\x1e.chromeos_update_engine.Extent\x12\x32\n\nfec_extent\x18\x0f \x01(\x0b\x32\x1e.chromeos_update_engine.Extent\x12\x14\n\tfec_roots\x18\x10 \x01(\r:\x01\x32\"L\n\x15\x44ynamicPartitionGroup\x12\x0c\n\x04name\x18\x01 \x02(\t\x12\x0c\n\x04size\x18\x02 \x01(\x04\x12\x17\n\x0fpartition_names\x18\x03 \x03(\t\"s\n\x18\x44ynamicPartitionMetadata\x12=\n\x06groups\x18\x01 \x03(\x0b\x32-.chromeos_update_engine.DynamicPartitionGroup\x12\x18\n\x10snapshot_enabled\x18\x02 \x01(\x08\"\xb1\x06\n\x14\x44\x65ltaArchiveManifest\x12\x44\n\x12install_operations\x18\x01 \x03(\x0b\x32(.chromeos_update_engine.InstallOperation\x12K\n\x19kernel_install_operations\x18\x02 \x03(\x0b\x32(.chromeos_update_engine.InstallOperation\x12\x18\n\nblock_size\x18\x03 \x01(\r:\x04\x34\x30\x39\x36\x12\x19\n\x11signatures_offset\x18\x04 \x01(\x04\x12\x17\n\x0fsignatures_size\x18\x05 \x01(\x04\x12>\n\x0fold_kernel_info\x18\x06 \x01(\x0b\x32%.chromeos_update_engine.PartitionInfo\x12>\n\x0fnew_kernel_info\x18\x07 \x01(\x0b\x32%.chromeos_update_engine.PartitionInfo\x12>\n\x0fold_rootfs_info\x18\x08 \x01(\x0b\x32%.chromeos_update_engine.PartitionInfo\x12>\n\x0fnew_rootfs_info\x18\t \x01(\x0b\x32%.chromeos_update_engine.PartitionInfo\x12\x39\n\x0eold_image_info\x18\n \x01(\x0b\x32!.chromeos_update_engine.ImageInfo\x12\x39\n\x0enew_image_info\x18\x0b \x01(\x0b\x32!.chromeos_update_engine.ImageInfo\x12\x18\n\rminor_version\x18\x0c \x01(\r:\x01\x30\x12;\n\npartitions\x18\r \x03(\x0b\x32\'.chromeos_update_engine.PartitionUpdate\x12\x15\n\rmax_timestamp\x18\x0e \x01(\x03\x12T\n\x1a\x64ynamic_partition_metadata\x18\x0f \x01(\x0b\x32\x30.chromeos_update_engine.DynamicPartitionMetadataB\x02H\x03')
TypeError: __new__() got an unexpected keyword argument 'serialized_options'

I have issues extracting images from realme GT master edition payload.bin. The tool throws errors on erofs images. payload_dumper tool however works fine.

OTA Link : https://download.c.realme.com/flash/Rollbackpack/realme_GT_Master_Edition/RMX3360_11_A_fullota_00011011.zip
System,system_ext,vendor,product,odm are erofs

Extracting 'system.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'system_ext.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'product.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'vbmeta_system.img'
Extracting 'cpucp.img'
Extracting 'featenabler.img'
Extracting 'hyp.img'
Extracting 'multiimgoem.img'
Extracting 'oplus_sec.img'
Extracting 'oplusstanvbk.img'
Extracting 'qweslicstore.img'
Extracting 'splash.img'
Extracting 'uefisecapp.img'
Extracting 'vm-bootsys.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'shrm.img'
Extracting 'boot.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'vendor_boot.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'vendor.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'odm.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'dtbo.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'vbmeta.img'
Extracting 'vbmeta_vendor.img'
Extracting 'xbl.img'
Extracting 'xbl_config.img'
Extracting 'aop.img'
Extracting 'tz.img'
Extracting 'modem.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'bluetooth.img'
Extracting 'abl.img'
Extracting 'dsp.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'keymaster.img'
Extracting 'devcfg.img'
Extracting 'qupfw.img'
Extracting 'imagefv.img'
Extracting 'my_product.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'my_engineering.img'
Extracting 'my_stock.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'my_carrier.img'
Extracting 'my_region.img'
Failed: Unhandled operation type (6 - ZERO)
Extracting 'my_heytap.img'

Hello, I do not have any idea what goes on behind the scenes in the python file, but the result doesn't seem to work for me.

I'm trying to extract a payload.bin from a Nokia 7 OTA file (~150MB). The phone does use A/B.

The command stars and then stops with just 2 files in the output directory. These are:

1. cda.img
2. splash.img

Here's the output from my terminal.

narender@narender-MS-7816:/media/narender/Windows 7 (x86 & x64)/Nokia 7 Root/extract_android_ota_payload-master$ python extract_android_ota_payload.py payload.b
in abc/
Extracting 'abl.img'
Failed: Unhandled operation type (4)
Extracting 'bluetooth.img'
Failed: Unhandled operation type (4)
Extracting 'boot.img'
Failed: Unhandled operation type (5)
Extracting 'cda.img'
Extracting 'cmnlib.img'
Failed: Unhandled operation type (4)
Extracting 'cmnlib64.img'
Failed: Unhandled operation type (4)
Extracting 'devcfg.img'
Failed: Unhandled operation type (4)
Extracting 'dsp.img'
Failed: Unhandled operation type (4)
Extracting 'hidden.img'
Failed: Unhandled operation type (4)
Extracting 'hyp.img'
Failed: Unhandled operation type (4)
Extracting 'keymaster.img'
Failed: Unhandled operation type (4)
Extracting 'mdtp.img'
Failed: Unhandled operation type (4)
Extracting 'mdtpsecapp.img'
Failed: Unhandled operation type (4)
Extracting 'modem.img'
Failed: Unhandled operation type (5)
Extracting 'nvdef.img'
Failed: Unhandled operation type (4)
Extracting 'pmic.img'
Failed: Unhandled operation type (4)
Extracting 'rpm.img'
Failed: Unhandled operation type (5)
Extracting 'splash.img'
Extracting 'system.img'
Failed: Unhandled operation type (5)
Extracting 'systeminfo.img'
Failed: Unhandled operation type (5)
Extracting 'tz.img'
Failed: Unhandled operation type (4)
Extracting 'xbl.img'
Failed: Unhandled operation type (4)
Extracting 'vendor.img'
Failed: Unhandled operation type (4)

python3 extract_android_ota_payload.py payload.bin

  File "extract_android_ota_payload.py", line 13, in <module>
    import update_metadata_pb2
  File "/home/ezra/git/extract_android_ota_payload/update_metadata_pb2.py", line 768, in <module>
    Extent = _reflection.GeneratedProtocolMessageType('Extent', (_message.Message,), {
TypeError: expected bytes, Descriptor found```

ssh@bash:~/ext$ python extract_android_ota_payload.py up.zip out
Extracting 'payload.bin' from OTA file...
Extracting 'abl.img'
Traceback (most recent call last):
File "extract_android_ota_payload.py", line 75, in
main(sys.argv)
File "extract_android_ota_payload.py", line 72, in main
parse_payload(payload, p, out_f)
File "extract_android_ota_payload.py", line 40, in parse_payload
raise update_payload.error.PayloadError('Unknown operation type (%d)' % operat
ion.type)
update_payload.error.PayloadError: Unknown operation type (4)

Would you accept a PR adding a Pipenv file? A one-off script like this seems like a perfect application of Pipenv. Happy to submit a PR.

 $ python ~/src/extract_android_ota_payload/extract_android_ota_payload.py payload.bin payload
Traceback (most recent call last):
  File "/home/jake/src/extract_android_ota_payload/extract_android_ota_payload.py", line 14, in <module>
    raise Exception("Python 2.x is required")
Exception: Python 2.x is required

and warning from pip2 install -r requirements.txt

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7

Hi, I would like to integrate this codebase into an MIT licenced project. Currently I can't do this due to this repo falling under "All Rights Reserved" due to the lack of a licence file. Could you please add a licence?

I am extracting oneplus 6 firmware (10.3.9), the extractor is stuck on system.img already for 20 minutes while it keeps freezing my PC and system.img does not change in size!

Hi! I was running into issues with the Python version and dependencies which I finally fixed by using a Docker container. This might be helpful for others as well?

To unpack payload.bin we start from the Docker image continuumio/miniconda to install dependencies via Miniconda:

docker run --name payload.bin --rm -it \
   -v `pwd`"/data":/opt/data \
   continuumio/miniconda /bin/bash

In the container we install protobuf and clone the project:

conda install protobuf --yes
git clone --depth=1 https://github.com/cyxx/extract_android_ota_payload.git /opt/bin

Next download and unzip and OTA image, e.g. for Oneplus:

url=https://oxygenos.oneplus.net/OnePlus6Oxygen_22_OTA_034_all_1909112343_31f86cec5f8d4c7b.zip
wget --output-document=/opt/data/image.zip $url
python -m zipfile -e /opt/data/image.zip /opt/data/image

And finally unpack the payload.bin:

python /opt/bin/extract_android_ota_payload.py /opt/data/image/payload.bin /opt/data/payload