shivamkumarjha / android_tools

Bash scripts for ROM development stuff.

Android tools

Collection of scripts to help with Android ROM stuff.

Setup:
chmod +x setup.sh
sudo bash setup.sh

1. blobs_downloader.sh: A script to download selected blobs from AndroidBlobs etc repo. Example.
   Usage: ./tools/blobs_downloader.sh <raw dump repo URL> <path to proprietary-files.txt>

2. common_blobs.sh: A script to list common and device specific blob's between two ROM's.
   Usage: ./tools/common_blobs.sh <path to source rom dump> <path to target rom dump>

3. common_props.sh: A script to list common and device specific prop's between two ROM's.
   Usage: ./tools/common_props.sh <path to source rom dump> <path to target rom dump>

4. deltaota.sh: A script to extract delta OTA.
   Usage: ./tools/deltaota.sh <path to full OTA> <path to incremental OTA(s)>

5. dt_repos.sh: A script to create empty device, kernel & vendor tree of a device in GitHub with model as repo descripton. Usage:
   export GIT_TOKEN=<KEY>
./tools/dt_repos.sh <path to rom>

6. dummy_dt.sh: A script which prepares a dummy device & vendor tree from a ROM dump.
   Usage: ./tools/dummy_dt.sh <path to ROM dump>

7. dump_push.sh: A script to push local ROM dump to GitHub.
   Usage:
   export GIT_TOKEN=<KEY>
./tools/dump_push.sh <path to dump>

8. proprietary-files.sh: A script to prepare proprietary blobs list from a ROM.
   Usage:
   For online git repo: ./tools/proprietary-files.sh <raw file link of all_files.txt>
   For local dump: ./tools/proprietary-files.sh <path to ROM dump OR path to all_files.txt>

9. rebase_kernel.sh: A script to rebase OEM compressed kernel source to its best CAF base.
   Usage: ./tools/rebase_kernel.sh <kernel zip link/file> <repo name>

10. rom_compare.sh: A script to compare source & target ROM. It lists Added, common, missing & modified blobs.
    Usage: ./tools/rom_compare.sh <path to source ROM dump> <path to target ROM dump>

11. rom_extract.sh: A script to extract OTA files.
    Usage: ./tools/rom_extract.sh <path to OTA file(s)>

12. rootdir.sh: A script to prepare ramdisk from a ROM dump along with Makefile.
    Usage: ./tools/rootdir.sh <path to ROM dump>

13. system_vendor_prop.sh: A script to prepare properties Makefile from a ROM dump. (Does not support lahaina, use vendor_prop.sh instead)
    Usage: ./tools/system_vendor_prop.sh <path to ROM dump>
    Output: system.prop & vendor_prop.mk files.

14. vendor_prop.sh: A script to prepare and filter properties Makefile from a ROM dump.
    Usage: ./tools/vendor_prop.sh <path to ROM dump>
    Output: vendor_prop.mk file.

15. vendor_tree.sh: A script to prepare vendor tree from a ROM dump after generating proprietary-files.txt and push it to GitHub.
    To extract from a specific proprietary-files.txt, place it before in working/proprietary-files.txt.
    Usage:
    export GIT_TOKEN=<KEY>
./tools/vendor_tree.sh <path to ROM dump>

android_tools's Issues

system.prop and vendor_prop.mk are empty

i used the system_vendor_prop.sh script to generate props from vince_global dump on androiddumps repo, and the generated files were empty. this is the output i got when i ran the command.

KenHV@kenhv:~/android_tools$ ./tools/system_vendor_prop.sh ~/dump
find: '/home/KenHV/dump/vendor': No such file or directory
./tools/system_vendor_prop.sh: line 45: 41
51: syntax error in expression (error token is "51")
sed: -e expression #1, char 3: unknown command: `
'
sed: -e expression #1, char 1: unknown command: `,'
/home/KenHV/android_tools/working/system.prop
/home/KenHV/android_tools/working/vendor_prop.mk prepared!

Can't use blobs_downloader.sh for oppo Reno 6 5G

./tools/blobs_downloader.sh out proprietary-files.txt
cp: cannot stat 'proprietary-files.txt': No such file or directory
sed: can't read /home/harshita-billi/android_tools/working/list.txt: No such file or directory
sed: can't read /home/harshita-billi/android_tools/working/list.txt: No such file or directory
sed: can't read /home/harshita-billi/android_tools/working/list.txt: No such file or directory
cat: /home/harshita-billi/android_tools/working/list.txt: No such file or directory

Finished. Check /home/harshita-billi/android_tools/working

but i have already created 'proprietary-files.txt' file

About system requirements

Hello.
Just found out about this "Swiss Army Knife". I'm trying to run the proprietary-files.sh, system_vendor_prop.sh and vendor_prop.sh scripts, but with a lot of bugs. I not sure if this is a bug from the tool very own, a problem with my Ubuntu version, a problem with my Python version, my pip version or even with the Android dump version (Marshmallow).
This tool along with Android Blob Utility seems to be very promising for ROMs development, but it would be easier for new users give a proper feedback with a more detailed description of System Requirements.

Vendor_prop issue

./tools/system_vendor_prop.sh /home/user/dumpyara/working/XT1802_SANDERS_RETBR_8.1.0_OPS28.65-36-14_cid50_subsidy-DEFAULT_regulatory-DEFAULT_CFC.xml/system
./tools/system_vendor_prop.sh: line 45: 85
95: syntax error in expression (error token is "95")
sed: -e expression #1, char 3: unknown command: ' sed: -e expression #1, char 1: unknown command:,'
/home/user/dumpyara/android_tools/working/system.prop
/home/user/dumpyara/android_tools/working/vendor_prop.mk prepared!
vendor prop and system.prop is empty

Error extracting Vendor Tree updated situation with logs

Says file not found even though they exist...I checked the ROM dump manually and the files exist.

Add example for making dummy device tree

Please add some examples for making dummy device tree so people can easily making dummy device tree

protobuf error probably need code update

here is the log when i tried to extract any a/b ota,

AB OTA detected
Traceback (most recent call last):
File "/home/harun/android_tools/tools/Firmware_extractor/tools/extract_android_ota_payload/extract_android_ota_payload.py", line 13, in
import update_metadata_pb2
File "/home/harun/android_tools/tools/Firmware_extractor/tools/extract_android_ota_payload/update_metadata_pb2.py", line 34, in
_descriptor.EnumValueDescriptor(
File "/usr/local/lib/python3.10/dist-packages/google/protobuf/descriptor.py", line 755, in new
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:

1. Downgrade the protobuf package to 3.20.x or lower.
2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
No system.img found. Exiting

@ShivamKumarJha thanks for the reliable tools you made anyway..

Some scripts are not working

I have run "tools/rom_extract.sh" ,but in the dumps folder ,the "Recovery" and "tz" folders are blank.What's more,when I ran tools/dummy_dt.sh , it show me find: '/home/seimo/android_tools/ishtar.zip/': Not a directory ,but I'm sure I give it a file.

However,I have no more logs could be gave.I think maybe it's because of Android 13 or erosfs file system?I'm new for this thing,please help me.

Device: Xiaomi 13 Ultra (ishtar)
Android version of the unpacked ROM: Android 13
Working system: Ubuntu 22.04.3 LTS
The ROM: Download

Nothing downloaded. Repo dir is empty. None file downloaded

PATH: /home/Scissordragonboy/android/working/vendor/lib/libmmjpeg.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libmmqjpeg_codec.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libmmqjpegdma.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libmpbase.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libois_lc898122.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/liboptizoom.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libqomx_jpegdec.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libqomx_jpegenc.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libqomx_jpegenc_pipe.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libremosaic_daemon.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libseemore.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libtrueportrait.so
PATH: /home/Scissordragonboy/android/working/vendor/lib/libubifocus.so

Finished. Check /home/Scissordragonboy/android/working
Scissordragonboy@dump:/android$ cd working
Scissordragonboy@dump:/android/working$ ls
bin etc lib lib64 list.txt vendor
Scissordragonboy@dump:/android/working$ cd vendor/lib
Scissordragonboy@dump:/android/working/vendor/lib$ ls
Scissordragonboy@dump:/android/working/vendor/lib$ cd ..
Scissordragonboy@dump:/android/working/vendor$ cd lib
Scissordragonboy@dump:/android/working/vendor/lib$ ls
Scissordragonboy@dump:/android/working/vendor/lib$ cd ..
Scissordragonboy@dump:/android/working/vendor$ cd ..
Scissordragonboy@dump:/android/working$ cd lib
Scissordragonboy@dump:~/android/working/lib$ ls
GNU nano 2.9.3 list.txt
etc/acdbdata/QRD/QRD_Bluetooth_cal.acdb:vendor/etc/acdbdata/QRD/QRD_Bluetooth_cal.acdb
etc/acdbdata/QRD/QRD_General_cal.acdb:vendor/etc/acdbdata/QRD/QRD_General_cal.acdb
etc/acdbdata/QRD/QRD_Global_cal.acdb:vendor/etc/acdbdata/QRD/QRD_Global_cal.acdb
etc/acdbdata/QRD/QRD_Handset_cal.acdb:vendor/etc/acdbdata/QRD/QRD_Handset_cal.acdb
etc/acdbdata/QRD/QRD_Hdmi_cal.acdb:vendor/etc/acdbdata/QRD/QRD_Hdmi_cal.acdb
etc/acdbdata/QRD/QRD_Headset_cal.acdb:vendor/etc/acdbdata/QRD/QRD_Headset_cal.acdb
etc/acdbdata/QRD/QRD_Speaker_cal.acdb:vendor/etc/acdbdata/QRD/QRD_Speaker_cal.acdb
bin/mm-qcamera-daemon:vendor/bin/mm-qcamera-daemon
vendor/lib/libChamomilePA.so
vendor/lib/libchromaflash.so
vendor/lib/libflash_pmic.so
vendor/lib/libjpegdhw.so
vendor/lib/libjpegdmahw.so
vendor/lib/libjpegehw.so
vendor/lib/libmmcamera2_c2d_module.so
vendor/lib/libmmcamera2_cpp_module.so
vendor/lib/libmmcamera2_dcrf.so
vendor/lib/libmmcamera2_frame_algorithm.so
vendor/lib/libmmcamera2_iface_modules.so
vendor/lib/libmmcamera2_imglib_modules.so
vendor/lib/libmmcamera2_is.so
vendor/lib/libmmcamera2_isp_modules.so
vendor/lib/libmmcamera2_mct.so
vendor/lib/libmmcamera2_mct_shimlayer.so
vendor/lib/libmmcamera2_pp_buf_mgr.so
vendor/lib/libmmcamera2_pproc_modules.so
vendor/lib/libmmcamera2_q3a_core.so
vendor/lib/libmmcamera2_sensor_modules.so
vendor/lib/libmmcamera2_stats_algorithm.so
vendor/lib/libmmcamera2_stats_modules.so
vendor/lib/libmmcamera_atmel_at24c32e_eeprom.so
vendor/lib/libmmcamera_chromaflash_lib.so
vendor/lib/libmmcamera_csidtg.so
vendor/lib/libmmcamera_dbg.so
vendor/lib/libmmcamera_dcrf_lib.so
vendor/lib/libmmcamera_dummyalgo.so
vendor/lib/libmmcamera_dw9761b_2d_eeprom.so
vendor/lib/libmmcamera_eebinparse.so
vendor/lib/libmmcamera_eeprom_util.so
vendor/lib/libmmcamera_eztune_module.so
vendor/lib/libmmcamera_facedetection_lib.so
vendor/lib/libmmcamera_faceproc.so
vendor/lib/libmmcamera_faceproc2.so
vendor/lib/libmmcamera_hdr_gb_lib.so
vendor/lib/libmmcamera_imglib.so
vendor/lib/libmmcamera_imx258.so
vendor/lib/libmmcamera_imx258_gt24c16_eeprom.so
vendor/lib/libmmcamera_imx258_gt24c32_eeprom.so
vendor/lib/libmmcamera_imx258_mono.so
vendor/lib/libmmcamera_imx258_mono_gt24c32_eeprom.so
vendor/lib/libmmcamera_imx258_ofilm.so
vendor/lib/libmmcamera_imx258_qtech.so
vendor/lib/libmmcamera_imx258_sunny.so
vendor/lib/libmmcamera_isp_abf40.so
vendor/lib/libmmcamera_isp_bcc40.so

Scissordragonboy@dump:/android/working$ ls
bin etc lib lib64 list.txt vendor
Scissordragonboy@dump:/android/working$ cd etc
Scissordragonboy@dump:/android/working/etc$ ls
:vendor acdbdata camera
Scissordragonboy@dump:/android/working/etc$ cd camera
Scissordragonboy@dump:/android/working/etc/camera$ ls
:vendor
Scissordragonboy@dump:/android/working/etc/camera$ cd :vemdor
-bash: cd: :vemdor: No such file or directory
Scissordragonboy@dump:/android/working/etc/camera$ cd :vendor
Scissordragonboy@dump:/android/working/etc/camera/:vendor$ ls
etc
Scissordragonboy@dump:/android/working/etc/camera/:vendor$ cd etc
Scissordragonboy@dump:/android/working/etc/camera/:vendor/etc$ ls
camera
Scissordragonboy@dump:/android/working/etc/camera/:vendor/etc$ cd camera
Scissordragonboy@dump:/android/working/etc/camera/:vendor/etc/camera$ ls
Scissordragonboy@dump:/android/working/etc/camera/:vendor/etc/camera$ cd
Scissordragonboy@dump:$ cd android

New Hotword blobs on Android 11

Hotword functionality requires the following files on Android 11

system_ext/etc/permissions/com.android.hotwordenrollment.common.util.xml
system_ext/framework/com.android.hotwordenrollment.common.util.jar

These file are currently listed under Misc section

Reference:
LineageOS/android_device_xiaomi_sdm845-common@3ed901c
LineageOS/android_device_oneplus_sm8250-common@d3a5d47

Android tools

Error Extracting vendor

Extracting vendor
Traceback (most recent call last):
  File "/home/hadenix/Загрузки/android_tools/tools/Firmware_extractor/tools/sdat2img.py", line 142, in <module>
    main(sys.argv)
  File "/home/hadenix/Загрузки/android_tools/tools/Firmware_extractor/tools/sdat2img.py", line 87, in main
    version, new_blocks, commands = parse_transfer_list_file(TRANSFER_LIST_FILE)
  File "/home/hadenix/Загрузки/android_tools/tools/Firmware_extractor/tools/sdat2img.py", line 55, in parse_transfer_list_file
    trans_list = open(TRANSFER_LIST_FILE, 'r')
FileNotFoundError: [Errno 2] No such file or directory: 'vendor.transfer.list'

rebase-kernel: need a update for Specifying Branch!!

Bruh Why Can't We Specific The CAF Branch We Need Fo Merge?? the automated one is really bad which is merging the old tags. So please update it.
@ShivamKumarJha

Setup script need adjustment to work with Ubuntu 20.04 Focal Fossa

• python-pip package is deprecated
• replace pip to pip3

And,

Package android-tools-fsutils is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  android-sdk-libsparse-utils android-sdk-ext4-utils

E: Package 'android-tools-fsutils' has no installation candidate

Potentially replace it with one of these packages.

If you need me to make a PR for it please let me know.

Minor derp in dummy_dt function

When generating dummy dt, the PRODUCT_MANUFACTURER variable get sets with a minor derp

android_tools/tools/dummy_dt.sh

Line 275 in 483c4b4

printf "\nPRODUCT_MANUFACTURER := "$BRAND"" >> "$DT_DIR"/lineage_"$DEVICE".mk

For instance, in the case of Xiaomi devices, it gets written as
PRODUCT_MANUFACTURER := xiaomi [1]
instead of
PRODUCT_MANUFACTURER := Xiaomi [2]

1. https://github.com/AndroidBlobs/device_xiaomi_surya/blob/46efa22aa8274a6d919961fa2843f831c71c0cce/lineage_surya.mk#L13
2. https://github.com/LineageOS/android_device_xiaomi_surya/blob/4f03c8bd9cd3b51e67b815ea4eaafd8a032e811d/lineage_surya.mk#L31

on other OEMs, incorrect values gets written, eg. OnePlus

Not working on arm device

Is the script mainly made for arm64 devices?
The proprietary-files.txt as well as vendor tree generated with this script on an arm device skips a lot of files. Please make it compitable for arm also..

it doesnt work on nougat?

cause i tried to generate an dummy tree it shows pre oreo roms not supported

proprietary-files.txt is not properly filled

./tools/androidblobs.sh /home/user/firmware/miui_GRUSEEAGlobal_V10.3.4.0.PFBEUXM_409f6dbb6f_9.0.zip
Binary file (standard input) matches
/home/user/firmware/tools/android_tools/working/proprietary-files.txt prepared!
BRAND: 
DEVICE: 
DESCRIPTION: 
FINGERPRINT: 
MODEL: 
SECURITY_PATCH: 
VERSION: 
FLAVOR: 
ID: 
INCREMENTAL: 
TAGS: 
$DEVICE must be set before including this script!

➜  working git:(master) ✗ cat proprietary-files.txt 

# Audio

# Bluetooth-AptX
file

# Camera

# DRM

# Display-Hardware

# GPS

# GPS-Hardware

# Graphics

# Listen

# Media

# Media-Hardware

# Perf

# Postprocessing

# QMI

# Radio

# Sensors

# Time

# WiFi

# Misc

vendor prop issue

./tools/system_vendor_prop.sh /home/user/dumpyara/working/XT1802_SANDERS_RETBR_8.1.0_OPS28.65-36-14_cid50_subsidy-DEFAULT_regulatory-DEFAULT_CFC.xml
find: '/home/user/dumpyara/working/XT1802_SANDERS_RETBR_8.1.0_OPS28.65-36-14_cid50_subsidy-DEFAULT_regulatory-DEFAULT_CFC.xml/vendor': No such file or directory
./tools/system_vendor_prop.sh: line 45: 85
95: syntax error in expression (error token is "95")
sed: -e expression #1, char 3: unknown command: ' sed: -e expression #1, char 1: unknown command:,'
/home/user/dumpyara/android_tools/working/system.prop
/home/user/dumpyara/android_tools/working/vendor_prop.mk prepared! or
./tools/system_vendor_prop.sh /home/user/dumpyara/working/XT1802_SANDERS_RETBR_8.1.0_OPS28.65-36-14_cid50_subsidy-DEFAULT_regulatory-DEFAULT_CFC.xml/
find: '/home/user/dumpyara/working/XT1802_SANDERS_RETBR_8.1.0_OPS28.65-36-14_cid50_subsidy-DEFAULT_regulatory-DEFAULT_CFC.xml//vendor': No such file or directory
./tools/system_vendor_prop.sh: line 45: 85
95: syntax error in expression (error token is "95")
sed: -e expression #1, char 3: unknown command: ' sed: -e expression #1, char 1: unknown command:,'
/home/elias_souza/dumpyara/android_tools/working/system.prop