nlitsme / ubidump Goto Github PK

I need to extract files from a ubi image.
I use command: python ubidump.py 205108.ubi -s save
but I got this result:
==> 205108.ubi <==
1 named volumes found, 2 physical volumes, blocksize=0x20000
== volume app ==
E: magic num mismatch

I had checked the ubi image, there were some "UBI#".
I don't know how to solve it? Could you help me?
205108.zip

watz@antares:~/Downloads$ ubidump/ubidump.py --listfiles 2-rootfs1.bin
==> 2-rootfs1.bin <==
4 named volumes found, 5 physical volumes, blocksize=0x40000
== volume b'rootfs_ubifs' ==
volume contains a squashfs filesystem, extract with --saveraw, and then use unsquashfs
E: node magic num mismatch
== volume b'METADATA' ==
unknown magic: 1b000000
E: node magic num mismatch
== volume b'METADATACOPY' ==
unknown magic: 1b000000
E: node magic num mismatch
== volume b'filestruct_full_debug.bin' ==
unknown magic: ef1a0000
E: node magic num mismatch
watz@antares:~/Downloads$

Help says "--saveraw FILENAME save the entire volume to the specified file"

Which one of the volumes? How can I specify it?

Thanks!

c:\ubidumptest\ubidump.py -s 12 1.ubifs
ERROR 'module' object has no attribute 'symlink'

When decompressing the ubifs file on the windows system, the creation fails because the windows system does not have symlink

change
elif typ == inode.ITYPE_SYMLINK:
os.symlink(inode.data, fullpath)

to
import platform

            elif typ == inode.ITYPE_SYMLINK:
                if(platform.system()=='Windows'):
                    print("create symlink %s" % fullpath)
                    open(fullpath+'_symlink', 'wb').write('symlink to fullpath')
                else:
                    os.symlink(inode.data, fullpath)  

You can continue to unzip the file

Hello, I encountered such an error while using your tool. Can you help me solve it?

error log ：
ERROR Could not determine UBI image blocksize

Command:

username@ localhost in ~/Desktop/ubidump $ python3  ubidump.py -v -c /etc/passwd 21486C.ubi

Output (Error):

==> 21486C.ubi <==
1 named volumes found, 2 physical volumes, blocksize=0x20000

===== block =====
EC: magic=b'UBI#', v1, ec=0, vidhdr=800, data=1000, imgseq=69e6d86e  VID: magic=b'UBI!', v1, vt=1, cp=0, compat=5, volid=7fffefff, lnum=[0], dsize=0, usedebs=0, datapad=0, datacrc=0, sqnum=0
  VREC: rsvpebs=234, align=1, datapad=0, voltype=1, updmark=0, flags=1, name=b'system'
volume 7fffefff : 2 lebs
volume 0 : 234 lebs

== volume b'system' ==

*** superblock ***
SUPER: kh:0, fmt:0, flags=0, minio=2048, lebsize=0x1f000, lebcount=234, maxleb=984, maxbud=8388608, loglebs=5, lptlebs=2, orphlebs=1, jheads=1, fanout=8, lsave=256, fmt=v4, compr=2, rp=0:0, rpsize=0, timegran=1000000000, uuid=b'2898977b6e52472c9069e4b7d8476cd9', rocompat=0

*** masterblock ***
MST: max_inum=02347, cmtno=0, flags=2, loglnum=3, root=[233:0x06648], rootlen=128, gc_lnum=[228], ihead=[233:0x06800], ixsize=533960, total(free:286720, dirty:193488, used:27301480, dead:185504, dark:52328), lpt=[08:0x0049c], nhead=[08:0x00800], ltab=[08:0x004a8], lsave=[00:0x00000], lscan=[11], empty=1, idx=5, nleb=234


ERROR a bytes-like object is required, not 'int'
Traceback (most recent call last):
  File "ubidump.py", line 936, in main
    processfile(fn, args)
  File "ubidump.py", line 919, in processfile
    fs.savefile(inum, sys.stdout, args.cat)
  File "ubidump.py", line 804, in savefile
    dat = c.getnode()
  File "ubidump.py", line 728, in getnode
    return self.fs.readnode(page.branches[ix].lnum, page.branches[ix].offs)
  File "ubidump.py", line 647, in readnode
    node.parse(nodedata)
  File "ubidump.py", line 337, in parse
    self.data = decompress(data[self.hdrsize:], self.size, self.compr_type)
  File "ubidump.py", line 291, in decompress
    return unzlib(buflen)
  File "ubidump.py", line 284, in unzlib
    return zlib.decompress(data)
TypeError: a bytes-like object is required, not 'int'

I had a nand image with ecc corrected data. But when i try to dump/read the files on any folder i get this error message:

==> root.bin.dd <==
1 named volumes found, 2 physical volumes, blocksize=0x20000
== volume rootfs ==
E: [Errno 13] Permission denied: '/etc/rootfs'

How to have acess to this file system?

Thanks!

Hello sir i have a box Enigma2 VU+ solo2 and he use Ubifs File System In Their Image

root_cfe_auto.bin The extract Ubifs Is OK No problem Butwhen i Edit The content Of This Image

and Repack with Mkfs.ubifs and I reflash My Box VU+ solo2 the receiver not Start and In the fron On the box stay In "Starting....."

can you help me for This Sir

thank you

python ubidump.py -s rfs file.ubi
==> file.ubi <==
1 named volumes found, 2 physical volumes, blocksize=0x20000
== volume rootfs ==
ERROR function takes exactly 1 argument (3 given)

Here's the traceback with verbose mode on:
Traceback (most recent call last):
File "ubidump.py", line 939, in main
processfile(fn, args)
File "ubidump.py", line 899, in processfile
fs.savefile(inum, fh, os.path.join(path))
File "ubidump.py", line 804, in savefile
dat = c.getnode()
File "ubidump.py", line 727, in getnode
return self.fs.readnode(page.branches[ix].lnum, page.branches[ix].offs)
File "ubidump.py", line 646, in readnode
node.parse(nodedata)
File "ubidump.py", line 334, in parse
self.data = decompress(data[self.hdrsize:], self.size, self.compr_type)
File "ubidump.py", line 286, in decompress
return unlzo(data, buflen)
File "ubidump.py", line 278, in unlzo
return lzo.decompress(data, False, buflen)
TypeError: function takes exactly 1 argument (3 given)

ubidump has trouble with this file, as you can see it is a 5 volume ubi the output is the same on more than one pc. I only need the rfs issue solved I have a workaround for the other volumes but not for the rfs

$ ubidump --s ./ systemcut.ubi
==> systemcut.ubi <==
5 named volumes found, 3 physical volumes, blocksize=0x40000
== volume rootfs ==
E: global name 'PermissionError' is not defined
== volume usrfs ==
E: volume does not contain lnum
== volume cachefs ==
E: volume does not contain lnum
== volume systemrw ==
E: volume does not contain lnum
== volume persist ==
E: volume does not contain lnum

If I break it down to ubifs, here is the output

ubidump --s ./ --debug rootfs.ubifs
==> rootfs.ubifs <==
ERROR global name 'PermissionError' is not defined
Traceback (most recent call last):
File "/home/bin/ubidump", line 11, in
sys.exit(main())
File "/home/lib/python2.7/site-packages/ubidump.py", line 1561, in main
processfile(fn, args)
File "/home/lib/python2.7/site-packages/ubidump.py", line 1518, in processfile
processvolume(RawVolume(fh), b"raw", args)
File "/home/lib/python2.7/site-packages/ubidump.py", line 1334, in processvolume
except PermissionError as e:
NameError: global name 'PermissionError' is not defined

any help would be great thank you

Hello, i am using ubidump.py - for unpacking UBI, all OK, super tools. but one device used hardlink, and it is lost. when I extracted files. therefore, when a new UBI is created, it grows in size. how difficult it is to fix/add ? Thanks.

ubidump.py old.ubi  --dumptree | grep "inum=00707"
 0 BRANCH: ref=[061:0x16458] len=  63 -- key=00695:2:0ac5ac7a ->  [061:0x16458-0x16497] DIRENT: key=00695:2:0ac5ac7a, inum=00707, type=0:reg -- b'mke2fs'
 6 BRANCH: ref=[062:0x0c490] len=  69 -- key=00695:2:0e26a803 ->  [062:0x0c490-0x0c4d5] DIRENT: key=00695:2:0e26a803, inum=00707, type=0:reg -- b'mkfs.ext4dev'
 0 BRANCH: ref=[062:0x0f370] len=  66 -- key=00695:2:19e760aa ->  [062:0x0f370-0x0f3b2] DIRENT: key=00695:2:19e760aa, inum=00707, type=0:reg -- b'mkfs.ext2'
 1 BRANCH: ref=[062:0x0c448] len=  66 -- key=00695:2:19e7615a ->  [062:0x0c448-0x0c48a] DIRENT: key=00695:2:19e7615a, inum=00707, type=0:reg -- b'mkfs.ext3'
 2 BRANCH: ref=[062:0x14140] len=  66 -- key=00695:2:19e7620a ->  [062:0x14140-0x14182] DIRENT: key=00695:2:19e7620a, inum=00707, type=0:reg -- b'mkfs.ext4'

ubidump.py old.ubi --listfiles | grep 95176
-rwxr-xr-x  5 0     0          95176 2021-08-12 10:22:28 sbin/mke2fs
-rwxr-xr-x  5 0     0          95176 2021-08-12 10:22:28 sbin/mkfs.ext4dev
-rwxr-xr-x  5 0     0          95176 2021-08-12 10:22:28 sbin/mkfs.ext2
-rwxr-xr-x  5 0     0          95176 2021-08-12 10:22:28 sbin/mkfs.ext3
-rwxr-xr-x  5 0     0          95176 2021-08-12 10:22:28 sbin/mkfs.ext4

ls -lih | grep 93K
38155996 -rwxr-xr-x 1 root root  93K Aug 12  2021 mke2fs
38156044 -rwxr-xr-x 1 root root  93K Aug 12  2021 mkfs.ext2
38156045 -rwxr-xr-x 1 root root  93K Aug 12  2021 mkfs.ext3
38156046 -rwxr-xr-x 1 root root  93K Aug 12  2021 mkfs.ext4
38156002 -rwxr-xr-x 1 root root  93K Aug 12  2021 mkfs.ext4dev

Ran into some issues on Ubuntu 16.04.3 LTS and python-lzo (1.08-1):

$ python2.7 ubidump.py -s test/ test.ubi --verbose
==> test.ubi <==
1 named volumes found, 2 physical volumes, blocksize=0x20000
===== block =====
EC: magic=UBI#, v1, ec=0, vidhdr=800, data=1000, imgseq=24cfe7e9  VID: magic=UBI!, v1, vt=1, cp=0, compat=5, volid=7fffefff, lnum=[0], dsize=0, usedebs=0, datapad=0, datacrc=0, sqnum=0
  VREC: rsvpebs=826, align=1, datapad=0, voltype=1, updmark=0, flags=1, name=rootfs
volume 0 : 588 lebs
volume 7fffefff : 2 lebs
== volume rootfs ==
*** superblock ***
SUPER: kh:0, fmt:0, flags=4, minio=2048, lebsize=0x1f000, lebcount=588, maxleb=2047, maxbud=8388608, loglebs=5, lptlebs=2, orphlebs=1, jheads=1, fanout=8, lsave=256, fmt=v4, compr=1, rp=0:0, rpsize=0, timegran=1000000000, uuid=a8c44348f43d4c059429b3348700911e, rocompat=0
*** masterblock ***
MST: max_inum=03760, cmtno=0, flags=2, loglnum=3, root=[587:0x0f420], rootlen=68, gc_lnum=[577], ihead=[587:0x0f800], ixsize=1204736, total(free:356352, dirty:474928, used:71229136, dead:448768, dark:146832), lpt=[08:0x00c05], nhead=[08:0x01000], ltab=[08:0x00c11], lsave=[00:0x00000], lscan=[11], empty=1, idx=10, nleb=588
ERROR function takes exactly 1 argument (3 given)
Traceback (most recent call last):
  File "ubidump.py", line 942, in main
    processfile(fn, args)
  File "ubidump.py", line 902, in processfile
    fs.savefile(inum, fh, os.path.join(path))
  File "ubidump.py", line 807, in savefile
    dat = c.getnode()
  File "ubidump.py", line 730, in getnode
    return self.fs.readnode(page.branches[ix].lnum, page.branches[ix].offs)
  File "ubidump.py", line 649, in readnode
    node.parse(nodedata)
  File "ubidump.py", line 337, in parse
    self.data = decompress(data[self.hdrsize:], self.size, self.compr_type)
  File "ubidump.py", line 289, in decompress
    return unlzo(data, buflen)
  File "ubidump.py", line 281, in unlzo
    return lzo.decompress(data, False, buflen)
TypeError: function takes exactly 1 argument (3 given)

Seemed that “lzo.decompress” expected one argument, so I removed the two additional parameters, and got the following error:

$ python2.7 ubidump.py -s test/ test.ubi --verbose
==> test.ubi <==
1 named volumes found, 2 physical volumes, blocksize=0x20000
===== block =====
EC: magic=UBI#, v1, ec=0, vidhdr=800, data=1000, imgseq=24cfe7e9  VID: magic=UBI!, v1, vt=1, cp=0, compat=5, volid=7fffefff, lnum=[0], dsize=0, usedebs=0, datapad=0, datacrc=0, sqnum=0
  VREC: rsvpebs=826, align=1, datapad=0, voltype=1, updmark=0, flags=1, name=rootfs
volume 0 : 588 lebs
volume 7fffefff : 2 lebs
== volume rootfs ==
*** superblock ***
SUPER: kh:0, fmt:0, flags=4, minio=2048, lebsize=0x1f000, lebcount=588, maxleb=2047, maxbud=8388608, loglebs=5, lptlebs=2, orphlebs=1, jheads=1, fanout=8, lsave=256, fmt=v4, compr=1, rp=0:0, rpsize=0, timegran=1000000000, uuid=a8c44348f43d4c059429b3348700911e, rocompat=0
*** masterblock ***
MST: max_inum=03760, cmtno=0, flags=2, loglnum=3, root=[587:0x0f420], rootlen=68, gc_lnum=[577], ihead=[587:0x0f800], ixsize=1204736, total(free:356352, dirty:474928, used:71229136, dead:448768, dark:146832), lpt=[08:0x00c05], nhead=[08:0x01000], ltab=[08:0x00c11], lsave=[00:0x00000], lscan=[11], empty=1, idx=10, nleb=588
ERROR Header error - invalid compressed data
Traceback (most recent call last):
  File "ubidump.py", line 943, in main
    processfile(fn, args)
  File "ubidump.py", line 903, in processfile
    fs.savefile(inum, fh, os.path.join(path))
  File "ubidump.py", line 808, in savefile
    dat = c.getnode()
  File "ubidump.py", line 731, in getnode
    return self.fs.readnode(page.branches[ix].lnum, page.branches[ix].offs)
  File "ubidump.py", line 650, in readnode
    node.parse(nodedata)
  File "ubidump.py", line 338, in parse
    self.data = decompress(data[self.hdrsize:], self.size, self.compr_type)
  File "ubidump.py", line 290, in decompress
    return unlzo(data, buflen)
  File "ubidump.py", line 279, in unlzo
    return lzo.decompress(data)
error: Header error - invalid compressed data

Some research led me to this: https://irdataaddict.wordpress.com/2015/03/17/python-lzo/. Apparently “lzo.decompress” freaks out if no valid LZO header is present (need to confirm that buflen is the equivalent of the decompressed_size):

def unlzo(data, buflen):
    header = '\xf0' + pack('>I', buflen)
    return lzo.decompress(header + data)

I’m then able to perform what appears to be a proper extraction:

$ python2.7 ../ubidump.py -s extract/ test.ubi --verbose
==> test.ubi <==
1 named volumes found, 2 physical volumes, blocksize=0x20000
===== block =====
EC: magic=UBI#, v1, ec=1, vidhdr=800, data=1000, imgseq=78377008  VID: magic=UBI!, v1, vt=1, cp=0, compat=5, volid=7fffefff, lnum=[0], dsize=0, usedebs=0, datapad=0, datacrc=0, sqnum=2
  VREC: rsvpebs=1105, align=1, datapad=0, voltype=1, updmark=0, flags=0, name=rootfs
volume 0 : 583 lebs
volume 7fffefff : 2 lebs
== volume rootfs ==
*** superblock ***
SUPER: kh:0, fmt:0, flags=4, minio=2048, lebsize=0x1f000, lebcount=556, maxleb=2047, maxbud=8388608, loglebs=5, lptlebs=2, orphlebs=1, jheads=1, fanout=8, lsave=256, fmt=v4, compr=1, rp=0:0, rpsize=0, timegran=1000000000, uuid=ad06b37975ab4dccb012abc6cc7f58eb, rocompat=0
*** masterblock ***
MST: max_inum=03614, cmtno=0, flags=2, loglnum=3, root=[555:0x1e3b8], rootlen=68, gc_lnum=[546], ihead=[555:0x1e800], ixsize=1139120, total(free:339968, dirty:441824, used:67281008, dead:413928, dark:185792), lpt=[08:0x00b59], nhead=[08:0x01000], ltab=[08:0x00b65], lsave=[00:0x00000], lscan=[11], empty=1, idx=9, nleb=556
saved 2404 files

Hello,

Is not yet an issue with ubidump but in the steps to mount in order to could use it.

Was following the steps to perform the mount but I am struggling at this step.

1. Dumped full NAND using JTAG moving all content into RAM and after dumping into file.

2. Performing "strings" and HEX visualization seems correct dump

3. Trying to mount it as UBIFS but fails in last step.

Output.

laptop@laptop ~ $ sudo modprobe nandsim first_id_byte=0x01 second_id_byte=0xF1 third_id_byte=0x00 fourth_id_byte=0x1D
laptop@laptop ~ $ sudo modprobe ubi
laptop@laptop ~ $ sudo modprobe ubifs

laptop@laptop ~ $ sudo dd if=wholechip.bin of=/dev/mtd0
262144+0 registos dentro
262144+0 registos fora
134217728 bytes (134 MB) copiados, 1,16179 s, 116 MB/s

laptop@laptopL ~ $ sudo ubiattach --mtdn=0
UBI device number 0, total 1024 LEBs (132120576 bytes, 126.0 MiB), available 998 LEBs (128765952 bytes, 122.8 MiB), LEB size 129024 bytes (126.0 KiB)
laptop@laptop ~ $

laptop@laptop ~ $ cat /proc/mtd
dev: size erasesize name
mtd0: 08000000 00020000 "NAND simulator partition 0"

laptop@laptop ~ $ ls -ls /dev/ubi*
0 crw------- 1 root root 243, 0 Mar 27 21:53 /dev/ubi0
0 crw------- 1 root root 10, 54 Mar 27 21:51 /dev/ubi_ctrl

laptop@laptop ~$ sudo mount -t ubifs -o ro /dev/ubi0_0 /mnt/ubifs/
mount: wrong fs type, bad option, bad superblock on /dev/ubi0_0,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount. helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so

laptop@laptop ~$ dmesg | tail
[ 257.406948] ubi0: scanning is finished
[ 257.407873] ubi0: attached mtd0 (name "NAND simulator partition 0", size 128 MiB)
[ 257.407879] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 129024 bytes
[ 257.407882] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 512
[ 257.407885] ubi0: VID header offset: 512 (aligned 512), data offset: 2048
[ 257.407888] ubi0: good PEBs: 1024, bad PEBs: 0, corrupted PEBs: 0
[ 257.407891] ubi0: user volume: 0, internal volumes: 1, max. volumes count: 128
[ 257.407894] ubi0: max/mean erase counter: 2/1, WL threshold: 4096, image sequence number: 360987593
[ 257.407897] ubi0: available PEBs: 998, total reserved PEBs: 26, PEBs reserved for bad PEB handling: 20
[ 257.407907] ubi0: background thread "ubi_bgt0d" started, PID 3144

Any one have any idea what I am doing wrong?

Is there anyway to preserve the permissions of the directories and files when unpacking them with the --savedir option? Like with fakeroot or something similar?

Hi @nlitsme,

is it possible to modify the code and allow the "--preserve" switch to export UID and GID?

I had lot of problem on repacking a firmware from a vendor, because they use some custom UID\GID, and when exported with ubidump, all stuff is owned by root.

I've just change that part in this way:

            if args.preserve and typ != inode.ITYPE_SYMLINK:
                # note: we have to do this after closing the file, since the close after exportfile
                # will update the last-modified time.
                os.utime(fullpath, (inode.atime(), inode.mtime()))
                os.chmod(fullpath, inode.mode)
                os.chown(fullpath, inode.uid, inode.gid)

Adding os.chown function will export files with their original UID\GID.

I can create a pull request if you want.

Thanks!

This software is really Excellent work.
Is it possible to support zstd decompression?

Hello,
trying to list contents of a UBI image and get the following error. I am not sure I follow entirely your code but I discovered that volid in this line when 0 and does not find a vol entry in vtbl, throws an exception, while in my ubi image, the valid volume is in entry [1] in vtbl. I can provide a link offline to the UBI image.

Please see a picture during debugging a working UBI file (mtdb3) and one that gives the error (mtdb4)
Dank je!

ubidump -v -l mtdb4.bin
==> mtdb4.bin <==
1 named volumes found, 2 physical volumes, blocksize=0x20000
===== block =====
EC: magic=UBI#, v1, ec=0, vidhdr=800, data=1000, imgseq=6d52bfe2 VID: magic=UBI!, v1, vt=1, cp=0, compat=5, volid=7fffefff, lnum=[0], dsize=0, usedebs=0, datapad=0, datacrc=0, sqnum=1
VREC: rsvpebs=3057, align=1, datapad=0, voltype=1, updmark=0, flags=0, name=app_fs
volume 1 : 962 lebs
volume 7fffefff : 2 lebs
== volume ==
ERROR volume does not contain lnum

Traceback (most recent call last):
File "ubidump", line 1025, in main
processfile(fn, args)
File "ubidump", line 954, in processfile
fs = UbiFs(vol)
File "ubidump", line 675, in init
self.load()
File "ubidump", line 687, in load
self.sb = self.readnode(0, 0)
File "ubidump", line 704, in readnode
hdrdata = self.vol.read(lnum, offs, ch.hdrsize)
File "ubidump", line 181, in read
return self.blks.readvolume(self.volid, lnum, self.dataofs+offs, size)
File "ubidump", line 300, in readvolume
raise Exception("volume does not contain lnum")
Exception: volume does not contain lnum

I can enumerate files in a ubi archive with:
ubidump -l bar.ubi

Then I can try to extract a binary file with:
ubidump -c foo.bin bar.ubi > foo.bin

But I end up with three lines of text preceding the binary.
These lines are output of the program itself.
The lines have to be manually stripped out.
Perhaps an -o option for outputting to a file?

Over-scoping a little bit, but I'd like to see an "extract all" option too.

Mounting doesn't work great on WSL 1 (If one were to get at the contents that way).
binwalk doesn't seem to extract the ubi format IIRC.
This could be a great addition to binwalk with a little work.

I'll be glad to make a pull request if there's an active maintainer.

Feature request:
Command line parameter to include files which are deleted but which data was not overwritten.

I have no idea how UbiFS marks deleted files, and whether it's easy to recover them. But I de-ubinized an image (by attaching it and doing proper dd) and looked at the content - some of the data I found was not included in "--savedir" output. I found file names which no longer exist in the filesystem - so maybe names are preserved in folders. Or maybe the directory content was just written to different sector after the file was deleted.

Either way, it should be possible to recover files; if not by processing entries marked as deleted, then by searching the whole image for old directory entries in unused blocks.

Anyway, great tool - thank you!