When scanning a JPEG file successfully the following is printed to the logs:

Error closing: 'NoneType' object has no attribute 'close'

This error message is coming from pillow and probably this is because of load() that is called unnecessarily and can likely be removed without affecting functionality.

Error with Python's testtar.tar archive:

$ python3 bang-scanner -f /tmp/testtar.tar
--- Logging error ---
Traceback (most recent call last):
  File "/usr/lib64/python3.9/logging/__init__.py", line 1082, in emit
    stream.write(msg + self.terminator)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 50-56: surrogates not allowed
Call stack:
  File "/home/armijn/tmp/binaryanalysis-ng/src/bang-scanner", line 414, in <module>
    main(sys.argv)
  File "/home/armijn/tmp/binaryanalysis-ng/src/bang-scanner", line 294, in main
    process.start()
  File "/usr/lib64/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/lib64/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib64/python3.9/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/lib64/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib64/python3.9/multiprocessing/popen_fork.py", line 71, in _launch
    code = process_obj._bootstrap(parent_sentinel=child_r)
  File "/usr/lib64/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib64/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/armijn/tmp/binaryanalysis-ng/src/ScanJob.py", line 694, in processfile
    scanjob.check_entire_file(unpacker)
  File "/home/armijn/tmp/binaryanalysis-ng/src/ScanJob.py", line 556, in check_entire_file
    log(logging.DEBUG, "TRYING %s %s at offset: 0" %
  File "/home/armijn/tmp/binaryanalysis-ng/src/banglogging.py", line 7, in log
    logging.log(level, message)
Message: 'TRYING testtar.tar-0x00000000-tar-1/ustar/umlauts-\udcc4\udcd6\udcdc\udce4\udcf6\udcfc\udcdf script at offset: 0'

what should I do after nix-shell is ready?

When I run "python3 -m bang.cli scan -u xxx xxx", It shows error: Error while finding module specification for 'bang.cli' (ModuleNotFoundError: No module named 'bang')

When I run the bang-scanner I get:

Traceback (most recent call last):
  File "/usr/src/bang/src/ScanJob.py", line 689, in processfile
    scanjob.carve_file_data(unpacker)
  File "/usr/src/bang/src/ScanJob.py", line 481, in carve_file_data
    outfile = open(outfile_full, 'wb')
OSError: [Errno 36] File name too long: '/usr/src/bang/src/unpacked/bang-scan-zzd2vg9u/unpack/firmware.bin-0x00000076-lz4-1/unpacked-from-lz4-0x00def93d-synthesized-3/unpacked-0xdef93d-0x109123c-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900-0x00000000-synthesized-1/unpacked-0x0-0x2a1900'


Is there a way to prevent this?

there are various files that all have the same MZ signature. Right now BANG does not consistently unpack these: sometimes the PE unpacker is run first, sometimes the DOS MZ unpacker is run first. Merge these (as far as possible) so scanning is more consistent.

I've just been through the nix setup (ubuntu 23.04) and now at the stage of running for the first time, but i get the below error:

Are you able to expand on the usage info as there isn't much on the readme?
maybe some more examples or a yt video

thanks

[nix-shell:~/tools/binaryanalysis-ng]$ python3 -m bang.cli     
/nix/store/4agknr9yslk6rd1n5s45pgxlmpfb4vvq-python3-3.10.11-env/bin/python3.10: Error while finding module specification for 'bang.cli' (ModuleNotFoundError: No module named 'bang')

How does this relate to diffoscope?

Hi I am currently looking into the different unpackers of BANG. I notice that although BANG supports more than 100 different filetypes, only a few exists in the test directory. Do you have a test corpus of all the filetypes that BANG supports? If so, would you mind sharing the test corpus so I can also test BANG from my end? Thanks!

tagging duplicate files doesn't seem to work anymore

Can you tag a release and package it in nixpkgs?

Related to #26

After cloning today from master and building the Docker container:

$ docker build -t bang
(...)
  python3-pytz-2018.5-1.fc29.noarch
  python3-webencodings-0.5.1-6.fc29.noarch
  yajl-2.1.0-11.fc29.x86_64

Complete!
Removing intermediate container 727e3b8f8134
 ---> 798710dfd0fb
Step 5/5 : CMD ["python3","bangshell"]
 ---> Running in de71b6f2177b
Removing intermediate container de71b6f2177b
 ---> eceae9b2e1dc
Successfully built eceae9b2e1dc
Successfully tagged bang:latest

Then inside the container itself interactively, bang-scanner does not seem to behave as specified in the README.md's invocation:

$ docker run -it bang /bin/bash
[root@b6cc367e62c7 bang]# ./src/bang-scanner
bash: ./src/bang-scanner: Permission denied
[root@b6cc367e62c7 src]# chmod +x bang-scanner
[root@b6cc367e62c7 src]# ./bang-scanner
Traceback (most recent call last):
  File "./bang-scanner", line 52, in <module>
    import elasticsearch
ModuleNotFoundError: No module named 'elasticsearch'

The commit in 299da18 seems to break logging.

[davidak@ethmoid:~/code/binaryanalysis-ng]$ nix-shell 
error: undefined variable 'dockerfile-parse' at /home/davidak/code/binaryanalysis-ng/shell.nix:6:5
(use '--show-trace' to show detailed location information)

On which channel are you on?

• system: "x86_64-linux"
• host os: Linux 4.19.49, NixOS, 19.03.172866.4649b6ef4b5 (Koi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.2.2
• channels(root): "nixos-19.03.172979.8634c3b6199, nixos-hardware, nixos-unstable-19.09pre183392.83ba5afcc96"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

What is the expected behaviour for signature matching in the following situation?
signature = aba
string in which to search = abababc
There are three possibilities: position 0 only, position 2 only, or both positions. The current code picks the first alternative, but it might miss some signatures. Some signatures might overlap, e.g. ico and truetype combined with padding.

The new scanjob setup has a problem with broken links:

$ time python3 bang-scanner -c bang.config -f /tmp/TEW-636APB-1002.bin
Process Process-2:
Traceback (most recent call last):
File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "bang-scanner", line 1120, in processfile
scanjob.check_for_signatures(unpacker, fileresult, scanfilequeue, scanenvironment, unpackdirectory, temporarydirectory)
File "bang-scanner", line 378, in check_for_signatures
j = ScanJob(pathlib.Path(unpackedfile), unpackedlabel, self.filename, [], {})
File "bang-scanner", line 107, in init
self._stat_file()
File "bang-scanner", line 119, in _stat_file
self.stat = os.stat(self.filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/armijn/tmp/bang-scan-2dchgsnf/unpack/TEW-636APB-1002.bin-squashfs-1/usr/sbin/rc'

This file is a broken symlink.

The init() method of the ScanJob class calls self.stat_file() which throws an exception.

docker image build -t bang .
Sending build context to Docker daemon  2.027MB
Step 1/16 : FROM kaitai as builder
pull access denied for kaitai, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Makefile:52: recipe for target 'ctrbuild' failed
make: *** [ctrbuild] Error 1

I tried a few Docker images for kaitai:

• blacktop/kaitai only provides 0.8
• librespace/kaitai does not contain /kaitai_struct
• kaitai/ksv also does not contain /kaitai_struct

So where does the kaitai image come from?

So you can easily set up a python virtual env? If you would consider merging I can create a fork.
Regards,

Jaap

I tried to get binaryanalysis-ng working in docker. Here are the problems i encountered:

1. The docker instructions in the README doesn't work
2. In Dockerfile.kaitai the filename kaitai-struct-compiler-0.10-SNAPSHOT.zip is hardcoded but have since changed (same as issue #121 )

Step 7/12 : RUN unzip -d / jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip
 ---> Running in b74c7540079c
unzip:  cannot find or open jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip, jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip.zip or jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip.ZIP.
The command '/bin/sh -c unzip -d / jvm/target/universal/kaitai-struct-compiler-0.9-SNAPSHOT.zip' returned a non-zero code: 9
make: *** [Makefile:12: docker-kaitai] Error 9

3. src/Makefile assumes that kaitai-struct-compiler is in PATH but it is not

Step 5/16 : RUN make
 ---> Running in b21cb46187e5
kaitai-struct-compiler  -t python --outdir `dirname "parsers/font/pcf/pcf_font.ksy"` parsers/font/pcf/pcf_font.ksy
/bin/sh: 1: kaitai-struct-compiler: not found
Makefile:25: recipe for target 'parsers/font/pcf/pcf_font.py' failed
make: *** [parsers/font/pcf/pcf_font.py] Error 127
The command '/bin/sh -c make' returned a non-zero code: 2

4. src/Dockerfile assumes that bangshell is in the PATH which it is not
   python3: can't open file '/kaitai_struct/runtime/python/bangshell': [Errno 2] No such file or directory

I'm going to create a pull request that fixes these issues.

Besides that I think the following things could be changed:

1. It's a little confusing that the makefile target is called "docker-kaitai" but the default value for "CTR_BUILDER" is not docker. Maybe podman and docker should just have different makefile targets instead of being dependent on the "CTR_BUILDER" variable.
2. It's not clear how to invoke bang-scanner from the docker image.

Currently the assumption is that if the System Use field is used the data stored is a Rockridge extension. This is not necessarily true: in the '90s Apple used the System Use field to store HFS specific data (see http://www.reverse-engineering.info/CD/iso9660.pdf page 33)

From https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT section 4.3.9.1:

For ZIP64(tm) format archives, the compressed and uncompressed sizes are 8 bytes each.

Currently this is not supported correctly.

My test file openwrt-18.06.1-brcm2708-bcm2710-rpi-3-ext4-sysupgrade.img.gz contains an image, which file identifies as MBR image:

$ file openwrt-18.06.1-brcm2708-bcm2710-rpi-3-ext4-sysupgrade.img
openwrt-18.06.1-brcm2708-bcm2710-rpi-3-ext4-sysupgrade.img: DOS/MBR boot sector; partition 1 : ID=0xc, active, start-CHS (0x20,2,3), end-CHS (0xc3,0,12), startsector 8192, 40960 sectors; partition 2 : ID=0x83, start-CHS (0xe3,2,15), end-CHS (0x14,0,16), startsector 57344, 524288 sectors

Bang does not find it. It mistakenly identifies a FAT filesystem because it recognizes the magic number for MBR \x55\xaa. As a result, bang is trying many other filetypes and clutters the unpack directory.

There should be an unpacker for MBR images.

I used the Dockerfile to set up BANG and first, I couldn't figure out how to actually scan a file using the default bangshell. So instead, I modified the Dockerfile to just open up a bash instance and I'm trying to execute the following:

[root@0b7e5774a21c src]# python3 bang-scanner -c bang.config -f firmware.bin
I get:

Base unpack directory /root/tmp does not exist, exiting

At first, I thought this was due to where I had the . bin file but then I moved it to the current directory and I still get the error. In fact, I get it even if I just call python3 bang-scanner with no args at all.

What am I doing wrong? And Is there a way to do this from within the bangshell? Thanks!
Also note: root/tmp directory DOES exist...

When building the environment with nix-shell and then launching bang-scanner an error is thrown:

Traceback (most recent call last):
  File "bang-scanner", line 414, in <module>
    main(sys.argv)
  File "bang-scanner", line 87, in main
    options = BangScannerOptions().get()
  File "/home/armijn/tmp/binaryanalysis-ng/src/bangscanneroptions.py", line 43, in __init__
    self._read_configuration_file()
  File "/home/armijn/tmp/binaryanalysis-ng/src/bangscanneroptions.py", line 127, in _read_configuration_file
    self.config = configparser.ConfigParser(os.environ)
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/configparser.py", line 639, in __init__
    self._read_defaults(defaults)
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/configparser.py", line 1219, in _read_defaults
    self.read_dict({self.default_section: defaults})
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/configparser.py", line 752, in read_dict
    raise DuplicateOptionError(section, key, source)
configparser.DuplicateOptionError: While reading from '<dict>': option 'shell' in section 'DEFAULT' already exists

The bangscanneroptions.py code does the following:

self.config = configparser.ConfigParser(os.environ)

os.environ has an environment variable SHELL. The nix-shell command introduces a variable 'shell' to the environment. These conflict as configparser first lowercases everything before adding it to a ConfigParser instance.

Proposal: rewrite to something using YAML or so.

file test_password.7z found at https://github.com/fkie-cad/FACT_core/tree/master/src/plugins/unpacking/sevenz/test/data cannot be unpacked without hitting "enter".

When I execute the following command, I report the following error. How can I solve it?

yqbboy@ubuntu:~/binaryanalysis-ng-master/src$ python3 bang-scanner -c bang.config -f /home/yq/samples/
/usr/lib/python3/dist-packages/requests/init.py:80: RequestsDependencyWarning: urllib3 (1.25.10) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Traceback (most recent call last):
File "bang-scanner", line 55, in
from bangsignatures import maxsignaturesoffset
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 487, in
extension_to_unpackparser = get_unpackers_for_extensions()
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 481, in get_unpackers_for_extensions
for u in get_unpackers():
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 476, in get_unpackers
pathlib.Path(os.path.dirname(parsers.file)), pathlib.Path('.'))
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 470, in _get_unpackers_recursive
unpackers_root, full_module_path ))
File "/home/secneo/binaryanalysis-ng-master/src/bangsignatures.py", line 461, in _get_unpackers_recursive
module = importlib.import_module(module_name)
File "/usr/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/secneo/binaryanalysis-ng-master/src/parsers/archivers/cpio/UnpackParser.py", line 5, in
from . import cpio_new_ascii
ImportError: cannot import name 'cpio_new_ascii'

If specifying the unpack directory or temporary directory as a relative path name, the scan takes forever even on a small file.

The GIF .ksy file is not correct for application identifiers.

Upon Running docker image build -t bang . I get the following error:

> docker image build -t bang .
[+] Building 2.0s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                                                                                                                                         0.0s 
 => => transferring dockerfile: 1.24kB                                                                                                                                                                       0.0s 
 => [internal] load .dockerignore                                                                                                                                                                            0.1s 
 => => transferring context: 45B                                                                                                                                                                             0.0s 
 => CANCELED [internal] load metadata for docker.io/library/fedora:33                                                                                                                                        1.9s 
 => ERROR [internal] load metadata for docker.io/library/kaitai:latest                                                                                                                                       1.9s 
 => [auth] library/kaitai:pull token for registry-1.docker.io                                                                                                                                                0.0s 

 > [internal] load metadata for docker.io/library/kaitai:latest:

failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

I have authenticated to Docker Hub, so I'm not sure where the issue is stemming from here. I am running this on a Debian system under WSL2.

HI
I run bang-scanner for an error

root@ubuntu:/home/test/binaryanalysis-ng/src# python3 bang-scanner -c bang.config -f /home/test/Desktop/S29AL016D90-200000H.bin
Traceback (most recent call last):
File "bang-scanner", line 403, in
main(sys.argv)
File "bang-scanner", line 168, in main
startedfile = open(scandirectory / "STARTED", 'wb')
TypeError: invalid file: PosixPath('/root/tmp/bang-scan-r42t3f24/STARTED')
root@ubuntu:/home/test/binaryanalysis-ng/src#

I'm trying a program to explore the possibilities of scanning

[kea@localhost src]$ python3 ./bang-scanner -c ./bang.config -f ./openwrt-realtek-A2004NS-AP-fw_SDK_v2.4_20150724_release_by_Igor_T.bin

and program hung ( not response over 1 hour )

if interested, then this file https://cloud.mail.ru/public/3snL/3gnzzsBGx

(venv) freedom@freedom-virtual-machine:~/project/newbinaryscan/binaryanalysis-ng/src$ sudo /home/freedom/project/newbinaryscan/venv/bin/python bang-scanner -c bang.config -f test/testdata/unpackers/
Traceback (most recent call last):
  File "bang-scanner", line 55, in <module>
    from bangsignatures import maxsignaturesoffset
  File "/home/freedom/project/newbinaryscan/binaryanalysis-ng/src/bangsignatures.py", line 27, in <module>
    import bangandroid
  File "/home/freedom/project/newbinaryscan/binaryanalysis-ng/src/bangandroid.py", line 37, in <module>
    import bangunpack
  File "/home/freedom/project/newbinaryscan/binaryanalysis-ng/src/bangunpack.py", line 58, in <module>
    import snappy
  File "/home/freedom/project/newbinaryscan/venv/lib/python3.6/site-packages/snappy/__init__.py", line 7, in <module>
    from .SnapPy import (AbelianGroup, HolonomyGroup, FundamentalGroup,
  File "cython/core/basic.pyx", line 45, in init SnapPy
  File "/home/freedom/project/newbinaryscan/venv/lib/python3.6/site-packages/snappy/horoviewer.py", line 3, in <module>
    from .CyOpenGL import (HoroballScene, OpenGLOrthoWidget,
  File "opengl/CyOpenGL.pyx", line 38, in init CyOpenGL
AttributeError: type object 'CyOpenGL.vector3' has no attribute '__reduce_cython__'
(venv) freedom@freedom-virtual-machine:~/project/newbinaryscan/binaryanalysis-ng/src$ 

python3.6.9
ubuntu18.04

We installed BANG successfully and able to unpack binary files (.bin-flashwares).

But we are not able to locate result files for Open Source License scanning.
We see "LicenseIdentifierScanner.py" file in \src but not sure if it is called during unpacking or has to be called separately?

Are you interested perhaps in re-packing capabilities for reengineering?
Think e.g. round-tripping apk-s through apktool, repacking archives/installers after editing the contents, etc.

I'm trying a program to explore the possibilities of scanning

[kea@localhost src]$ python3 ./bang-scanner -c ./bang.config -f ./openwrt-18.06.1-brcm2708-bcm2710-rpi-3-squashfs-factory.img.gz

and get errors

Process Process-4:
Traceback (most recent call last):
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 673, in processfile scanjob.check_for_signatures(unpacker)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 297, in check_for_signatures signature, offset)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/Unpacker.py", line 198, in try_unpack_file_for_signatures return bangsignatures.signaturetofunction[signature](fileresult, scanenvironment, offset, self.dataunpackdirectory)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/bangunpack.py", line 12346, in unpack_compress p = subprocess.Popen(['uncompress'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/lib64/python3.7/subprocess.py", line 775, in init restore_signals, start_new_session)
File "/usr/lib64/python3.7/subprocess.py", line 1522, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'uncompress': 'uncompress'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run()
File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 745, in processfile raise ScanJobError(scanjob, e)
ScanJob.ScanJobError: Exception for scanjob:
file:
openwrt-18.06.1-brcm2708-bcm2710-rpi-3-squashfs-factory.img.gz-gzip-1/openwrt-18.06.1-brcm2708-bcm2710-rpi-3-squashfs-factory.img
labels:

Traceback (most recent call last):
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 673, in processfile scanjob.check_for_signatures(unpacker)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 297, in check_for_signatures signature, offset)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/Unpacker.py", line 198, in try_unpack_file_for_signatures return bangsignatures.signaturetofunction[signature](fileresult, scanenvironment, offset, self.dataunpackdirectory)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/bangunpack.py", line 12346, in unpack_compress p = subprocess.Popen(['uncompress'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
File "/usr/lib64/python3.7/subprocess.py", line 775, in init restore_signals, start_new_session)
File "/usr/lib64/python3.7/subprocess.py", line 1522, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'uncompress': 'uncompress'

if interested, then this file https://cloud.mail.ru/public/4Znn/sNB3zAYja
openwrt-18.06.1-brcm2708-bcm2710-rpi-3-squashfs-factory.img.gz

Steps to reproduce

load a css file with some non printable characters, my example had
['0xe2', '0x9e', '0x99'] in them.

Expected behaviour

file should have label the text, not binary

Observed behaviour

file has both text and binary as labels

Remarks

The extension unpacker adds the labels text and css, the IsTextComputer adds binary.
What is the correct labeling in this case?

squashfs 4.4 treats non-fatal errors the same as fatal errors and always exits with 1. In case a squashfs file system has files that cannot be unpacked, but which are irrelevant, such as device files, pipes, etc. then unsquashfs will exit with 1 which BANG will treat as an error although.

Newer versions of squashfs have a workaround, see plougher/squashfs-tools#94

I'm trying out BANG for the first time. Executing nix-shell analysis.nix results in the following error message:

error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)

I'm using an Ubuntu 22.04.2 LTS.

The archive at https://archive.synology.com/download/DSM/release/6.2.1/23824/DSM_DS112+_23824.pat contains a certificate file that is unpacked to

unpack/DSM_DS112+_23824.pat-tar-1/hda1.tgz-xz-1/unpacked-from-xz-tar-1/usr/lib/python2.7/ensurepip/_bundled/pip-8.1.1-py2.py3-none-any.whl-zip-1/pip/_vendor/requests/cacert.pem

unpackCertificate rejects this file and says that it is not a certificate.

https://github.com/KOLANICH-specs/kaitai_struct_formats/blob/qt_installer_framework/archive/qt_installer_framework.ksy
https://github.com/prebuilder/QtInstallerFrameworkExtractor

Example: two GIF files concatenated together:

test.gif-0x00000000-gif-1/unpacked.gif
test.gif-0x000351b5-gif-1/unpacked.gif

The counter should have been increased.

Hi, I am trying to get BANG to work, believing that it's a tool that could be very useful, but I have not had a lot of success with it yet.

1. The first invocation on a large (saved) docker image led to a lot of output, but then BANG would apparently hang. (I may open a separate issue about this.)
2. I then tried with a much smaller, simpler docker image, and got no errors and no hang. Yet… I wonder what I output I should expect, and where? (There are only lines that look like filenames / image contents.) I could not find any created log file or report, for instance.

Since I used the command listed in the README, I would suggest to put a sentence below that (in the "Invocation" section) what kind of output one should expect.

I'm trying a program to explore the possibilities of scanning

python3 ./bang-scanner -c ./bang.config -f ./gpt99_vfat16_sdb88.bin

and get errors

Process Process-2:
Traceback (most recent call last):
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 673, in processfile scanjob.check_for_signatures(unpacker)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 297, in check_for_signatures signature, offset)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/Unpacker.py", line 198, in try_unpack_file_for_signatures return bangsignatures.signaturetofunction[signature](fileresult, scanenvironment, offset, self.dataunpackdirectory)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/bangfilesystems.py", line 3032, in unpack_fat chainindex = clustervals[chainindex]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib64/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run()
File "/usr/lib64/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 745, in processfile raise ScanJobError(scanjob, e)

ScanJob.ScanJobError: Exception for scanjob:
file:
gpt99_vfat16_sdb88.bin
labels: root
Traceback (most recent call last):
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 673, in processfile scanjob.check_for_signatures(unpacker)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/ScanJob.py", line 297, in check_for_signatures signature, offset)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/Unpacker.py", line 198, in try_unpack_file_for_signatures return bangsignatures.signaturetofunction[signature](fileresult, scanenvironment, offset, self.dataunpackdirectory)
File "/home/kea/Загрузки/_soft/binaryanalysis/binaryanalysis-ng-master/src/bangfilesystems.py", line 3032, in unpack_fat chainindex = clustervals[chainindex]
IndexError: list index out of range

if interested, then this file https://cloud.mail.ru/public/5MZc/4GfiFTVwq

gpt99_vfat16_sdb88.zip

There is a race condition that affects the android_sparse_data unpacker: successful unpacking of this format depends on the presence of other files (such as .transferlist). Since files are yielded by the other unpackers it could happen that the main file has already been yielded and in the scan queue, but that the transferlist file has not been yielded yet or is empty. This does not happen when running with a single thread.

It is a bit problematic to see all the specs in this repo when they reside in different branches. So I propose to organize an own separate repo mirroring kaitai_struct_formats (ksf) structure (so its contents can be copied over it (except the dotted dirs, of course) for the specs not yet merged into ksf.

Probably it may make sense to create an org first and move this repo into it.

Since e2tools 0.1.0 the file mode is by default pretty printed with a 10 character string instead of numeric. There doesn't seem to be a way to get it in numeric.

I am getting this after setting json = yes in my config:

Traceback (most recent call last):
  File "bang-scanner", line 414, in <module>
    main(sys.argv)
  File "bang-scanner", line 374, in main
    JsonReporter(jsonfile).report(scanresult)
  File "/home/hmeine/tmp/tern-test/binaryanalysis-ng/src/JsonReporter.py", line 43, in report
    for a, h in fileresult.get_hashresult().items():
AttributeError: 'dict' object has no attribute 'get_hashresult'

$ python3 bang-scanner -f /tmp/test.wad
Traceback (most recent call last):
File "bang-scanner", line 414, in
main(sys.argv)
File "bang-scanner", line 380, in main
HumanReadableReporter(reportfile).report(scanresult)
File "/home/armijn/tmp/binaryanalysis-ng/src/reporter/humanreadablereport.py", line 101, in report
s += self._fileunpackedfiles(fn)
File "/home/armijn/tmp/binaryanalysis-ng/src/reporter/humanreadablereport.py", line 65, in _fileunpackedfiles
l['offset'], l['type'], " ".join(sorted(l['files']))
TypeError: sequence item 0: expected str instance, PosixPath found

I'm trying a program to explore the possibilities of scanning

[kea@localhost src]$ python3 ./bang-scanner -c ./bang.config -f ./fw-UR-825AC-12.11.15.bin

the program creates a file of infinite size ( I interrupted on a file size "sda" of 100GB )
bang-scan-ejl4sdbz/unpack/fw-UR-825AC-12.11.15.bin-0x0014c822-squashfs-1/dev/sda

if interested, then this file https://cloud.mail.ru/public/481V/5kbYd9dLz

printing a ScanJobError fails:

AttributeError: 'ScanJob' object has no attribute 'filename'

@timhemel what did you exactly have in mind with ScanJobError?