hexhive / t-fuzz

T-Fuzz

T-Fuzz consists of 2 components:

• Fuzzing tool (TFuzz): a fuzzing tool based on program transformation
• Crash Analyzer (CrashAnalyzer): a tool that verifies whether crashes found transformed programs are true bugs in the original program or not (coming soon).

OS support

The current version is tested only on Ubuntu-16.04, while trying to run the code, please use our tested OS.

Prerequisite

T-Fuzz system is built on several opensource tools.

• angr
• shellphish fuzzer
• angr tracer
• radare2 and its python wrapper r2pipe

Installing radare2

$ git clone https://github.com/radare/radare2.git
$ cd radare2
$ ./sys/install.sh

Installing python libraries

installing some dependent libraries

Note: to use apt-get build-dep, you need to uncomment the deb-src lines in your apt source file (/etc/apt/sources.list) and run apt-get update.

$ sudo apt-get install build-essential gcc-multilib libtool automake autoconf bison debootstrap debian-archive-keyring
$ sudo apt-get build-dep qemu-system
$ sudo apt-get install libacl1-dev

installing pip and setting up virtualenv & wrapper

$ sudo apt-get install python-pip python-virtualenv
$ pip install virtualenvwrapper

Add the following lines to your shell rc file (~/.bashrc or ~/.zshrc).

export WORKON_HOME=$HOME/.virtual_envs
source /usr/local/bin/virtualenvwrapper.sh

Creating a python virtual environment

$ mkvirtualenv tfuzz-env

Installing dependent libraries

This command will install all the dependent python libraries for you.

$ workon tfuzz-env
$ pip install -r req.txt

Fuzzing target programs with T-Fuzz

$ ./TFuzz  --program  <path_to_target_program> --work_dir <work_dir> --target_opts <target_opts>

Where

• <path_to_target_program>: the path to the target program to fuzz
• <work_dir>: the directory to save the results
• <target_opts>: the options to pass to the target program, like AFL, use @@ as placeholder for files to mutate.

Examples

1. Fuzzing base64 with T-Fuzz

$ ./TFuzz  --program  target_programs/base64  --work_dir workdir_base64 --target_opts "-d @@"

2. Fuzzing uniq with T-Fuzz

$ ./TFuzz  --program  target_programs/uniq  --work_dir workdir_uniq --target_opts "@@"

3. Fuzzing md5sum with T-Fuzz

$ ./TFuzz  --program  target_programs/md5sum  --work_dir workdir_md5sum --target_opts "-c @@"

4. Fuzzing who with T-Fuzz

$ ./TFuzz  --program  target_programs/who  --work_dir workdir_who --target_opts "@@"

Using CrashAnalyzer to verify crashes

T-Fuzz CrashAnalyzer has been put in a docker image, however, it is still not working in all binaries we tested, we are still investigating it the cause.

Here is how:

Run the following command to run our docker image

$ [sudo] docker pull tfuzz/tfuzz-test
$ [sudo] docker run  --security-opt seccomp:unconfined -it tfuzz/tfuzz-test  /usr/bin/zsh 

In the container:

There are 3 directories:

• release: contains code the built lava binaries
• results: contains some results we found in lava-m dataset
• radare2: it is a program used by T-Fuzz.

Currently, T-Fuzz may not work, because the tracer crashes accidentally. And the CrashAnalyzer can not work on all results. But some cases can be recovered.

For example:

To verify bugs in base64, first goto release and checkout ca_base64:

$ cd release
$ git checkout ca_base64

Then we use a transformed program to recover the crash in the original program:

1. Choose a transformed program and run it on the input found by a fuzzer:

$ cd ~
$./results/ca_base64/554/base64_tfuzz_28/base64_tfuzz_28 -d ./results/ca_base64/554/crashing_inputs_from/results_saved_0_from 
[1]    131 segmentation fault (core dumped)  ./results/ca_base64/554/base64_tfuzz_28/base64_tfuzz_28 -d

2. Recover an input from this transformed program and crashing input

$ ./release/CrashAnalyzer  --tprogram ./results/ca_base64/554/base64_tfuzz_28/base64_tfuzz_28 --target_opts "-d @@" --crash_input ./results/ca_base64/554/crashing_inputs_from/results_saved_0_from --result_dir base64_result --save_to recover
WARNING | 2018-12-04 04:28:22,350 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
Trying /root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from
WARNING | 2018-12-04 04:28:23,228 | angr.project | Address is already hooked, during hook(0x9021cd0, <SimProcedure ReturnUnconstrained>). Re-hooking.
WARNING | 2018-12-04 04:28:23,228 | angr.project | Address is already hooked, during hook(0x90dd000, <SimProcedure ReturnUnconstrained>). Re-hooking.
WARNING | 2018-12-04 04:28:23,229 | angr.simos.linux | Tracer has been heavily tested only for CGC. If you find it buggy for Linux binaries, we are sorry!
Adding <Bool Or(((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_0_0_8 >= 65) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_0_0_8 <= 90)), ((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_0_0_8 >= 97) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_0_0_8 <= 122)), ((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_0_0_8 >= 48) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_0_0_8 <= 57)), (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_0_0_8 == 43), (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_0_0_8 == 47))>
Adding <Bool Or(((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_1_1_8 >= 65) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_1_1_8 <= 90)), ((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_1_1_8 >= 97) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_1_1_8 <= 122)), ((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_1_1_8 >= 48) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_1_1_8 <= 57)), (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_1_1_8 == 43), (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_1_1_8 == 47))>
Adding <Bool Or(((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_2_2_8 >= 65) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_2_2_8 <= 90)), ((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_2_2_8 >= 97) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_2_2_8 <= 122)), ((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_2_2_8 >= 48) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_2_2_8 <= 57)), (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_2_2_8 == 43), (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_2_2_8 == 47))>
Adding <Bool Or(((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_3_3_8 >= 65) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_3_3_8 <= 90)), ((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_3_3_8 >= 97) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_3_3_8 <= 122)), ((file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_3_3_8 >= 48) && (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_3_3_8 <= 57)), (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_3_3_8 == 43), (file_/root/results/ca_base64/554/crashing_inputs_from/results_saved_0_from_9_3_3_8 == 47))>
results saved to /root/base64_result/recover_0

Then /root/base64_result/recover_0 is generated, we can use it to trigger a crash in the original program.

3. verify the input by running the generated input on the original program

$ ./results/base64 -d base64_result/recover_0 
Successfully triggered bug 554, crashing now!
Successfully triggered bug 554, crashing now!
Successfully triggered bug 554, crashing now!
[1]    177 segmentation fault (core dumped)  ./results/base64 -d base64_result/recover_0

t-fuzz's Issues

verison of capstone does not support MIPS instruction

Hi,

I have installed T-Fuzz on Windows Subsystem for Linux (Ubuntu). and I get the following error:

WARNING | 2018-06-10 10:28:51,171 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
Traceback (most recent call last):
  File "./TFuzz", line 64, in <module>
    main()
  File "./TFuzz", line 15, in main
    __angr_Fuzzer._perform_env_checks()
  File "/root/.virtual_envs/tfuzz-env/local/lib/python2.7/site-packages/fuzzer/fuzzer.py", line 572, in _perform_env_checks
    with open("/proc/sys/kernel/core_pattern") as f:
IOError: [Errno 2] No such file or directory: '/proc/sys/kernel/core_pattern'

Any pointers around this?

Fuzzer says no program left

Deprecation warning: Use main_object instead of main_bin
WARNING | 2019-07-18 18:55:31,206 | tfuzz.tfuzz_sys | </home/ubuntu/T-Fuzz/workdir_who/who_tfuzz/who_tfuzz> failed to start, skip, but there still might be some crashes in crashing seeds
WARNING | 2019-07-18 18:55:31,207 | tfuzz.tfuzz_sys | No program left
I saw these messages and there was no progess in fuzzing for hours at all.

Maybe manually installing python package shellphish-afl caused this error? I installed shellphish-afl 1.2.1 with source code, and installed other packages by req.txt

Deprecation warning: Use main_object instead of main_bin

at first, after above line, fuzzer had error because there was no ~/.virtual_envs/tfuzz_env/bin/afl-unix/afl-fuzz
so i manually copied afl-fuzz to ~/.virtual_envs/tfuzz_env/bin/afl-unix/ does it caused problem?

Got error when I tried to fuzz the target_programs just like examples

After finished install all the dependent python library, I tried to fuzz the binary in target_programs just like the examples, and I got this error:

WARNING | 2018-06-07 09:19:28,663 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000013_64_32{UNINITIALIZED}>
Deprecation warning: Use main_object instead of main_bin
WARNING | 2018-06-07 09:19:31,096 | fuzzer.fuzzer | not forced
Traceback (most recent call last):
File "./TFuzz", line 64, in
main()
File "./TFuzz", line 55, in main
tfuzzsys.run()
File "/home/starpt/Documents/T-Fuzz/tfuzz/tfuzz_sys.py", line 160, in run
if not self.__fuzz_one_program():
File "/home/starpt/Documents/T-Fuzz/tfuzz/tfuzz_sys.py", line 139, in __fuzz_one_program
self.__current_fuzzer.start()
File "/home/starpt/Documents/T-Fuzz/tfuzz/tfuzz_fuzzer.py", line 88, in start
self._fuzzer.start()
File "/home/starpt/.virtualenvs/t-fuzz/local/lib/python2.7/site-packages/fuzzer/fuzzer.py", line 232, in start
self._start_afl()
File "/home/starpt/.virtualenvs/t-fuzz/local/lib/python2.7/site-packages/fuzzer/fuzzer.py", line 543, in _start_afl
master = self._start_afl_instance() # the master fuzzer
File "/home/starpt/.virtualenvs/t-fuzz/local/lib/python2.7/site-packages/fuzzer/fuzzer.py", line 535, in _start_afl_instance
return subprocess.Popen(args, stdout=fp, close_fds=True)
File "/usr/lib/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

I guessed it happened because I didn't export the AFL_PATH environment, so I tried to:
export AFL_PATH=~/.virtualenvs/t-fuzz/bin, and I got same error again......

How should I do to fixed this problem? Thank you

T-Fuzz generated too many transformed binaires which take too much disk space

When fuzzing programes like flvmeta, we found that T-Fuzz has generated too many transformed binaries within 24 hours (du -sh shows 9.1G), which take too much disk space and impact I/O performance.

According to your code, whenever the afl gots stuck, NCCDector will be run to generate a bunch of transformed binaries, and then fuzzing them one by one. Can this be improved by set a maxinum number of generated programs? Or immediately start new fuzzing process after one transformed binary generated?

TFuzz take too long time for create_dict when fuzzing big binaries

When fuzzing big binaries, like ffmpeg (about 130MB), it takes a very long time and huge memory to create dict.

I think create dict is not necessary and should be skipped or time-limited for big binaries.

Problems during installation

when i try to install T-Fuzz following instructions, there are some problems with shellphish-afl, python package.
req.txt install shellphish-afl version 1.1, but building wheel for shellphish-afl fails.
So i installed shellphish-afl 1.21 manually, but then errors occur when running TFuzz.
When i tried to install shellphish-afl 1.1 manually, it failed. but i didn't go deeper to it.
is shellphish-afl 1.21 also usable in TFuzz? or should i manually install shellphish-afl 1.1?

No matching distribution found for pkg-resources==0.0.0

I've the error after:

pip install -r req.txt

Log:

...
Collecting pkg-resources==0.0.0 (from -r req.txt (line 53))
  Could not find a version that satisfies the requirement pkg-resources==0.0.0 (from -r req.txt (line 53)) (from versions: )
No matching distribution found for pkg-resources==0.0.0 (from -r req.txt (line 53))

OS: Ubuntu

Some people wrote that it's problem in Ubuntu and fix - should delete line pkg-resources==0.0.0

Dockerfile request

Hi,
The shellphish fuzzer repository is archived and angr tracer is deprecated.
Seems also that you are using angr 7.
Can you provide a Dockerfile for Ubuntu 16.04 that downloads all the correct versions of this old software?
Or at least make a list of compatible versions.
Thank you.

Regards,
Andrea

T-Fuzz was aborted when fuzz the target_programs like examples

T-Fuzz was aborted in a few minutes when I tried to fuzz the target_programs just like examples, and I have no idea about the reason. Please kindly check it.

Here is part of the output:

$ ./TFuzz --program target_programs/uniq --work_dir workdir_uniq --target_opts "@@"

WARNING | 2018-07-30 04:45:40,595 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
/home/public/.virtual_envs/tfuzz-env/T-Fuzz/target_programs/uniq
WARNING | 2018-07-30 04:45:41,120 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
WARNING | 2018-07-30 04:45:50,107 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000000_14_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:45:50,214 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000004_20_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:45:50,242 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c000008_26_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:45:50,295 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c00000d_35_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:45:50,362 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c000011_50_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:45:52,092 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c000016_60_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:45:52,199 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c00001b_69_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:46:00,625 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c000001_17_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:46:00,679 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c000005_23_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:46:00,723 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c000009_29_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:46:00,844 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c00000d_44_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:46:00,871 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c000011_50_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:46:02,015 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c000016_60_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:46:02,122 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained
; skipping. <BV32 global_c00001b_69_32{UNINITIALIZED}>
WARNING | 2018-07-30 04:46:03,518 | fuzzer.fuzzer | not forced
WARNING | 2018-07-30 04:47:05,627 | tfuzz.qemu_runner | One trace is found to be malformated,
it is possible that the log file size exceeds the 1G limit,
meaning that there might be infinite loops in the target program
WARNING | 2018-07-30 04:47:05,684 | tfuzz.qemu_runner | One trace is found to be malformated,
it is possible that the log file size exceeds the 1G limit,
meaning that there might be infinite loops in the target program
WARNING | 2018-07-30 04:47:05,738 | tfuzz.qemu_runner | One trace is found to be malformated,
it is possible that the log file size exceeds the 1G limit,
meaning that there might be infinite loops in the target program
WARNING | 2018-07-30 04:47:05,796 | tfuzz.qemu_runner | One trace is found to be malformated,
it is possible that the log file size exceeds the 1G limit,
meaning that there might be infinite loops in the target program
WARNING | 2018-07-30 04:47:05,853 | tfuzz.qemu_runner | One trace is found to be malformated,
it is possible that the log file size exceeds the 1G limit,
meaning that there might be infinite loops in the target program
...
WARNING | 2018-07-30 04:47:08,787 | tfuzz.qemu_runner | One trace is found to be malformated,
it is possible that the log file size exceeds the 1G limit,
meaning that there might be infinite loops in the target program
WARNING | 2018-07-30 04:47:08,842 | tfuzz.qemu_runner | One trace is found to be malformated,
it is possible that the log file size exceeds the 1G limit,
meaning that there might be infinite loops in the target program
WARNING | 2018-07-30 04:47:08,899 | tfuzz.qemu_runner | One trace is found to be malformated,
it is possible that the log file size exceeds the 1G limit,
meaning that there might be infinite loops in the target program
WARNING | 2018-07-30 04:47:08,905 | tfuzz.tfuzz_sys | No program left

Then the fuzzer was aborted.

ModuleNotFoundError: No module named 'ah' , dpkt

The dpkt package seems to have an issue.
running

pip install -r req.txt

outputs

Collecting dpkt-fix==1.7 (from -r req.txt (line 25))
  Downloading https://files.pythonhosted.org/packages/25/3c/bc19ab90abd0e47f848d04dca71f8795dd857b369fb30df6d8c037afd0f9/dpkt-fix-1.7.tar.gz (59kB)
    100% |████████████████████████████████| 61kB 1.3MB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/tmp/pip-install-6eq2sr65/dpkt-fix/setup.py", line 4, in <module>
        import dpkt
      File "/private/tmp/pip-install-6eq2sr65/dpkt-fix/dpkt/__init__.py", line 13, in <module>
        import ah
    ModuleNotFoundError: No module named 'ah'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/tmp/pip-install-6eq2sr65/dpkt-fix/

I removed dpkt from the req.txt file and tried install manually but still get the same issue.
I run MacOS. Anyone know how to solve this?

T-Fuzz used wrong code to filter crashing seed?

From our experiment to fuzz wav2swf using T-Fuzz, we found an interesting result: The coverage result from the afl queue folder is smaller than provided seed.

So, we investigate this finding, here may be the cause:

T-Fuzz/tfuzz/executor.py

Lines 39 to 48 in 7d150e4

	def _run(self):
	try:
	if self.record_stdout:
	self.stdout = subprocess32.check_output(self.args, timeout=self.timeout)
	else:
	subprocess32.check_output(self.args, timeout=self.timeout)
	except subprocess32.TimeoutExpired:
	self.tmout = True
	except subprocess32.CalledProcessError:
	self.crash = True

T-Fuzz/tfuzz/tfuzz_fuzzer.py

Lines 40 to 50 in 7d150e4

	for sf in seed_files:
	logger.debug("Trying running %s", sf)
	ret = self.__get_exec_result_on_input(sf)
	if ret == 2:
	# the seed results in crash
	self.crashing_inputs.append(sf)
	elif ret == 1:
	# the seed results in timeout
	self.tmout_inputs.append(sf)
	else:
	seeds.append(file(sf).read())

you can see that whenever a CalledProcessError is raised, T-Fuzz will ignore this seed, copy to crashing_seed. And if the whole seed folder all leads to crash, then T-Fuzz will generate a random seed for fuzzing.

However, CalledProcessError will also be raised if the exit code is not 0:

>>> subprocess32.check_output(["/d/p/justafl/4.wav2swf", "-o","/dev/null","crash_seed_12"], timeout=1)
Unsupported bitspersample value: 24
Error: no mp3 soundstream support compiled in.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/subprocess32.py", line 638, in check_output
    raise CalledProcessError(retcode, process.args, output=output)
subprocess32.CalledProcessError: Command '['/d/p/justafl/4.wav2swf', '-o', '/dev/null', 'crash_seed_12']' returned non-zero exit status 1

I think exit status 1 does not mean a crash, only those be killed by a signal should be.

An error in create_dict in utils.py

Hi, all!
I built T-Fuzz in Ubuntu 16.04.5 LTS, when I ran the following example, T-Fuzz got stuck.
$./TFuzz --program target_programs/base64 --work_dir workdir_base64 --target_opts "-d @@"

Here is the output:

WARNING | 2019-01-07 13:21:00,518 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
/root/tfuzz/T-Fuzz/target_programs/base64
WARNING | 2019-01-07 13:21:01,424 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
WARNING | 2019-01-07 13:21:19,691 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000000_23_32{UNINITIALIZED}>
WARNING | 2019-01-07 13:21:19,730 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000004_29_32{UNINITIALIZED}>
WARNING | 2019-01-07 13:21:19,772 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000008_35_32{UNINITIALIZED}>
WARNING | 2019-01-07 13:21:22,801 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c00000e_54_32{UNINITIALIZED}>
WARNING | 2019-01-07 13:21:22,866 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000013_64_32{UNINITIALIZED}>
WARNING | 2019-01-07 13:21:38,761 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000001_24_32{UNINITIALIZED}>
WARNING | 2019-01-07 13:21:38,801 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000005_30_32{UNINITIALIZED}>
WARNING | 2019-01-07 13:21:38,876 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000009_45_32{UNINITIALIZED}>
WARNING | 2019-01-07 13:21:40,609 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c00000e_55_32{UNINITIALIZED}>
WARNING | 2019-01-07 13:21:40,752 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000013_64_32{UNINITIALIZED}>
Deprecation warning: Use main_object instead of main_bin
WARNING | 2019-01-07 13:21:43,102 | fuzzer.fuzzer | not forced
WARNING | 2019-01-07 13:21:45,116 | tfuzz.tfuzz_sys | </root/tfuzz/T-Fuzz/workdir_base64/base64_tfuzz/base64_tfuzz> failed to start,                 skip, but there still might be some                 crashes in crashing seeds
WARNING | 2019-01-07 13:21:45,118 | tfuzz.tfuzz_sys | No program left

It seems that the fuzzer failed to start:
$cat workdir_base64/fuzzing_base64_tfuzz/base64_tfuzz/stat

status:failed to start

cat workdir_base64/fuzzing_base64_tfuzz/base64_tfuzz/fuzzer-master.log

afl-fuzz 2.52b by <[email protected]>
[+] Looks like we're not running on a tty, so I'll be a bit less verbose.
[+] You have 80 CPU cores and 16 runnable tasks (utilization: 20%).
[+] Try parallel jobs - see /usr/local/share/doc/afl/parallel_fuzzing.txt.
[*] Checking CPU core loadout...
[+] Found a free CPU core, binding to #0.
[*] Checking core_pattern...
[*] Setting up output directories...
[*] Scanning '/root/tfuzz/T-Fuzz/workdir_base64/fuzzing_base64_tfuzz/base64_tfuzz/input'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Loading extra dictionary from '/root/tfuzz/T-Fuzz/workdir_base64/base64.dict' (level 0)...

The issue occurs in the dictionary:

$cat workdir_base64/base64.dict | head -10

Deprecation warning: Use eval(expr, cast_to=str) instead of any_str
string_0="\x00"
string_1="\x5f\x5fAFL\x5fDEFER\x5fFORKSRV\x00"
string_2="help"
string_3="Written by \x25s\x2c \x25s\x2c \x25s\x2c\x0a\x25s\x2c \x25s\x2c \x25s\x2c \x25s\x2c\x0a\x25s\x2c \x25s\x2c and others.\x0a\x00"
string_4="Written by \x25s\x2c \x25s\x2c \x25s\x2c\x0a\x25s\x2c \x25s\x2c \x25s\x2c \x25s\x2c\x0aand \x25s.\x0a\x00"
string_5="\x28C\x29\x00"
string_6="\x2550s"
string_7="\x5f\x5fAFL\x5fSHM\x5fID\x00"
string_8="Simon"

This first line of the dict is a warning, which leads to AFL failure.

I think this issue is caused by the following line, which attempts write all of stdout in the file dict:

T-Fuzz/tfuzz/utils.py

Line 13 in 7d150e4

p = subprocess.Popen(args, stdout=df)

This issue can be solved by checking like this:
fouzhe@0aa9684

tfuzz did not consider cannot-renamed programs

There is a type of programs whose behaviour depend to which name they are. busybox is a good example, only one file is real ELF file and others are just symlink to it.

In this scenario, it's important for fuzzer to keep the original ELF filename, otherwise desired program behaviour will not be triggered and fuzzed.

I want to fuzz infotocap, which is a symlink to tic

But tfuzz actually runs: (which means it has been renamed to infotocap_tfuzz)

 /usr/bin/afl-unix/afl-fuzz -i /d/output/reruninfotocap_tfuzz6_1/fuzzing_infotocap_tfuzz/infotocap_tfuzz/input -o /d/output/reruninfotocap_tfuzz6_1/fuzzing_infotocap_tfuzz/infotocap_tfuzz/sync -m 8G -M fuzzer-master -x /d/output/reruninfotocap_tfuzz6_1/infotocap.dict -- /d/output/reruninfotocap_tfuzz6_1/infotocap_tfuzz/infotocap_tfuzz -o /dev/null @@

Could you consider this type of program? Thanks~

afl master only run first cycle?

master.log shows:

�[1;90m    Test case count : �[0m2 favored, 0 variable, 100 total
�[1;90m       Bitmap range : �[0m1011 to 1013 bits (average: 1011.50 bits)
�[1;90m        Exec timing : �[0m138k to 310k us (average: 237k us)
�[0m
�[1;94m[*] �[0mNo -t option specified, so I'll use exec timeout of 480 ms.�[0m
�[1;92m[+] �[0mAll set and ready to roll!�[0m
�[1;94m[*] �[0mEntering queue cycle 1.�[0m
�[1;94m[*] �[0mFuzzing test case #0 (100 total, 0 uniq crashes found)...�[0m

In normal afl runs, there should be more logs?

also, the runtime CPU is so low...

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
2744d8ed2340        tfuzz3_30           3.25%               349.4MiB / 2GiB     17.06%              3.95kB / 0B         0B / 18.6MB         7

For running 24 hours, the crash folder in sync/fuzzer-master/crashes is also empty for mp3gain 1.5.2-r2. And there is only one output file in sync/fuzzer-master/crashes for tiffsplit (libtiff-3.9.7)

Please push the code

Hey, your paper on T-Fuzz sounds great. Please push your code so we can test and contribute !

Initial seeds

How to set some initial seeds before executing T-Fuzz?

AFL Error: Suboptimal CPU scaling governor

I tried T-fuzz, but I got the following error. I think this may be related to the change of angr. Because angr is changed to support python3 and its 'tracer' is merged into angr. So maybe you can push the whole codes including angr and shellphish fuzzer. my system is ubuntu 16.04.
./TFuzz --program target_programs/base64 --work_dir workdir_base64 --target_opts "-d @@"
`WARNING | 2018-10-28 23:45:10,451 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
ERROR | 2018-10-28 23:45:10,721 | fuzzer.fuzzer | AFL Error: Pipe at the beginning of core_pattern
execute 'echo core | sudo tee /proc/sys/kernel/core_pattern'
AFL Error: Suboptimal CPU scaling governor
execute 'cd /sys/devices/system/cpu; echo performance | sudo tee cpu*/cpufreq/scaling_governor'
AFL Warning: We probably want the fork() children to run first
execute 'echo 1 | sudo tee /proc/sys/kernel/sched_child_runs_first'

Traceback (most recent call last):
File "./TFuzz", line 64, in
main()
File "./TFuzz", line 15, in main
__angr_Fuzzer._perform_env_checks()
File "/home/rosen/anaconda3/envs/T-Fuzz/lib/python2.7/site-packages/fuzzer/fuzzer.py", line 599, in _perform_env_checks
raise InstallError(err)
fuzzer.fuzzer.InstallError: AFL Error: Pipe at the beginning of core_pattern
execute 'echo core | sudo tee /proc/sys/kernel/core_pattern'
AFL Error: Suboptimal CPU scaling governor
execute 'cd /sys/devices/system/cpu; echo performance | sudo tee cpu*/cpufreq/scaling_governor'
AFL Warning: We probably want the fork() children to run first
execute 'echo 1 | sudo tee /proc/sys/kernel/sched_child_runs_first'`

Thanks for your time.

Example issue

Hi, all.
I got error when executing this command in example:

./TFuzz  --program  target_programs/base64  --work_dir workdir_base64 --target_opts "-d @@"

The following is the error information:

WARNING | 2019-01-07 05:00:56,313 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
/root/T-Fuzz/target_programs/base64
WARNING | 2019-01-07 05:00:57,224 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
WARNING | 2019-01-07 05:01:15,075 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000000_14_32{UNINITIALIZED}>
WARNING | 2019-01-07 05:01:15,134 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000004_29_32{UNINITIALIZED}>
WARNING | 2019-01-07 05:01:15,174 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000008_35_32{UNINITIALIZED}>
WARNING | 2019-01-07 05:01:18,414 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c00000e_55_32{UNINITIALIZED}>
WARNING | 2019-01-07 05:01:18,576 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000013_64_32{UNINITIALIZED}>
WARNING | 2019-01-07 05:01:34,402 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000001_24_32{UNINITIALIZED}>
WARNING | 2019-01-07 05:01:34,466 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000005_39_32{UNINITIALIZED}>
WARNING | 2019-01-07 05:01:34,499 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000009_45_32{UNINITIALIZED}>
WARNING | 2019-01-07 05:01:36,095 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c00000e_54_32{UNINITIALIZED}>
WARNING | 2019-01-07 05:01:36,154 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000013_64_32{UNINITIALIZED}>
Deprecation warning: Use main_object instead of main_bin
WARNING | 2019-01-07 05:01:38,438 | fuzzer.fuzzer | not forced
Traceback (most recent call last):
  File "./TFuzz", line 64, in <module>
    main()
  File "./TFuzz", line 55, in main
    tfuzzsys.run()
  File "/root/T-Fuzz/tfuzz/tfuzz_sys.py", line 160, in run
    if not self.__fuzz_one_program():
  File "/root/T-Fuzz/tfuzz/tfuzz_sys.py", line 139, in __fuzz_one_program
    self.__current_fuzzer.start()
  File "/root/T-Fuzz/tfuzz/tfuzz_fuzzer.py", line 88, in start
    self._fuzzer.start()
  File "/root/.virtual_envs/tfuzz-env/local/lib/python2.7/site-packages/fuzzer/fuzzer.py", line 232, in start
    self._start_afl()
  File "/root/.virtual_envs/tfuzz-env/local/lib/python2.7/site-packages/fuzzer/fuzzer.py", line 542, in _start_afl
    master = self._start_afl_instance() # the master fuzzer
  File "/root/.virtual_envs/tfuzz-env/local/lib/python2.7/site-packages/fuzzer/fuzzer.py", line 534, in _start_afl_instance
    return subprocess.Popen(args, stdout=fp, close_fds=True)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

T-Fuzz stuck there.

I ran the T-fuzz on docker, I don't know if that matter. The output is:

root@5d42ab6b99cc:/apps/T-Fuzz# ./TFuzz --program target_programs/base64 --work_dir workdir_base64 --target_opts "-d @@"
WARNING | 2018-11-29 12:03:16,387 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
/apps/T-Fuzz/target_programs/base64
WARNING | 2018-11-29 12:03:17,409 | angr.analyses.disassembly_utils | Your verison of capstone does not support MIPS instruction groups.
WARNING | 2018-11-29 12:03:38,370 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000000_14_32{UNINITIALIZED}>
WARNING | 2018-11-29 12:03:38,439 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000004_29_32{UNINITIALIZED}>
WARNING | 2018-11-29 12:03:38,485 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000008_35_32{UNINITIALIZED}>
WARNING | 2018-11-29 12:03:41,969 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c00000e_55_32{UNINITIALIZED}>
WARNING | 2018-11-29 12:03:42,151 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000013_64_32{UNINITIALIZED}>
WARNING | 2018-11-29 12:04:00,529 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000000_14_32{UNINITIALIZED}>
WARNING | 2018-11-29 12:04:00,617 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000004_29_32{UNINITIALIZED}>
WARNING | 2018-11-29 12:04:00,662 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000008_35_32{UNINITIALIZED}>
WARNING | 2018-11-29 12:04:02,904 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c00000e_55_32{UNINITIALIZED}>
WARNING | 2018-11-29 12:04:03,086 | angr.engines.successors | Exit state has over 257 possible solutions. Likely unconstrained; skipping. <BV32 global_c000013_64_32{UNINITIALIZED}>
Deprecation warning: Use main_object instead of main_bin
WARNING | 2018-11-29 12:04:06,019 | fuzzer.fuzzer | not forced

Then, T-fuzz don't go further, it stucks here. The system on docker is ubuntu 16.04.