Coder Social home page Coder Social logo

softsec-kaist / eclipser Goto Github PK

View Code? Open in Web Editor NEW
147.0 8.0 16.0 240 KB

Grey-box Concolic Testing on Binary Code (ICSE '19)

License: MIT License

Shell 9.43% C 23.14% Makefile 0.64% F# 66.09% Dockerfile 0.70%
testcase-generator fuzzer concolic-testing fsharp fuzzing

eclipser's Introduction

Eclipser

Eclipser is a binary-based fuzz testing tool that improves upon classic coverage-based fuzzing by leveraging a novel technique called grey-box concolic testing. The details of the technique can be found in our paper "Grey-box Concolic Testing on Binary Code", which is published in ICSE 2019.

Installation

Eclipser currently supports Linux ELF binaries, and has been tested on Debian and Ubuntu. Eclipser is written in F# and runs on .NET Core. Also, Eclipser performs program instrumentation based on QEMU code.

  1. Install dependencies
$ sudo vim /etc/apt/sources.list # Uncomment the lines starting with 'deb-src'.
$ sudo apt-get update
$ sudo apt-get build-dep qemu
$ sudo apt-get install libtool libtool-bin wget automake autoconf bison gdb python
  1. Install .NET Core

Installation differs for each Linux distribution, so please refer to this link. Choose your Linux distribution and version from the tab and follow the instructions.

  1. Clone and build Eclipser
$ git clone https://github.com/SoftSec-KAIST/Eclipser
$ cd Eclipser
$ make

Usage

  • Running with AFL

Starting from v2.0, Eclipser only performs grey-box concolic testing for test case generation and relies on AFL to perform random-based fuzzing (for the context of this decision, refer to Eclipser v2.0 section below). Therefore, you should first launch AFL instances in parallel mode. Although it is possible to run Eclipser alone, it is intended only for simple testing and not for realistic fuzzing.

$ AFL_DIR/afl-fuzz -i <seed dir> -o <sync dir> -M <ID 1> \
  -f <input file to fuzz> -Q -- <target program cmdline>
$ AFL_DIR/afl-fuzz -i <seed dir> -o <sync dir> -S <ID 2> \
  -f <input file to fuzz>  -Q -- <target program cmdline>
$ dotnet ECLIPSER_DIR/build/Eclipser.dll \
  -t <timeout (sec)> -i <seed dir (optional)> -s <sync dir> -o <output dir> \
  -p <target program> --arg <target program cmdline> -f <input file to fuzz>

We note that the output directory for Eclipser should be placed under the synchronization directory (e.g. -s ../syncdir -o ../syncdir/eclipser-output). AFL will automatically create an output directory under the synchronization directory, using its specified ID. This way, Eclipser and AFL will share test cases with each other. To obtain the final result of the fuzzing, retrieve all the test cases under <sync dir>/*/queue/ and <sync dir>/*/crashes/.

Similarly to AFL, Eclipser will fuzz the file input specified by -f option, and fuzz the standard input when -f option is not provided. However, Eclipser does not support @@ syntax used by AFL.

  • Examples

You can find simple example programs and their fuzzing scripts in examples directory. An example script to run Eclipser with AFL can be found here. Note that we create separate working directories for each AFL instance and Eclipser in this script. This is to prevent the instances from using the same input file path for fuzzing.

  • Other options for fuzzing

You can get the full list of Eclipser's options and their descriptions by running the following command.

$ dotnet build/Eclipser.dll --help

Eclipser v2.0

Originally, Eclipser had its own simplified random-based fuzzing module, instead of relying on AFL. This was to support fuzzing multiple input sources (e.g. command-line arguments, standard input, and file input) within a single fuzzer run. We needed this feature for the comparison against KLEE on Coreutils benchmark, which was one of the main experimental targets in our paper.

However, as Eclipser is more often compared with other fuzzing tools, we abandon this feature and focus on fuzzing a single input source, as most fuzzers do. We also largely updated the command line interface of Eclipser accordingly. We note that you can still checkout v1.0 code from our repository to reproduce the Coreutils experiment result.

By focusing on fuzzing a single input source, we can now use AFL to perform random-based fuzzing. For this, from v2.0 Eclipser runs in parallel with AFL, as described above. This way, we can benefit from various features offered by AFL, such as source-based instrumentation, persistent mode, and deterministic mode. Still, the core architecture of Eclipser remains the same: it complements random-based fuzzing with our grey-box concolic testing technique.

Docker

To run the latest version of Eclipser, please use the Dockerfile provided in this repository. If you are interested in reproducing the results of ICSE 2019 paper, you can refer to Eclipser-Artifact repository, which uses Eclipser v0.1.

Supported Architectures

Eclipser currently supports x86 and x64 architecture binaries. We internally have a branch that supports ARM architecture, but do not plan to open source it. In default, Eclipser assumes that the target program is an x64 binary. If you want to fuzz an x86 binary, you should provide --architecture x86 option to Eclipser.

Citation

Please consider citing our paper (ICSE 2019):

@INPROCEEDINGS{choi:icse:2019,
  author = {Jaeseung Choi and Joonun Jang and Choongwoo Han and Sang Kil Cha},
  title = {Grey-box Concolic Testing on Binary Code},
  booktitle = {Proceedings of the International Conference on Software Engineering},
  pages = {736--747},
  year = 2019
}

eclipser's People

Contributors

jchoi2022 avatar neuromancer avatar sangkilc avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

eclipser's Issues

Allow the user to select the arquitecture to compile

Eclipser takes quite a long time to compile, since it recompiles QEMU for i386 and x64, however usually the user needs only one them. Is it possible to add a flag in the Makefile to compile for only one of these?

Fork server died: Success

I run Eclipser in a docker container and running a target results in a crash of Eclipser:

# dotnet /Eclipser/build/Eclipser.dll -t 5 -s eclipser -o out -p fuzz-target --exectimeout 5000 --arg foo -v 2
[00:00:00:00] [*] Fuzz target : /targets/test/fuzz-target
Fork server died: Success

Unhandled Exception: System.Exception: Failed to initialize fork server for coverage tracer
   at Eclipser.Executor.initializeForkServer(FuzzOption opt) in /Eclipser/src/Core/Executor.fs:line 75
   at Eclipser.Executor.initialize(FuzzOption opt) in /Eclipser/src/Core/Executor.fs:line 92
   at Eclipser.Fuzz.main(String[] args) in /Eclipser/src/Fuzz/Fuzz.fs:line 137
Aborted (core dumped)

the target runs without problem when called native or when fuzzed via afl-fuzz -Q.
what could the issue be?

Timeout of -1 does not work in old Eclipser

The argument is passed on to 'System.TimeSpan' and results in:

Parameter name: timeout
   at System.Threading.Thread.ToTimeoutMilliseconds(TimeSpan timeout)
   at [email protected](Unit unitVar) in /root/Eclipser/src/Fuzz/Fuzz.fs:line 122
   at Microsoft.FSharp.Control.AsyncPrimitives.CallThenInvoke[T,TResult](AsyncActivation`1 ctxt, TResult result1, FSharpFunc`2 part2)
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction)
--- End of stack trace from previous location where exception was thrown ---
   at [email protected](ExceptionDispatchInfo edi)
   at Microsoft.FSharp.Control.Trampoline.Execute(FSharpFunc`2 firstAction)
   at <StartupCode$FSharp-Core>[email protected](Object o)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
Aborted

#20 fixes it the same way as it is fixed in master (just not starting a timeout when using -1)

Fuzzing rocksdb doesn't work

Another DeepState example: https://github.com/agroce/testleveldb

  1. Install DeepState, leveldb, and rocksdb (no need to follow the instructions to add sanitizers or anything, just compile release versions)
  2. Change the makefile to compile with a recent clang
  3. Try using Eclipser via deepstate-eclipser

Eclipser claims to find a crashing input for the DiffTestDBs executable immediately, but it seems to be just a set of zero bytes that don't actually cause a crash. Incompatibility with something one of the DBs is doing and QEMU?

Fuzzing just leveldb works fine, so it's something RocksDB does. Not high priority, just FYI of an important fuzzing target that seems to break Eclipser.

Using DeepState makes graybox concolic much less effective.

The ICSE paper's example works great as a simple C program, but when I try to fuzz the same logic using DeepState (https://github.com/trailofbits/deepstate), it doesn't work. Graybox concolic testing of DeepState examples doesn't seem to work well, and produces surprisingly few interesting paths.

https://github.com/agroce/tryeclipser has some versions of the code. vulnmix avoids DeepState's more complex parsing of an input file, or need to set CLI arguments, and just reads eclipser.input but Eclipser still doesn't find the vulnerability with that approach.

Any ideas what's going on? Obviously with DeepState the time from program start until interesting things start happening based on input is much, much, longer. Be nice to work out what's going on, and if it can be fixed -- we'd love to have Eclipser as a DeepState backend, the approach is very nice, and results are impressive.

root@a621890e86cf:~/tryeclipser# dotnet ~/Eclipser/build/Eclipser.dll fuzz -p ./vulnmix --timelimit 60 --src file --fixfilepath eclipser.input
Fuzz target: /root/tryeclipser/vulnmix
Time: 60 sec
[00:00:00:00] Forkserver for coverage tracer : 95622
[00:00:00:00] Forkserver for branch tracer : 95624
[00:00:00:00] Fuzzing starts
[00:00:00:00] [*] Found by grey-box concolic (655 new nodes) : ("AAAAAAAA") File[0]=( e1* ...7bytes... (0) (Right))
[00:00:00:07] # of seeds before minimization : 2
[00:00:00:07] [*] (Redundant seed) : ("AAAAAAAA") File[0]=( ...8bytes... (0) (Right))
[00:00:00:07] Total 1 redundant seeds found
^C
root@a621890e86cf:~/tryeclipser# dotnet ~/Eclipser/build/Eclipser.dll fuzz -p ./vulnorig --timelimit 60 --src file --fixfilepath eclipser.input --initarg "eclipser.input"
Fuzz target: /root/tryeclipser/vulnorig
Time: 60 sec
[00:00:00:00] Forkserver for coverage tracer : 10992
[00:00:00:00] Forkserver for branch tracer : 10994
[00:00:00:00] Fuzzing starts
[00:00:00:00] [*] Found by grey-box concolic (225 new nodes) : ("eclipser.input") File[0]=( 34! 3d! ...6bytes... (0) (Right))
[00:00:00:00] Found crash seed : ("eclipser.input") File[0]=( 34! 3d! 00 00 42! 61! 64! 21! (4) (Right))
[00:00:00:00] [*] Found by grey-box concolic (598 new nodes) : ("eclipser.input") File[0]=( 34! 3d! 00 00 42! 61! 64! 21! (4) (Right))
[00:00:00:03] # of seeds before minimization : 2
[00:00:00:03] Total 0 redundant seeds found

Sampling input bytes instead of conditional statements in SELECT?

According to the paper:

SELECT randomly selects N conditional statements from the given sequence of conditional statements.

However, instead of selecting N randomly conditional statements, I'm wondering if you have tried to randomly selecting inputs bytes to generate conditional statements. In that case, the results will be (most likely) less or simpler conditions to solve but without the risk of finding unfeasible solutions. You could even select input bytes using different sampling distributions (uniform, or by ranges)

Any chance you can fix qemu to work on ubuntu 20.04

Short version: ubuntu 20.04 uses a later glibc, which leads to a failure to build qemu according to the recipe in the v1.1 ("old Eclipser") release. It looks like QEMU has fixed this, but presumably you are tagged to version 2.10.0 for good reasons. Any chance you can incorporate the fix here https://patchwork.openembedded.org/patch/165581/ to get Eclipser 1.1 runnable on 20.04?

We're trying to make a DeepState (https://github.com/trailofbits/deepstate) release that bases off 20.04 instead of 18.04, and we'd like to keep the "old" Eclipser in the fuzzer stable (and add Eclipser 2.0 when we get a chance; the switch to relying on AFL parallel mode makes this non-trivial). But when we try to build from docker, we run into:

/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64/linux-user/syscall.c:235:16: error: static declaration of 'gettid' follows non-static declaration
  235 | _syscall0(int, gettid)
      |                ^~~~~~
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64/linux-user/syscall.c:164:13: note: in definition of macro '_syscall0'
  164 | static type name (void)   \
      |             ^~~~
In file included from /usr/include/unistd.h:1170,
                 from /home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64/linux-user/syscall.c:27:
/usr/include/x86_64-linux-gnu/bits/unistd_ext.h:34:16: note: previous declaration of 'gettid' was here
   34 | extern __pid_t gettid (void) __THROW;
      |                ^~~~~~
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:235:16: error: static declaration of 'gettid' follows non-static declaration
  235 | _syscall0(int, gettid)
      |                ^~~~~~
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:164:13: note: in definition of macro '_syscall0'
  164 | static type name (void)   \
      |             ^~~~
In file included from /usr/include/unistd.h:1170,
                 from /home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:27:
/usr/include/x86_64-linux-gnu/bits/unistd_ext.h:34:16: note: previous declaration of 'gettid' was here
   34 | extern __pid_t gettid (void) __THROW;
      |                ^~~~~~
make[2]: *** [/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64/rules.mak:57: linux-user/syscall.o] Error 1
make[1]: *** [Makefile:173: subdir-x86_64-linux-user] Error 2
make[1]: Leaving directory '/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x64'
make: *** [Makefile:41: /home/user/Eclipser/Instrumentor/qemu/.compiled_x64] Error 1
make: *** Waiting for unfinished jobs....
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c: In function 'do_syscall':
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:5962:29: warning: implicit declaration of function 'stime'; did you mean 'utime'? [-Wimplicit-function-declaration]
 5962 |             ret = get_errno(stime(&host_time));
      |                             ^~~~~
      |                             utime
/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/linux-user/syscall.c:5962:29: warning: nested extern declaration of 'stime' [-Wnested-externs]
make[2]: *** [/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86/rules.mak:57: linux-user/syscall.o] Error 1
make[1]: *** [Makefile:173: subdir-i386-linux-user] Error 2
make[1]: Leaving directory '/home/user/Eclipser/Instrumentor/qemu/qemu-2.3.0-pathcov-x86'
make: *** [Makefile:37: /home/user/Eclipser/Instrumentor/qemu/.compiled_x86] Error 1

There may be other issues preventing Eclipser 1.1 from running on 20.04 but this one seems fixable, if you have time.

Understand the output directory

Hi,
I'm trying to fuzz tcpdump with Eclipse and I don't understand the purpose of the testcase directory in the output directory.
After 15 min of fuzzing there is only a testcase tc-0 inside the directory.
The starting input is uninformed, 72 0 (ascii).

Eclipse doesn't log inputs that trigger new coverage?

Confirmation on sharing state between multiple instances of Eclipser

I'd like to run multiple instances of Eclipser, is it just as performant to export the entire output directory? Or can I prioritize it so that /.coverage and /.internal (which I think contains the seed queues?) are on a fast-backing, and /crash, and /testcase are on slower disk-type backing?

Please provide a Dockerfile for Eclipser 2.0

So that Eclipser can be used easily on current Linux plus not needing to install all the dependencies required - could you please supply a Dockerfile for Eclipser 2.0?
Based on the fuzzbench builder.Dockerfile + runner.Dockerfile this should not be too much work?
(I would do it myself, but too much on my plate for afl++ at the moment ...)

Changelog from Eclipser 1.0?

I'm wondering if you can give some details regarding the recent changes included in Eclipser 1.0 release since the commit is quite large.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.