hexhive / magma

A ground-truth fuzzing benchmark suite based on real programs with real bugs.

Magma: A Ground-Truth Fuzzing Benchmark

The documentation has been moved to the Magma homepage.

magma's Issues

No corpus in libsndfile

Hi there,

I found that there's no corpus in libsndfile. Is this on purpose? I'm not able to run afl on libsndfile without corpus, however.

Thanks.

Is magma support QEMU mode ?

hi, guys! What should I do if test bianry-based fuzzer in magma ?

Poc extract only exp2json

Hi,
I'm trying to evaluate AFL++ without canaries (CANARY_MODE=2) and with POC_EXTRACT=1 to dedup the crashes.
There is a way to know which bug was triggered?

Looking at the monitor folders, they are empty, but the crashes/ folder of the fuzzer are not ofc and the poc/ folder in the workdir is filled with deduplicated testcases like "aflplusplus_openssl_asn1_NEW.uv1" (what is the meaning of the extension btw?).

Now, if I want to generate an experiment report, which is the way?
If it is not possible, is the number of testcases in poc/ the number of found bugs?

Thanks for your time,
Andrea

Who came up with the fuzzer setup?

Why these settings for afl++?
It is not optimal.
I would have expected that we receive a message asking for recommendations instead of a not well hand picked setup.
That is not very professional or academic.
I will create a PR over the days to fix this.
You should ask the other fuzzer maintainers what they recommend too.

Exclude seeds that trigger a bug without causing a crash

As ideal sanitization is disabled by default, some seeds that immediately trigger a bug (e.g., the PHP Exif regression tests) are not pruned at https://github.com/HexHive/magma/blob/v1.1/magma/run.sh#L27

The run script should check if ideal sanitization is disabled, and if so, check the monitor for triggered crashes during pruning, and prune those seeds.

interpreting bug survival graphs

This is a graph from the libpng report. I'm not exactly sure what information is depicted. Was this bug ever found? Other bugs (AAH002) don't have a graph, so I assume they were never found by any fuzzer, but this graph has symcc extended to 1w, but always 100% survival.

PHP build failure in aflplusplus-lto

We cannot build PHP targets by the setting of aflplusplus_lto.

ld.lld: error: undefined symbol: std::__throw_length_error(char const*)
>>> referenced by stl_vector.h:1505 (/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/stl_vector.h:1505)
>>>               lto.tmp:(std::vector<icu_60::UnicodeString, std::allocator<icu_60::UnicodeString> >::_M_check_len(unsigned long, char const*) const)
>>> referenced by stl_vector.h:1505 (/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/stl_vector.h:1505)
>>>               lto.tmp:(std::vector<icu_60::Formattable, std::allocator<icu_60::Formattable> >::_M_check_len(unsigned long, char const*) const)

It seems to be related to this issue. And in fact, as indicated in that issue, when we disable `intl', the build succeeds.

Though I'm not sure what is the root cause of this, we confirmed that we can successfully build it without LTO (aflplusplus). Therefore, LTO is doing something bad.

enhancement: check status/estimated finish time

For long jobs, it would be really nice if there was a way to check progress, either as percentage through all jobs, or an estimated time of finishing all runs.

Building docker always fails

Hello, I use magma in my Ubuntu docker, but when I run it ./run. sh, it always reports an error. The error is as follows:
error creating aufs mount to /var/lib/docker/aufs/mnt/f5ad5933593199ad2758075710eca3df75a5f59ed866c684bf95e82d9f8980a9-init: mount target=/var/lib/docker/aufs/mnt/f5ad5933593199ad2758075710eca3df75a5f59ed866c684bf95e82d9f8980a9-init data=br:/var/lib/docker/aufs/diff/f5ad5933593199ad2758075710eca3df75a5f59ed866c684bf95e82d9f8980a9-init=rw:/var/lib/docker/aufs/diff/153a85559fa6281add91df7718bc1d64036c5b28d14aab5cfdc1ddaf7479b38e=ro+wh,dio,xino=/dev/shm/aufs.xino: invalid argument

Unexpected _TARGETS behavior

Magma is really nifty and I'm plan to use it for my quals research on initial corpus impact.

Initially I wanted to just study seed behavior on one specific target (libxml2) and run all the fuzzers against it.

captainrc on line ~76 says

# [fuzzer_TARGETS]: an array of target names (from magma/targets/*) to fuzz with
# `fuzzer` (default: all targets)
# afl_TARGETS=(libpng libtiff libxml2)

So I ran with

# FUZZERS: an array of fuzzer names (from magma/fuzzers/*) to evaluate
FUZZERS=(afl aflfast)

# [fuzzer_TARGETS]: an array of target names (from magma/targets/*) to fuzz with
# `fuzzer` (default: all targets)
afl_TARGETS=(libxml2)

AFL ran just on libxml2 as expected, aflfast ran on all targets. I had noticed the name of the variable in the comment was paired to afl, however it's not uncommon for configuration variables to have outdated names or historical artifact names.

Based on #24 (comment) it sounds like the documentation is misleading. Perhaps the best fix is update the comments with more information. Where exactly is the fuzzer prefix derived from? Is it the directory name of the target? That would be a really handy thing to put in the documentation. I see you've added quite a few fuzzers since I first installed magma, so a static list probably isn't the best idea.

Thanks!

Unrelated to that, what is the best way to get in contact or stay abreast of new magma's plans? Issues? email? slack?

archive/tar failure causing openssl target not to build

openssl also failed to build on a clean copy of dev, although it's failing for a slightly different reason than libxml2, php, and poppler. This looks like it might originate in the new report tooling that is landing?

from run.sh in the terminal:

Building magma/afl/openssl
[2020-09-16 05:33] Failed to build magma/afl/openssl. Check build log for info.

from
../log/afl_openssl_build.log

this goes on for a while

tar: archive/tar: missed writing 8390 bytes"
time="2020-09-16T05:33:26Z" level=error msg="Can't add file /home/naak/magma/tools/report_df/Metric.py to tar: archive/tar: missed writing 8390 bytes"
time="2020-09-16T05:33:26Z" level=error msg="Can't add file /home/naak/magma/tools/report_df/main.py to tar: archive/tar: missed writing 8390 bytes"
time="2020-09-16T05:33:26Z" level=error msg="Can't add file /home/naak/magma/tools/report_df/templates to tar: archive/tar: missed writing 8390 bytes"
time="2020-09-16T05:33:26Z" level=error msg="Can't add file /home/naak/magma/tools/report_df/templates/fuzzer_template.md to tar: archive/tar: missed writing 8390 bytes"
time="2020-09-16T05:33:26Z" level=error msg="Can't add file /home/naak/magma/tools/report_df/templates/library_template.md to tar: archive/tar: missed writing 8390 bytes"
time="2020-09-16T05:33:26Z" level=error msg="Can't add file /home/naak/magma/tools/report_df/templates/report_template.md to tar: archive/tar: missed writing 8390 bytes"
time="2020-09-16T05:33:26Z" level=error msg="Can't close tar writer: archive/tar: missed writing 8390 bytes"
Sending build context to Docker daemon  154.1MB^M^M
Error response from daemon: Error processing tar file(exit status 1): unexpected EOF

No fuzzer included. This is just for building an analysis target

Hi, I found the run.sh in klee in the latest magma only contains

echo "No fuzzer included. This is just for building an analysis target."
exit 1

I modified it as below:

args="${ARGS/@@/"'$1'"}"
if [ -z "$args" ]; then
    args="'$1'"
fi

klee --libc=uclibc --posix-runtime "$OUT/$PROGRAM.bc" ${@:2} $args

However, I still get the same error message No fuzzer included. This is just for building an analysis target, the whole error message from the log file in ./work/log/klee_libpng_libpng_read_fuzzer_0_container.log is shown below:

[2021-07-17 19:28] Container for klee/libpng/libpng_read_fuzzer started in 5eed506edb6a
/magma/targets/libpng/corpus/libpng_read_fuzzer/not_kitty.png: exit_code 127^M
/magma/targets/libpng/corpus/libpng_read_fuzzer/not_kitty_alpha.png: exit_code 127^M
/magma/targets/libpng/corpus/libpng_read_fuzzer/not_kitty_gamma.png: exit_code 127^M
/magma/targets/libpng/corpus/libpng_read_fuzzer/not_kitty_icc.png: exit_code 127^M
Campaign launched at 2021-07-17 19:28^M
No fuzzer included. This is just for building an analysis target.^M
Campaign terminated at 2021-07-17 19:28^M

And the captainrc file is shown below:

WORKDIR=./workdir 
REPEAT=3
TIMEOUT=24h
POLL=5
FUZZERS=(klee)
afl_TARGETS=(libpng libtiff libxml2)

I tested AFL, which works fine. Would you please give me some suggestions about this error? Thank you.

Vulnerable functions that are not instrumented by ASan

Hi,

This is not so much an issue, but rather some insights I recently gained from looking more closely at the code locations instrumented by ASan and MSan. I noticed that none of the functions affected by the vulnerabilities CVE-2018-19058, CVE-2019-12360, and CVE-2017-18267 in Poppler are instrumented by ASan. The same is true for the vulnerabilities CVE-2019-9021, CVE-2018-20783, CVE-2016-10159, and CVE-2016-7414 in PHP. MSan, on the other hand, instruments all of them. So if a fuzzer fails to find these vulnerabilities, it may not be so much because of the fuzzer, but rather because the chosen sanitizer forgot to instrument the underlying vulnerable code locations.

Stephan

Update "Getting Started" document to follow the latest changes

I'm not sure whether this location is correct for posting an issue about the documents.

The current "Getting Started" document (https://hexhive.epfl.ch/magma/docs/getting-started.html) includes the following statement:

The collected Magma instrumentation can be found in name-timestamped files inside workdir/afl/libpng/libpng_read_fuzzer/0/monitor.

, but it seems obsolete. I think this should be The collected Magma instrumentation can be found in name-timestamped files inside workdir/ar/afl/libpng/libpng_read_fuzzer/0/ball.tar now.

Also git clone --branch v1.0.4 https://github.com/HexHive/magma.git should be git clone --branch v1.1 https://github.com/HexHive/magma.git now?

Magma build failure (and other issues)

Hello .

I tried many time to setup magma .

I want mainly :

1. Setup magma to run campaigns but unfortunately seems scripts are not working properly, see this log :

crypto@crypto-Standard-PC-i440FX-PIIX-1996:~/magma/tools/captain$ cat workdir/log/afl_libpng_build.log 
++ id -u root
++ id -g root
+ docker build -t magma/afl/libpng --build-arg fuzzer_name=afl --build-arg target_name=libpng --build-arg USER_ID=0 --build-arg GROUP_ID=0 --build-arg canaries=1 --build-arg isan=1 -f /home/crypto/magma/docker/Dockerfile /home/crypto/magma
Sending build context to Docker daemon  147.2MB
Step 1/59 : FROM ubuntu:18.04
 ---> 6526a1858e5d
Step 2/59 : RUN apt-get update && apt-get install -y sudo
 ---> Using cache
 ---> 5dc954e1e6e1
Step 3/59 : ARG magma_root=./
 ---> Using cache
 ---> 27be792df8c5
Step 4/59 : ENV MAGMA_R /magma
 ---> Using cache
 ---> 94e87f0117e6
Step 5/59 : ENV OUT             /magma_out
 ---> Using cache
 ---> 4c3fafcb1c5a
Step 6/59 : ENV SHARED  /magma_shared
 ---> Using cache
 ---> 17a7c6c4c785
Step 7/59 : ENV CC  /usr/bin/gcc
 ---> Using cache
 ---> f9804c69b838
Step 8/59 : ENV CXX /usr/bin/g++
 ---> Using cache
 ---> 28da6f77ed91
Step 9/59 : ENV LD /usr/bin/ld
 ---> Using cache
 ---> 3f7266ea28d7
Step 10/59 : ENV AR /usr/bin/ar
 ---> Using cache
 ---> 242350988509
Step 11/59 : ENV AS /usr/bin/as
 ---> Using cache
 ---> 55e88f84dcd4
Step 12/59 : ENV NM /usr/bin/nm
 ---> Using cache
 ---> add1d13f6a47
Step 13/59 : ENV RANLIB /usr/bin/ranlib
 ---> Using cache
 ---> d27110b1dc9b
Step 14/59 : ARG USER_ID=1000
 ---> Using cache
 ---> dacf05462bcb
Step 15/59 : ARG GROUP_ID=1000
 ---> Using cache
 ---> e2e8c6738607
Step 16/59 : RUN mkdir -p /home &&      groupadd -g ${GROUP_ID} magma &&        useradd -l -u ${USER_ID} -K UMASK=0000 -d /home -g magma magma &&      chown magma:magma /home
 ---> Running in c87a6f122e87
groupadd: GID '0' already exists
The command '/bin/sh -c mkdir -p /home &&       groupadd -g ${GROUP_ID} magma &&        useradd -l -u ${USER_ID} -K UMASK=0000 -d /home -g magma magma &&      chown magma:magma /home' returned a non-zero code: 4

This is the log after running ./run.sh in the captain dir and I don't know why it's not running .

2. I want to build a vulnerable target (say libpng) with my own fuzzer and have reports as in the magma documentation, how is this possible ? (As easy as changing scripts or the fuzzer should be built into a docker ?)

3. While setting up symcc_afl I got the following error :

[7/12] Building CXX object CMakeFiles/Symbolize.dir/compiler/Main.cpp.o
FAILED: /usr/bin/c++   -DSymbolize_EXPORTS -isystem /usr/lib/llvm-11/include -DNDEBUG  -std=c++17 -Wredundant-decls -Wcast-align -Wmissing-include-dirs -Wswitch-default -Wextra -Wall -Winvalid-pch -Wredundant-decls -Wformat=2 -Wmissing-format-attribute -Wformat-nonliteral -Werror -fPIC   -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -MMD -MT CMakeFiles/Symbolize.dir/compiler/Main.cpp.o -MF CMakeFiles/Symbolize.dir/compiler/Main.cpp.o.d -o CMakeFiles/Symbolize.dir/compiler/Main.cpp.o -c ../compiler/Main.cpp
In file included from /usr/include/llvm-11/llvm/PassSupport.h:27:0,
                 from /usr/include/llvm-11/llvm/Pass.h:318,
                 from /usr/include/llvm-11/llvm/IR/LegacyPassManager.h:19,
                 from ../compiler/Main.cpp:15:
/usr/include/llvm-11/llvm/ADT/StringRef.h:22:23: fatal error: string_view: No such file or directory
compilation terminated.
[7/12] Performing configure step for 'SymRuntime'
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler using: Ninja
-- Check for working C compiler using: Ninja -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler using: Ninja
-- Check for working CXX compiler using: Ninja -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /home/crypto/magma/fuzzers/symcc_afl/symcc/build/SymRuntime-prefix/src/SymRuntime-build
ninja: build stopped: subcommand failed. 

How to solve and build symcc_afl ?

Thanks!

php in magma_v1.2 build failed.

Hi there,

I checked out the latest magma_v1.2 and found that building php would fail with fuzzer afl. The original error log message is:

ESC[91m/usr/bin/ld: cannot find -lc++
ESC[0mESC[91mclang: error: linker command failed with exit code 1 (use -v to see invocation)
ESC[0mESC[91mmake: *** [sapi/fuzzer/php-fuzz-mbstring] Error 1
ESC[0mESC[91mmake: *** Waiting for unfinished jobs....
ESC[0mMakefile:295: recipe for target 'sapi/fuzzer/php-fuzz-mbstring' failed
ESC[91m/usr/bin/ld: cannot find -lc++
ESC[0mESC[91mclang: error: linker command failed with exit code 1 (use -v to see invocation)
ESC[0mESC[91mmake: *** [sapi/fuzzer/php-fuzz-unserialize] Error 1
ESC[0mMakefile:283: recipe for target 'sapi/fuzzer/php-fuzz-unserialize' failed
ESC[91m/usr/bin/ld: cannot find -lc++
ESC[0mESC[91m/usr/bin/ld: cannot find -lc++
ESC[0mESC[91mclang: error: linker command failed with exit code 1 (use -v to see invocation)
ESC[0mESC[91mclang: error: linker command failed with exit code 1 (use -v to see invocation)
ESC[0mMakefile:286: recipe for target 'sapi/fuzzer/php-fuzz-unserializehash' failed
ESC[91mmake: *** [sapi/fuzzer/php-fuzz-unserializehash] Error 1
ESC[0mMakefile:280: recipe for target 'sapi/fuzzer/php-fuzz-execute' failed
ESC[91mmake: *** [sapi/fuzzer/php-fuzz-execute] Error 1
ESC[0mESC[91m/usr/bin/ld: cannot find -lc++
ESC[0mESC[91mclang: error: linker command failed with exit code 1 (use -v to see invocation)
ESC[0mESC[91mmake: *** [sapi/fuzzer/php-fuzz-parser] Error 1
ESC[0mMakefile:277: recipe for target 'sapi/fuzzer/php-fuzz-parser' failed
ESC[91m/usr/bin/ld: cannot find -lc++
ESC[0mESC[91m/usr/bin/ld: cannot find -lc++
ESC[0mESC[91mclang: error: linker command failed with exit code 1 (use -v to see invocation)
ESC[0mESC[91mclang: error: linker command failed with exit code 1 (use -v to see invocation)
ESC[0mESC[91mmake: *** [sapi/fuzzer/php-fuzz-json] Error 1
ESC[0mMakefile:289: recipe for target 'sapi/fuzzer/php-fuzz-json' failed
ESC[91mmake: *** [sapi/fuzzer/php-fuzz-exif] Error 1
ESC[0mMakefile:292: recipe for target 'sapi/fuzzer/php-fuzz-exif' failed
The command '/bin/sh -c ${FUZZER}/instrument.sh' returned a non-zero code: 2

Thanks.

Program name of the bugs

Hi,

Thank you for open-sourcing this excellent fuzzing benchmark. I had a chance to run magma with different fuzzers and could trigger bugs for bunch of programs. One unclear thing in your documentation is related to the program names for specific bugs. You group bugs under libraries (e.g.. php has 16 bugs). However, most of the libraries have multiple programs and it would be good to provide the name of the program for each bug under the library (https://hexhive.epfl.ch/magma/docs/bugs.html). When I want to test a specific bug, I would be able to provide the specific program for that bug instead of fuzzing every single program in the library. Sorry this info is already documented somewhere in MAGMA. Unfortunately, I could not find after spending a good amount of time.

Sadullah

Campaign immediately terminates

First I had this error: #18, but now with the SHELL=/bin/bash it seems to get further.
However, even tho I have this captainrc script:

WORKDIR=./workdir
REPEAT=1
TIMEOUT=1m
FUZZERS=(afl)
afl_TARGETS=(libpng)

it terminates immediately:

[2020-08-31 13:30] Obtaining sudo permissions to mount tmpfs
[2020-08-31 13:30] Building magma/afl/libpng
[2020-08-31 13:30] Starting campaigns for libpng_read_fuzzer 
[2020-08-31 13:30] Building magma/curiousafl/libpng
[2020-08-31 13:30] Container afl/libpng/libpng_read_fuzzer/21 started on CPU 0
[2020-08-31 13:30] Container afl/libpng/libpng_read_fuzzer/21 stopped

Same for the log:

[2020-08-31 13:30] Container for afl/libpng/libpng_read_fuzzer started in b6e5ecbdaed7
Campaign launched at 2020-08-31 11:30
Campaign terminated at 2020-08-31 11:30

what am I doing wrong?

aflplusplus failed to fuzz binaries in persistent mode.

Hi there,

When I evaluated aflplusplus on Magma, I found that it would always find 0 bugs in any persistent-mode binaries. I checked the generated logs and found that fuzzing states remained unchanged. I attached one log file for libpng.

Thanks,
Shaohua

aflplusplus_libpng_libpng_read_fuzzer_0_container.log

Automatically stopping docker campaigns after a bug is triggered

Hi guys,

Is there a way to stop fuzzing campaigns if a specific bug is triggered? For instance, if JCH232 bug is triggered in an instance, I do not want to fuzz using that instance longer to save CPU time. If MAGMA does not have this feature and continues to fuzz until a predetermined fuzzing duration, can you point me any potential scripts that I need to look into for implementing this feature? I submit a bunch of jobs to a server (and pay) so automatically stopping campaigns will save some money + time.

Thanks

php target build error

Hi there,

I found that the current docker would fail to build the php target. The error message was "PHP does not support EBCDIC targets". php updated their config files a few days ago. Your build script unfortunately (at least on my side) does not work on the latest version. Thanks.

poppler build fails

Hi, I'm trying to use magma with my own custom fuzzer, but the poppler build seems to fail:

/magma/targets/poppler/repo/cpp/poppler-global.cpp:248:17: error: no matching function for call to 'iconv'
    size_t ir = iconv(ic, (ICONV_CONST char **)&me_data, &me_len_char, &str_data, &str_len_left);
                ^~~~~
/usr/include/iconv.h:42:15: note: candidate function not viable: no known conversion from 'const char **' to 'char **__restrict' for 2nd argument
extern size_t iconv (iconv_t __cd, char **__restrict __inbuf,
              ^
/magma/targets/poppler/repo/cpp/poppler-global.cpp:254:14: error: no matching function for call to 'iconv'
        ir = iconv(ic, (ICONV_CONST char **)&me_data, &me_len_char, &str_data, &str_len_left);
             ^~~~~
/usr/include/iconv.h:42:15: note: candidate function not viable: no known conversion from 'const char **' to 'char **__restrict' for 2nd argument
extern size_t iconv (iconv_t __cd, char **__restrict __inbuf,
              ^
/magma/targets/poppler/repo/cpp/poppler-global.cpp:302:17: error: no matching function for call to 'iconv'
    size_t ir = iconv(ic, (ICONV_CONST char **)&str_data, &str_len_char, &ret_data, &ret_len_left);
                ^~~~~
/usr/include/iconv.h:42:15: note: candidate function not viable: no known conversion from 'const char **' to 'char **__restrict' for 2nd argument
extern size_t iconv (iconv_t __cd, char **__restrict __inbuf,
              ^
/magma/targets/poppler/repo/cpp/poppler-global.cpp:308:14: error: no matching function for call to 'iconv'
        ir = iconv(ic, (ICONV_CONST char **)&str_data, &str_len_char, &ret_data, &ret_len_left);
             ^~~~~
/usr/include/iconv.h:42:15: note: candidate function not viable: no known conversion from 'const char **' to 'char **__restrict' for 2nd argument
extern size_t iconv (iconv_t __cd, char **__restrict __inbuf,
              ^
4 errors generated.
[100%] Linking CXX static library libpoppler-cpp.a
/usr/bin/ar: CMakeFiles/poppler-cpp.dir/poppler-global.cpp.o: No such file or directory
cpp/CMakeFiles/poppler-cpp.dir/build.make:406: recipe for target 'cpp/libpoppler-cpp.a' failed
make[3]: *** [cpp/libpoppler-cpp.a] Error 1
CMakeFiles/Makefile2:734: recipe for target 'cpp/CMakeFiles/poppler-cpp.dir/all' failed
make[2]: *** [cpp/CMakeFiles/poppler-cpp.dir/all] Error 2
make[1]: *** [cpp/CMakeFiles/poppler-cpp.dir/rule] Error 2
CMakeFiles/Makefile2:746: recipe for target 'cpp/CMakeFiles/poppler-cpp.dir/rule' failed
Makefile:383: recipe for target 'poppler-cpp' failed
make: *** [poppler-cpp] Error 2

The other targets are ok and this error should not be related to my fuzzer , I'm just using afl-clang-fast of AFL++ to instrument.

Maybe it is a cmake related error? https://gitlab.freedesktop.org/poppler/poppler/blob/master/cmake/modules/FindIconv.cmake#L26 here ICONV_CONST is set to const only after a test.

Missing main() in sapi/fuzzer/fuzzer-exif.c, etc?

@adrianherrera
Hi, Adrian, my own fuzzer cannot be running in the MAGMA docker.

I have to run my fuzzer outside the docker, so I need to compile the targets outside the docker.

All the targets succeeded except the php.

I can see the cause of failure is that there is no main() in sapi/fuzzer/fuzzer-exif.c, sapi/fuzzer/fuzzer-parser, etc.

I added

int main()
{
    return 0;
}

to these files, they can be compiled successfully.

But this compiled binary cannot run in the fuzzer.

Any tutorials to write the main() function or am I missing some steps in compiling the targets outside the docker?

Thanks!

New CVEs in libxml2

Hi,

There are three brand new CVEs found by OSS-Fuzz in libxml2 (CVE-2020-27917, CVE-2020-27911, CVE-2020-27926) - it would be good to include them in Magma as soon as the CVE descriptions are available.

Stephan

${MAGMA}/apply_patches.sh failing on libxml2, php, poppler

I pulled from dev( lovely to see all the report work landing :) ) and ran into some issues building the targets. These all look similar. In that it looks like patching failed. openssl also failed, but that looks like a different issue and merits its own ticket.

../log/afl_libxml2_build.log

Applying /magma/targets/libxml2/patches/bugs/AAH036.patch
patching file HTMLparser.c
Hunk #1 FAILED at 2787.
Hunk #2 FAILED at 2804.
Hunk #3 FAILED at 2822.
Hunk #4 FAILED at 2847.
Hunk #5 FAILED at 2854.
Hunk #6 FAILED at 2867.
Hunk #7 FAILED at 2882.
7 out of 7 hunks FAILED -- saving rejects to file HTMLparser.c.rej
The command '/bin/sh -c ${TARGET}/fetch.sh && ${MAGMA}/apply_patches.sh' returned a non-zero code: 1

/log/afl_php_build.log

Applying /magma/targets/php/patches/bugs/MAE015.patch
patching file ext/exif/exif.c
Hunk #1 FAILED at 3941.
Hunk #2 FAILED at 3962.
Hunk #3 succeeded at 4745 (offset 42 lines).
2 out of 3 hunks FAILED -- saving rejects to file ext/exif/exif.c.rej
The command '/bin/sh -c ${TARGET}/fetch.sh && ${MAGMA}/apply_patches.sh' returned a non-zero code: 1

/log/afl_poppler_build.log

Step 44/59 : RUN ${TARGET}/fetch.sh && ${MAGMA}/apply_patches.sh
 ---> Running in fc4a5ab1dc5b
^[[91mCloning into '/magma/targets/poppler/repo'...
^[[0m^[[91mCloning into '/magma/targets/poppler/freetype2'...
^[[0m^[[91mfind: '/magma/targets/poppler/patches/setup': No such file or directory
^[[0mApplying /magma/targets/poppler/patches/bugs/AAH051.patch
patching file poppler/Parser.cc
Hunk #1 FAILED at 283.
1 out of 1 hunk FAILED -- saving rejects to file poppler/Parser.cc.rej
The command '/bin/sh -c ${TARGET}/fetch.sh && ${MAGMA}/apply_patches.sh' returned a non-zero code: 1

Building targets only?

Is it possible to only build the targets, without Docker but directly on the local system? I tried with libpng and libtiff, by first executing fetch.sh and than build.sh, with the necessary flags set. But the result is a "undefined reference to `main'" error in both cases, so I guess I'm missing some important step?

Does a Bug ID to CVE ID mapping exist for the libtiff target?

Hello I am interested in the CVE ID for each BUG ID in the libtiff target. The files for libtiff are missing or do not exist in the magma/docs/Buggy_MDs folder from pull request #11 .
Does a Bug ID to CVE ID mapping exist for the libtiff target?

Bench/Fuzzer start failed

Hi,

Thank you for the work! However, something may wrong with run.sh during my trial.

pwd: magma/tools/captain
./run.sh
[2020-07-30 23:53] Obtaining sudo permissions to mount tmpfs
[2020-07-30 23:53] Building magma/afl/libpng
[2020-07-30 23:53] Starting campaigns for libpng_read_fuzzer
[2020-07-30 23:53] Waiting for jobs to finish
zsh:1: command not found: launch_campaign

The config file is listed below:

WORKDIR=./workdir
REPEAT=1
WORKERS=1
TIMEOUT=10m
POLL=5
FUZZERS=(afl)
afl_TARGETS=(libpng)

symcc_afl build failed

Hi, I'm having issues building symcc_afl with target libpng, It complains the cmake version is too low.

#17 887.8 /magma/fuzzers/symcc_afl/symcc
#17 887.8 CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
#17 887.8   CMake 3.13.4 or higher is required.  You are running version 3.10.2
#17 887.8 
#17 887.8 
#17 887.8 -- Configuring incomplete, errors occurred!
#17 ERROR: executor failed running [/bin/sh -c ${FUZZER}/fetch.sh && ${FUZZER}/build.sh]: exit code: 1

full build log:
symcc_afl_libpng_build.log

Which release to use?

Sorry about opening an issue that is really a usage question.

Your homepage under "getting started" recommends to use v1.0.4. However, this repo already has release v1.1.0 since October.

What do you suggest we use? (since magma is computationally quite heavy, I wanted to ask first before spending all the ressource on the wrong version)

thx! Also please let me know if there's another place that is more suited for usage questions..

exp2json vs poc_extract.sh discrepancy

Hi,

Thanks for this excellent dataset and the terrific documentation. I hope it will be widely used.

Problem: I invoked run.sh (captainrc below). After it finished, I expected the files in directory poc to match the triggerability results generated from using python3 ../benchd/exp2json.py workdir/ long_run.json. It does not match.

The results all match for Libfuzzer, but break for Angora. I include a small snippet below, but can include the entire output if it would help.

I investigated a bit further, and it seems something is off with the run_once.sh invoked from the poc_extract.sh script but I am unsure how to proceed. I also see in the FAQ The [fuzzer]/run_once.sh scripts in Magma are intended to emulate the fuzzer's execution environment to detect faults., suggesting this is intended?

My question is for triggerability in Angora, should I trust the *.json results or the files in poc ? Do you know why this might happen?

(base) user@host magma-1.1.0/tools/captain$ ls workdir/poc/ | uniq | rev | cut -c 4- | rev | uniq

angora_libxml2_xmllint_AAH037.
angora_libxml2_xmllint_AAH041.
libfuzzer_libxml2_libxml2_xml_read_memory_fuzzer_AAH037.
libfuzzer_libxml2_libxml2_xml_read_memory_fuzzer_AAH041.


(base) user@host magma-1.1.0/tools/captain$ python -m json.tool long_run.json
{
    "results": {
        "angora": {
            "libxml2": {
                "libxml2_xml_read_memory_fuzzer": {
                    "0": {
                        "reached": {
                            "AAH032": 15,
                            "AAH029": 15,
                            "AAH037": 10,
                            "AAH024": 15,
                            "AAH035": 25,
                            "AAH034": 15,
                            "AAH031": 75,
                            "AAH026": 15,
                            "AAH041": 15
                        },
                        "triggered": {
                            "AAH037": 15, # does not appear in poc/*
                            "AAH026": 25, # does not appear in poc/*
                            "AAH041": 15 # does not appear in poc/*
                        }
                    }
                },
                "xmllint": {
                    "0": {
                        "reached": {
                            "AAH032": 15,
                            "AAH029": 15,
                            "AAH037": 10,
                            "AAH024": 15,
                            "AAH035": 15,
                            "AAH034": 20,
                            "AAH031": 3615,
                            "AAH026": 15,
                            "AAH041": 15
                        },
                        "triggered": {
                            "AAH037": 12970,
                            "AAH041": 25
                        }
                    }
                }
            },
      ...
        "libfuzzer": {
            "libxml2": {
                "libxml2_xml_read_memory_fuzzer": {
                    "0": {
                        "reached": {
                            "AAH032": 15,
                            "AAH029": 15,
                            "AAH037": 10,
                            "AAH024": 15,
                            "AAH035": 15,
                            "AAH034": 15,
                            "AAH031": 15,
                            "AAH026": 15,
                            "AAH041": 15
                        },
                        "triggered": {
                            "AAH037": 15,
                            "AAH041": 15
                        }
                    }
                }
            },

Here is the captainrc

# This file contains the configuration for the run.sh script. It follows the
# Bash syntax and is sourced by the script to access the variables. Variables
# are mandatory unless marked with [brackets].

###
## Configuration parameters
###

# WORKDIR: path to directory where shared volumes will be created
WORKDIR=./workdir

# REPEAT: number of campaigns to run per program (per fuzzer)
REPEAT=1

# [WORKER_MODE]: defines the type of CPU resources to allocate (default: 1)
# - 1: logical cores (possibly SMT-enabled)
# - 2: physical cores
# - 3: physical sockets (1 worker per CPU socket)
# WORKER_MODE=1

# [WORKERS]: number of worker threads (default: all cores)
# WORKERS=3

# [WORKER_POOL]: a space-separated list of logical cores to allocate
# WORKER_POOL="1 3 5 7 9"

# [CAMPAIGN_WORKERS]: number of workers to allocate for a campaign (default: 1)
# CAMPAIGN_WORKERS=1

# [TIMEOUT]: time to run each campaign. This variable supports one-letter
# suffixes to indicate duration (s: seconds, m: minutes, h: hours, d: days)
# (default: 1m)
TIMEOUT=6h

# [POLL]: time (in seconds) between polls (default: 5)
POLL=5

# [CACHE_ON_DISK]: if set, the cache workdir is mounted on disk instead of
# in-memory (default: unset)
# CACHE_ON_DISK=1

# [NO_ARCHIVE]: if set, campaign workdirs will not be tarballed (default: unset)
NO_ARCHIVE=1

# [TMPFS_SIZE]: the size of the tmpfs mounted volume. This only applies when
# CACHE_ON_DISK is not set (default: 50g)
# TMPFS_SIZE=16g

# [MAGMA]: path to magma root (default: ../../)
# MAGMA=/path/to/magma/

# [CANARY_MODE]: defines the mode of canaries at compile time (default: 1)
# - 1: without fixes, with canaries
# - 2: without fixes, without canaries
# - 3: with fixes, without canaries
# CANARY_MODE=3

# [ISAN]: if set, build the benchmark with ISAN/fatal canaries (default: unset)
ISAN=1

# [HARDEN]: if set, build the benchmark with hardened canaries (default: unset)
# HARDEN=1

# [POC_EXTRACT]: if set, run the extract.sh script after the campaign is done
# (default: unset)
POC_EXTRACT=1


###
## Campaigns to run
###

# FUZZERS: an array of fuzzer names (from magma/fuzzers/*) to evaluate
FUZZERS=(angora libfuzzer)

# [fuzzer_TARGETS]: an array of target names (from magma/targets/*) to fuzz with
# `fuzzer`. The `fuzzer` prefix is a fuzzer listed in the FUZZERS array
# (default: all targets)

# [fuzzer_target_PROGRAMS]: an array of program names (from
# magma/targets/target/configrc) to use as execution drivers when fuzzing the
# `target`
# afl_libtiff_PROGRAMS=(tiffcp)

# [fuzzer_target_FUZZARGS]: a string containing fuzzer/target-specific arguments
# when fuzzing `target` with `fuzzer`
# afl_libpng_FUZZARGS="-x /magma_shared/png.dict"

# [fuzzer_CAMPAIGN_WORKERS]: overrides the global CAMPAIGN_WORKERS setting
# afl_CAMPAIGN_WORKERS=3

v1.2 symcc_afl/php failed to build

Hi, symcc_afl builds for all targets but php, here is the build log:

symcc_afl_php_build.log

Compiling each applications separately

Hi, I would like to compile Magma applications separately without requiring them to run in the docker. I wanted to have just simple plain applications but with bugs in them. Can we do that?

honggfuzz error in Lua

Hi there,

When I was trying to fuzz Lua with honggfuzz, I got the following running error log:

[2021-09-08T07:53:41+0000][E][555] cmdlineVerify():243 You must specify 'FILE' if the -s (stdin fuzzing) or --persistent options are not set
[2021-09-08T07:53:41+0000][F][555] main():330 Parsing of the cmd-line arguments failed.

Thanks,
Shaohua

Building the docker container fails

Instructions seem to be incorrect.
At the moment, it is not possible to build the container and there are some issues such as trying to copy from a parent path. Something that docker does not allow.

Failed to build magma/klee/libpng

I've tried to evaluate klee with magma and libpng, but I got the following error:

[2021-01-28 07:39] Building magma/klee/libpng
[2021-01-28 07:47] Failed to build magma/klee/libpng. Check build log for info.

The build log is as follows:

...
-- Found klee-uclibc library: "/magma/fuzzers/klee/uclibc/lib/libc.a"
-- klee-libcxx support enabled
-- Use libc++ include path: "/magma/fuzzers/klee/libcxx/libc++-install-9/include/c++/v1"
^[[91mCMake Error at CMakeLists.txt:579 (message):
   does not exist. Try passing -DKLEE_LIBCXXABI_SRC_DIR=<path>

^[[0m-- Configuring incomplete, errors occurred!
See also "/magma/fuzzers/klee/klee/build/CMakeFiles/CMakeOutput.log".
See also "/magma/fuzzers/klee/klee/build/CMakeFiles/CMakeError.log".

The complete reproducible steps are as follows:

1. Install VirtualBox on Mac OS
2. vagrant init generic/ubuntu1804 (so I'm using Ubuntu 18.04 on VirtualBox)
3. vagrant up --provider=virtualbox
4. ssh -p 2222 vagrant@localhost
5. Entere the Ubuntu terminal on VirtualBox
6. sudo apt-get update && sudo apt-get install -y util-linux inotify-tools docker.io git
7. sudo gpasswd -a $USER docker
8. sudo systemctl restart docker
9. git clone https://github.com/HexHive/magma.git
10. git log

commit 5d81041f3d9154ade9f0fe889fb65f3f77329571 (HEAD -> v1.1, origin/v1.1, origin/HEAD)
Author: Ahmad Hazimeh <[email protected]>
Date:   Thu Dec 17 17:44:47 2020 +0100

    Anchored sqlite3 to proper working reference

11. cd magma/tools/captain
12. Configure captainrc as follows:

REPEAT=1
TIMEOUT=1m
FUZZERS=(aflplusplus honggfuzz angora klee)
aflplusplus_TARGETS=(libpng)
honggfuzz_TARGETS=(libpng)
angora_TARGETS=(libpng)
klee_TARGETS=(libpng)

13. ./run.sh

Parmesan libpng: thread 'main' panicked at 'Could not read cfg targets file ...'

When starting parmesan for target libpng the campaign terminates immediately and I get the following error:

[2021-08-16 16:59] Container for parmesan/libpng/libpng_read_fuzzer started in 147b41eb0169
Campaign launched at 2021-08-16 14:59
thread 'main' panicked at 'Could not read cfg targets file: Custom { kind: InvalidData, error: Error("invalid digit found in string", line: 5, column: 1) }', fuzzer/src/fuzz_main.rs:42:72
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Campaign terminated at 2021-08-16 14:59

A may-be incorrect patch file of libpng

Hi!

Thanks for your great efforts of bringing Magam benchmark.

I am playing with the benchmark and try to do some hack. I noticed the following code.

magma/targets/libpng/patches/bugs/PNG005.patch

Line 24 in 4652a9c

+#if MAGMA_ENABLE_CANARIES

I suppose it should be

 #ifdef MAGMA_ENABLE_CANARIES 

I am not sure whether I completely understand the meaning of MAGMA_ENABLE_CANARIES. But if I simply define a macro named as MAGMA_ENABLE_CANARIES, the compilation would fail. Please kindly correct if I am wrong.

Thanks for your help in advance.

Hangs after building libpng.

afl_libpng_build.log
Hello,
To my understanding, the Magma benchmark will move on to build its next campaign after completing the build for one campaign so it can work on multiple campaigns in parallel. However, my benchmark never moves onto building the next campaign after finishing afl/libpng. I've attached the log of a previous attempt showing that afl/libpng was successfully built and a screenshot of my current attempt showing when I started and the time the screenshot was taken at.

Why put in a docker?

I am pretty curious of why put all the fuzzers and targets and fuzzing campaign inside a docker???

Imagine Jack is a skilled practioner in fuzzing. He already has all the settings and configurations (e.g., AFL, clang, ...) installed on his own computer.

One day he has to use this MAGMA data set. (Maybe suggested by reviewers from a top conference:-).) But all the fuzzers and targets are going to be reinstalled in a docker. If Jack is in China, the connection or git clone operation to GitHub is always unconnected these days. Sadly, Jack has no way to run this data set if the fuzzer and targets cannot be cloned into the docker.

I strongly suggest just give Jack the patched target source code. People who want to use this data set 100% have all the fuzzers and pre-settings installed on their own computer. Only the targets inserted with bugs are what Jack needs. And last but not least, don't try to push people to redo all the things inside a docker. It's really a waste of time for a skilled fuzzing practioner such as Jack. :-)

PoCs for vulnerabilities in Magma

Hi,
I am using Magma to test our fuzzing system but I can't find PoCs in the directory. Are stored in other locations?

angora and parmesan failing builds

Hi, i found some builds that fail. Angora fails for targets libsndfile, php, and openssl, while parmesan fails with libsndfile, and sqlite3. Building parmesan with target php seems to not finish or it takes a very long time. Other fuzzers seem to build with these targets. Providing build logs below. Error for libsndfile seem to be same issue for both targets (undefined reference in alsa_open).

parmesan_php_build.log (canceled by user)
parmesan_sqlite3_build.log
angora_libsndfile_build.log
angora_openssl_build.log
angora_php_build.log
parmesan_libsndfile_build.log

Openssl bugs to potentially graveyard?

Thank you for this challenging and useful dataset! After reviewing some of the openssl bugs manually, I currently believe

MAE114 appears to be untriggerable. The bug condition here requires peek=1 . However, the harnesses set peek=0 explicitly or implicitly here and here. The original bug report confirms this property here.

MAE111 appears to be untriggerable too. I did not analyze it as carefully as the others, but upon reading this comprehensive description here, the default certificates lacking a particular elliptic curve provided by the fuzzer harnesses may never trigger the bug.

I am happy to provide a POC for demonstrability of MAE114.

Given this information, should these bugs be moved to the graveyard or should the harnesses be fixed? What do you think?

Magma Raw Data

Where can I access the raw data from the MAGMA experiments (the data shown in the sample report in the MAGMA homepage and the paper)?

Libpng bugs to potentially graveyard?

Thank you for this challenging and useful dataset! After reviewing some of the libpng bugs manually, I currently believe

AAH004 appears to be untriggerable by the current harness. First, the harness validates that height*width < 100000000. In the best case, height=1, width=100000000. Equivalently, 2^26 < width < 2^27. Second, even with the largest transformed_pixel_depth of 64 set around here and checked here, the pixel_depth is divided by 8 at the bug site. Hence, 2^26 * 2^3 cannot overflow 2^32 as required by the Magma bug condition. Note that because the harness also bounds memory allocations here, there will also be a nullptr supplied here that I believe will early terminate the program as well.

AAH005 appears to be untriggerable by the current harness for similar reasons. The PNG_ROWBYTES macro performs a similar divide-by-8 operation at the bug site here. Additionally, the bug site is also guarded by the height*width < 100000000 check. Interestingly enough, AAH001 does not suffer from this issue because it is called before the height*width < 100000000 check activates.

I am happy to provide POCs for demonstrability of both.

Given this information, should these bugs be moved to the graveyard or should the harnesses be fixed? What do you think?

PoC for the untriggered bugs

Hi!

Thanks again for the meaningful work!

I have encountered some libpng bugs which seems untriggerable. Other issues (libpng, openssl, and php) discuss those bugs with more details.

Given those untriggerable bugs, it would be hard to gauge the capability of a vulnerability detection technique (i.e., we cannot tell an untriggered bug is hidden from the technique or indeed untriggerable).

Hence, I am wondering whether it is possible to provide the PoC for those untriggered bugs (i.e., not triggered by any measured fuzzers)? It would greatly reduce the manual efforts of calibrating untriggerable bugs.

For those triggered bugs, I believe we can access their PoC from the Magma homepage.

I sincerely understand that we may not have PoC for all the untriggered bugs.

Thanks!

Failed to build magma targets with KLEE

Hi, I still have a problem building magma targets with KLEE. The commit version is shown below:

1e9d2399 (HEAD -> dev, origin/dev) benchd: save raw time-to-bug values

The error message from the log is shown below:

Step 44/59 : RUN ${TARGET}/fetch.sh && ${MAGMA}/apply_patches.sh
 ---> Running in c3882604ffc2
^[[91mCloning into '/magma/targets/libtiff/repo'...
^[[0m^[[91mwarning: redirecting to https://gitlab.com/libtiff/libtiff.git/
^[[0mApplying /magma/targets/libtiff/patches/setup/libtiff.patch
patching file CMakeLists.txt
Hunk #1 succeeded at 595 (offset -107 lines).
Applying /magma/targets/libtiff/patches/bugs/AAH012.patch
patching file libtiff/tif_ojpeg.c
Hunk #1 succeeded at 790 (offset 1 line).
Applying /magma/targets/libtiff/patches/bugs/AAH013.patch
patching file libtiff/tif_luv.c
Hunk #1 succeeded at 1566 (offset -21 lines).
Hunk #2 succeeded at 1577 (offset -21 lines).
Applying /magma/targets/libtiff/patches/bugs/AAH019.patch
patching file libtiff/tif_print.c
Hunk #1 FAILED at 544.
1 out of 1 hunk FAILED -- saving rejects to file libtiff/tif_print.c.rej
The command '/bin/sh -c ${TARGET}/fetch.sh && ${MAGMA}/apply_patches.sh' returned a non-zero code: 1

And the captainrc file is shown below:

WORKDIR=./workdir
REPEAT=3
TIMEOUT=24h
POLL=5
FUZZERS=(klee)
klee_TARGETS=(libtiff)

I also tested other targets, but none of them were successfully built with KLEE.

Support directed fuzzing

Hi!
I'm trying to use magma to evaluate the performance of directed fuzzing.
I think the idea of having patches with ground-truth bugs really makes it a wonderful data set.

I made some changes to make magma work for directed fuzzers. I use the 'patches/bugs' as goals for directed fuzzers to reach.
You can find them in this fork:
https://github.com/usc-isi-bass/magma/tree/aflgo

It is not yet completed though. I'm opening this issue to see if you guys are interested in this feature, and I can create a PR later.

Currently my fork can run aflgo on libpng, libxml2 without any issues.
For other targets, I'm still trying to compile them. Also, the 'distance calculation' phase of aflgo for libxml2 takes extremely long time to compute.
I might open an issue at aflgo to see if the authors can help for these problems.

Php bugs to potentially graveyard?

Thank you for this challenging and useful dataset. After reviewing some of the php bugs manually, I currently believe

MAE006 appears to be untriggerable with the current harness. The harness sets read_thumbnail=0. Consequently, ImageInfo.Thumbnail.data remains 0 and hence, the logical AND of MAE006 short circuits. Moreover, this effect results in an early exit from exif_scan_thumbnail; this likely explains why bugs MAE010 and MAE015 have never been reached.

MAE004 can be triggered by the current harness, but it is not detected by Magma. The bug condition relies on an architecture-dependent SIZE_MAX. Since dir_offset is a 32-bit value, the detection logic fails. The original bug report here confirms this property.

I am happy to provide POCs for demonstrability of both.

Given this information, should these bugs be moved to the graveyard or should the harnesses be fixed? What do you think?

Patching errors for poppler, php, libxml2, openssl

Running fuzzers for these targets fails due to patching errors:

• poppler: AAH042
• php: MAE006
• libxml2: AAH036
• openssl: MAE110

I saw that for the targets you clone the master branch and apply the patches on this branch, maybe you could pin the cloning to a specific commit where you know the patches will succeed?