This repository contains a general coverage-guided fuzz testing framework, DeepHunter
, for testing deep neural networks.
Installation (Automatic)
Run installation script
bash setup.sh
Entering/Exiting Environment
Enter virtual environment with:
. envs.sh
Exit virtual environment with:
deactivate
Installation (Manual)
Get pip for python2.
wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
Install pip for python2.
python2 get-pip.py
Set up virtual environment in .venv
folder.
python2 -m pip install virtualenv && python2 -m virtualenv .venv
Enter virtual environment
. .venv/bin/activate
Install requirements.
pip install -r requirements.txt
Our original DeepHunter (version 1.0) is implemented on top of AFL. We publicize DeepHunter 2.0 which is a reimplementation on top of the TensorFuzz framework. It is structured as follows:
├── README.md
├── deephunter/
├── lib/
├── test_seeds/
└── requirements.txt
This directory contains the core implementation of DeepHunter, including the metamorphic mutation, coverage analysis (i.e., Neuron Coverage and the metrics in DeepGuage), and test selection strategies.
The subdirectory profile provides 7 profiling files (for 5 trained models and 2 ImageNet models in Keras) required for coverage analysis in DeepGuage. 4 trained models including LeNet-1 LeNet-4 LeNet-5 and ResNet-20 can be found. For other 3 models, VGG16 are available here, and MobileNet and RestNet-50 are the pre-trained weights from Keras2.1.3.
Similar to TensorFuzz, this directory contains the general implementation of fuzz loop and the seed queue.
We provide some initial test seeds for MNIST, CIFAR-10 and ImageNet-10.
The user can generate more initial seeds with utils/ConstructInitialSeeds.py
.
We provide a script to automatically run DeepHunter fuzzer as well as results analysis.
cd deephunter_source/deephunter/
sh test_example.sh
The scrpit tests LeNet-5 with different seed selection strategies and two testing criteria (i.e., KMNC, NBC). The user can follow the commands to add configurations with other testing criteria.
cd deephunter_source/deephunter/
python comprehensive_fuzzer.py
-i ../test_seeds/mnist_seeds
-o mnist_output
-model lenet5
-criteria kmnc
-random 0 -select prob
The meanings of the options are:
-model
determines which model to be tested, where the 7 pre-profiled files are:vgg16
,resnet20
,mobilenet
,resnet50
,lenet1
,lenet4
,lenet5
. To be started easily, we integrated 7 models, i.e., their profiling files can be found by DeepHunter. It is fairly possible to test other models; in such scenarios, please useutils/Profile.py
to profile the new model firstly. Then, it is also easy to integrate the new model and the profiling file in DeepHunter.-criteria
selects different testing criteria; possible values are:kmnc
,nbc
,snac
,bknc
,tknc
,nc
.-random
determines whether applying random testing (1) or coverage-guided testing (0); defaults to 0.-select
chooses the selection strategies (when-random
is 0) fromuniform
,tensorfuzz
,deeptest
,prob
.
Note that LeNet models are used for MNIST; vgg16 and resnet20 are for CIFAR-10; mobilenet and resnet50 are for ImageNet.
So if you choose one specific model, please make sure that the initial seeds (-i
) are consistent with the model.
cd deephunter_source/deephunter/
python utils/Profile.py
-model model_path
-train mnist
-o example.pickle
The meanings of the options are:
-model
specifies the path of the model which can someh5
file such as lenet1.h5.-train
chooses the datasets to be profiled which can be eithermnist
(MNIST) andcifar
(CIFAR-10). These are widely used datasets that can be obtained from Keras. It is absolutely easy to extend with other datasets.
Please cite the following paper if DeepHunter helps you on the research:
@inproceedings{deephunter,
Author = {Xiaofei Xie and Lei Ma and Felix Juefei-Xu and Minhui Xue and Hongxu Chen
and Yang Liu and Jianjun Zhao and Bo Li and Jianxiong Yin and Simon See},
Booktitle = {28th ACM SIGSOFT International Symposium on Software Testing and Analysis},
Title = {DeepHunter: A Coverage-Guided Fuzz Testing Framework for Deep Neural Networks},
Year = {2019}}
cd text_mutation_testing/
python2 mut_test.py
The output will go into mutation_test_single_mutation.out and mutation_test_10x_mutation.out.