peikexin9 / deepxplore

DeepXplore code release

DeepXplore: Systematic DNN testing (SOSP'17)

See the SOSP'17 paper DeepXplore: Automated Whitebox Testing of Deep Learning Systems for more details.

Prerequisite

Python

The code should be run using python 2.7.12, Tensorflow 1.3.0, Keras 2.0.8, PIL, h5py, and opencv-python

Tensorflow

sudo pip install tensorflow

if you have gpu,

pip install tensorflow-gpu

Keras

pip install keras

To set Keras backend to be tensorflow (two options):

1. Modify ~/.keras/keras.json by setting "backend": "tensorflow"
2. KERAS_BACKEND=tensorflow python gen_diff.py

PIL

pip install Pillow

h5py

pip install h5py

opencv-python

pip install opencv-python

Mimicus

Install from here.

File structure

• MNIST - MNIST dataset.
• ImageNet - ImageNet dataset.
• Driving - Udacity self-driving car dataset.
• PDF - Benign/malicious PDFs captured from VirusTotal/Contagio/Google provided by Mimicus.
• Drebin - Drebin Android malware dataset.

To run

In every directory

python gen_diff.py

Note

The trained weights are provided in each directory (if required). Drebin's weights are not part of this repo as they are too large to be hosted on GitHub. Download from here and put them in ./Drebin/.

Note that as DeepXplore use randomness for its exploration, you should fix the seed of the random number generator if you want deterministic and reproducable results. An example is shown below.

import numpy as np
import random

random.seed(1)
np.random.seed(1)

Coming soon

How to test your own DNN models.

deepxplore's Issues

How to maximize neuron coverage?

Hi~@peikexin9
In my experiment, I use the parameters you provided and I also try 50 grad_iterations.
The neuron which is random selected in the program, will not be activated after all iterations done.
So, I want to find the answer about how to increase the neuron coverage of one model, but I still didn't understand.
If the grad_iterations is not big enough to find an input to activate the neuron, when the program find the differential-inducing input?
Thank you~

License missing

Can you select a license and add it to the project?

Why to calculate K.means during calculate loss?

Hi I'm graduate students who are interesting in DeepXplore. Actually I saw your presentation in conference. After the conference I tried to understand your source code more deeply, however I got some issues.

One thing is that when you calculate the loss, actually not only loss(loss*, loss*_neuron,...) you all add K.means function to them. I have no idea why do we calculate the means of such losses. For instance, in case of loss1, is the tensor type just one scalar value which represents confidence price of label orig_label? If not, can you give me an advice?

Thanks

Why do you add 103.939 to R channel of image in deprocess_image function?

Caffe instance

Hi Pei!,

what is the feasibility of deploying the framework described in your paper to the Caffe based instance, is it possible enough.

Thanks-
Rajasehkar

The problem of neuron output

According to the definition of neuron, A neuron is an individual computing unit inside a DNN that applies an activation function on its inputs and passes the result to other connected neurons. I get a ndarray called feature map in experiment. Is the feature map the output of neuron or np.mean(feature map).
Thank you for your reply

Required packages

In order to get this running I additionally needed to install:
PIL, h5py and opencv-python

keras must also be configured with a tensor flow backend, e.g. export KERAS_BACKEND=tensorflow

Ambiguity about Coverage

It's not clear how covered is calculated.

• Why are the neurons scaled?
• Why are they averaged to decide to decide if they are fired?

deepxplore/PDF/utils.py

Line 93 in dc8b405

if np.mean(scaled[..., num_neuron]) > threshold and not model_layer_dict[(layer_names[i], num_neuron)]:

Adversarial example classified correctly

The adversarial samples like 'occl_4_to_7_model1.png' are being classifed correctly by model1.

there are two problem I need your help when I run the program Driving.

The command I used is:
python gen_diff.py occl -t 0 1.0 0.1 10 10 10 0
And my env is Tensorflow1.4.0，keras 2.1.1，and so on.

the problem is :
/deep-test/deepxplore/Driving/utils.py:133: RuntimeWarning: invalid value encountered in divide
intermediate_layer_output.max() - intermediate_layer_output.min())

Traceback (most recent call last):
File "gen_diff.py", line 155, in
gen_img_deprocessed = draw_arrow(deprocess_image(gen_img), angle1, angle2, angle3)
File "/deep-test/deepxplore/Driving/utils.py", line 49, in deprocess_image
x[:, :, 1] += 116.779
IndexError: index 1 is out of bounds for axis 2 with size 1

I changed the version of Keras, but it's useless.

So, I want to get your help to deal with it.

Looking forward to your reply!

IndexError: list index out of range

Hi Pei!,

I'm using below environment for code execution:
OS: Debian GNU/Linux (sid)
Tensorflow, Keras installed, and working well.

After setting up the environment, this IndexError is keep on getting. But the random.choice is considered to take the 'img_paths' as list, then i don't know why.. could you help. Below is the Traceback verbose. I Issued python gen_diff.py occl 1 0.1 10 1 50 0 in the ImageNet folder.

Traceback (most recent call last): File "gen_diff.py", line 55, in <module> gen_img = preprocess_image(random.choice(img_paths)) File "/usr/lib/python2.7/random.py", line 277, in choice return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty IndexError: list index out of range

a question about Dring model

The paper of NVIDIA about self-driving introduce the model you use, but the input layer is 200*66.

In your model, the size of input tensor is 100*100.

Which one is the right size in real world?

Thank you~~~

运行Derbin报错

我下载了Mode1-3文件并存在./Drebin/目录下，运行python gen_diff.py 1 0.5 20 50 0报错

然后我尝试将test文件夹修改为dataset，仍然报错，且错误提示和#12
中相同
ValueError: Dimension 0 in both shapes must be equal, but are 295 and 545334 for 'Assign' (op: 'Assign') with input shapes: [295,200], [545334,200].

The code you provide does not include all the experiments in paper？

Hello, Thank you very much for providing the source code of the paper.
but, The code you provide does not include all the experiments in paper？eg.,the code of section 7.3?

Where to find the Drebin models?

Hi,

Are the drebin models that you used in your study available anywhere? There is a link in the README, however it does not seem to work.

cannot run gen_diff.py under Drebin

got the error below when run "python gen_diff.py 1 0.5 20 50 0" under Drebin, do you know where the issue might be?

ValueError: Dimension 0 in both shapes must be equal, but are 295 and 545334 for 'Assign' (op: 'Assign') with input shapes: [295,200], [545334,200].

Labels in Driving/testing/final_example.csv

Hi, I am confused about the meaning of Steering Angle in final_example.csv, as they doesn't match the ground truth steering angle.
Thank you for your reply~

Computing the coverage

Hi, How can I compute the coverage of an NN? Is the code in this repo?

Neuron coverage

First of all thanks for providing the code to your paper that really helps to understand the approach better.

As far as I see, neuron coverage is calculated based on the outputs of all layers except input layers and flatten layers, e.g. init_dict in utils.py. So neuron coverage is basically activation function coverage for each "relevant" layer, i.e. each layer that "performs some computation".
As such Flatten is excluded and I guess simular layers such as Reshape and Permute should be excluded as well if needed.
For dropout, I'm not sure if it should be considered, since it's just randomly dropping inputs during training, so I'm not sure that I can see the benefit of including it in the coverage. Intuitively, since it's only used for regularization in training I'm not sure if it should be included.

Finally, there seems to be a minor issue in the code w.r.t. coverage calculation. The last "layer" in the models is currently separated into a Dense and a Lambda layer. This last layer should probably be treated like all the other layers, where Dense and Lamba are "joined" and only the output joined layer is used for coverage. So, the final Dense layer should be excluded from coverage as well and only the Lambda one should be considered.

Why to calculate mean in constraint_black function？

In this function, only when the mean of the 10*10 gradient value bigger than zero, we will add them to the original image data.
I still haven't understood the reason for it.
Can you give me some advice?
Thank you~~

Reproducible results

In order to get reproducible results I would suggest to add seeds to the different gen_diff.py's, e.g.:

import numpy as np
import random

random.seed(1)
np.random.seed(1)

Getting Error message while running imageNet.

Hi, I am trying to run the ImageNet but am getting error message like Traceback (most Recent call last):
File "gen_diff.py" line 53, in

img_pahts = image.list_pictures('./seeds', ext ='JPEG')

Attribute Error: 'Module' object has no attribute 'list_pictures'

GoogleNet for Image Classification

Hi Pei,

There is an image classification task for identifying an image consisting of a Bus, Car, Van, Truck. This is implemented by using GoogleNet (Caffe) model. Now, in order to fool the model by identifying a Bus as Car, how can we use DeepXplore for this particular type of classification.

Thanks.

Parameters to reproduce results from the paper

Hi folks,

Thanks for sharing this code! It looks like the gen_diff.py script takes a large number of arguments (https://github.com/peikexin9/deepxplore/blob/master/MNIST/gen_diff.py#L21) many of which are not optional. Can you share the values you used to generate the results in the paper? And also, can you comment on what are reasonable values for each of these arguments?

Manasi

About saving the result picture

Hello,
I get some result pictures. But when I using the model to predict the result picture, it still get the right prediction, which should be wrong predicted.
I am wondering about the process of saving the difference-inducing result. Take MNIST as a example, after we get the tensor gen_img, which has caused the difference, to save gen_img as a picture, there are some operations on gen_img, like x *= 255, x = np.clip(x, 0, 255).astype('uint8'). These operations change the value of gen_img. Won't them influence the result of the model prediction?

occlusion dark spots

Hi,

could you help me to get the blackout dark spots on the generated images, in the way you used in the paper (Figure 8).

Here is what my images look alike:

can i get the shorter dark rectangles, could you reproduce the size that we have to use for occl_size with the arg.parse().
Thanks.