raghavchalapathy / oc-nn_old

Models for one class neural networks

Keras-Tensorflow Implementation of One Class Neural Networks.

This repository provides a Keras-Tensorflow implementation of the One Class Neural Network method presented in our paper ”Anomaly Detection using One Class Neural Networks”.

Citations and Contact.

You find a PDF of the One Class Neural Network paper at:

If you use our work, please also cite the paper:

 @inproceedings{chalapathy2017robust,
 title={Robust, deep and inductive anomaly detection},
 author={Chalapathy, Raghavendra and Menon, Aditya Krishna and Chawla, Sanjay},
 booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},
 pages={36--51},
 year={2017},
 organization={Springer}
}

You find a PDF of the Robust, Deep and Inductive Anomaly Detection paper at: https://arxiv.org/pdf/1704.06743.pdf

If you use our work, please also cite the paper:

 @inproceedings{chalapathy2017robust,
 title={Robust, deep and inductive anomaly detection},
 author={Chalapathy, Raghavendra and Menon, Aditya Krishna and Chawla, Sanjay},
 booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},
 pages={36--51},
 year={2017},
 organization={Springer}
}

If you would like to get in touch, please contact . [email protected]

Abstract

We propose a one-class neural network (OC-NN) model to detect anomalies in complex data sets. OC-NN combines the ability of deep networks to extract a progressively rich representation of data with the one-class objective of creating a tight envelope around normal data. The OC-NN approach breaks new ground for the following crucial reason: data representation in the hidden layer is driven by the OC-NN objective and is thus customized for anomaly detection. This is a departure from other approaches which use a hybrid approach of learning deep features using an autoencoder and then feeding the features into a separate anomaly detection method like one-class SVM (OC-SVM). The hybrid OC-SVM approach is sub-optimal because it is unable to influence representational learning in the hidden layers. A comprehensive set of experiments demonstrate that on complex data sets (like CIFAR and GTSRB), OC-NN performs on par with state-of-the-art methods and outperformed conventional shallow methods in some scenarios.

Installation

This code is written in Python 3.6.7 and runs on Google Colab environment

Clone the repository to your local machine and directory of choice and upload to your Google drive account and open the notebooks present in directory "notebooks" in order to execute reproduce the results.

PS:

Kindly make sure the PROJECT_DIR = "/content/drive/My Drive/2019/testing/oc-nn/" is set appropriately inside the notebook and in files src/models/RCAE.py, src/data/cifar10.py , src/data/mnist.py

Kindly make sure you configure the normal and anomalous class inside the src/config.py present within the Configuration class object for all the data sets and for any new data sets which would be added in future.

The data related to GTSRB experiments can be downloaded here. Please note that this data was shared to me by Lukas Ruff kindly request to acknowledge by citing his work as well.

Working with your own Data

Kindly customize the load_data function inside the CIFAR_10_DataLoader or MNIST_DataLoader class apppropriately to suite load experiment data. Also kindly RCAE.py and OneClass_SVDD.py to suite your training and test data needs.

Running experiments

We currently have implemented the MNIST (http://yann.lecun.com/exdb/mnist/) and CIFAR-10 (https://www.cs.toronto.edu/~kriz/cifar.html) datasets and simple LeNet-type networks.

MNIST Example

RCAE

%reload_ext autoreload
%autoreload 2
from src.models.RCAE import RCAE_AD
import numpy as np 
from src.config import Configuration as Cfg

DATASET = "mnist"
IMG_DIM= 784
IMG_HGT =28
IMG_WDT=28
IMG_CHANNEL=1
HIDDEN_LAYER_SIZE= 32
MODEL_SAVE_PATH = PROJECT_DIR + "/models/MNIST/RCAE/"
REPORT_SAVE_PATH = PROJECT_DIR + "/reports/figures/MNIST/RCAE/"
PRETRAINED_WT_PATH = ""
# RANDOM_SEED = [42,56,81,67,33,25,90,77,15,11]
RANDOM_SEED = [42]
AUC = []

for seed in RANDOM_SEED:  
  Cfg.seed = seed
  rcae = RCAE_AD(DATASET,IMG_DIM, HIDDEN_LAYER_SIZE, IMG_HGT, IMG_WDT,IMG_CHANNEL, MODEL_SAVE_PATH, REPORT_SAVE_PATH,PRETRAINED_WT_PATH,seed)
  print("Train Data Shape: ",rcae.data._X_train.shape)
  print("Train Label Shape: ",rcae.data._y_train.shape)
  print("Validation Data Shape: ",rcae.data._X_val.shape)
  print("Validation Label Shape: ",rcae.data._y_val.shape)
  print("Test Data Shape: ",rcae.data._X_test.shape)
  print("Test Label Shape: ",rcae.data._y_test.shape)
  print("===========TRAINING AND PREDICTING WITH DCAE============================")
  auc_roc = rcae.fit_and_predict()
  print("========================================================================")
  AUC.append(auc_roc)
  
print("===========TRAINING AND PREDICTING WITH DCAE============================")
print("AUROC computed ", AUC)
auc_roc_mean = np.mean(np.asarray(AUC))
auc_roc_std = np.std(np.asarray(AUC))
print ("AUROC =====", auc_roc_mean ,"+/-",auc_roc_std)
print("========================================================================")

OC_NN or Deep SVDD approach

%reload_ext autoreload
%autoreload 2

from src.models.OneClass_SVDD import OneClass_SVDD
from src.models.config import Configuration as Cfg

DATASET = "mnist"
IMG_DIM= 784
IMG_HGT =28
IMG_WDT=28
IMG_CHANNEL=1
HIDDEN_LAYER_SIZE= 32
MODEL_SAVE_PATH = PROJECT_DIR + "/models/MNIST/OC_NN/"
REPORT_SAVE_PATH = PROJECT_DIR + "/reports/figures/MNIST/OC_NN/"
PRETRAINED_WT_PATH = ""
#LOSS_FUNCTION = "SOFT_BOUND_DEEP_SVDD"
# LOSS_FUNCTION = "ONE_CLASS_DEEP_SVDD"
LOSS_FUNCTION = "ONE_CLASS_NEURAL_NETWORK"

import os
os.chdir(PROJECT_DIR)
# RANDOM_SEED = [42,56,81,67,33,25,90,77,15,11]
RANDOM_SEED = [42]
AUC = []

for seed in RANDOM_SEED:  
  Cfg.seed = seed
  ocnn = OneClass_SVDD(DATASET,LOSS_FUNCTION,IMG_DIM, HIDDEN_LAYER_SIZE, IMG_HGT, IMG_WDT,IMG_CHANNEL, MODEL_SAVE_PATH, REPORT_SAVE_PATH,PRETRAINED_WT_PATH,seed)
 
  print("[INFO:] Testing with ALL other  DIGITs  as anomalies")
  ocnn.fit()
  print("==============PREDICTING THE LABELS ==============================")
  auc_score = ocnn.predict()
  AUC.append(auc_score)

print("===========TRAINING AND PREDICTING WITH OCSVDD============================")
print("AUROC computed ", AUC)
auc_roc_mean = np.mean(np.asarray(AUC))
auc_roc_std = np.std(np.asarray(AUC))
print ("AUROC =====", auc_roc_mean ,"+/-",auc_roc_std)
print("========================================================================")

CIFAR-10 Example

RCAE

%reload_ext autoreload
%autoreload 2
from src.models.RCAE import RCAE_AD
import numpy as np 
from src.config import Configuration as Cfg

DATASET = "cifar10"
IMG_DIM= 3072
IMG_HGT =32
IMG_WDT=32
IMG_CHANNEL=3
HIDDEN_LAYER_SIZE= 128
MODEL_SAVE_PATH = PROJECT_DIR + "/models/cifar10/RCAE/"
REPORT_SAVE_PATH = PROJECT_DIR + "/reports/figures/cifar10/RCAE/"

PRETRAINED_WT_PATH = ""
# RANDOM_SEED = [42,56,81,67,33,25,90,77,15,11]
RANDOM_SEED = [42]
AUC = []

for seed in RANDOM_SEED:  
  Cfg.seed = seed
  rcae = RCAE_AD(DATASET,IMG_DIM, HIDDEN_LAYER_SIZE, IMG_HGT, IMG_WDT,IMG_CHANNEL, MODEL_SAVE_PATH, REPORT_SAVE_PATH,PRETRAINED_WT_PATH,seed)
  print("Train Data Shape: ",rcae.data._X_train.shape)
  print("Train Label Shape: ",rcae.data._y_train.shape)
  print("Validation Data Shape: ",rcae.data._X_val.shape)
  print("Validation Label Shape: ",rcae.data._y_val.shape)
  print("Test Data Shape: ",rcae.data._X_test.shape)
  print("Test Label Shape: ",rcae.data._y_test.shape)
  print("===========TRAINING AND PREDICTING WITH DCAE============================")
  auc_roc = rcae.fit_and_predict()
  print("========================================================================")
  AUC.append(auc_roc)
  
print("===========TRAINING AND PREDICTING WITH DCAE============================")
print("AUROC computed ", AUC)
auc_roc_mean = np.mean(np.asarray(AUC))
auc_roc_std = np.std(np.asarray(AUC))
print ("AUROC =====", auc_roc_mean ,"+/-",auc_roc_std)
print("========================================================================")

OC_NN or Deep SVDD approach

%reload_ext autoreload
%autoreload 2

from src.models.OneClass_SVDD import OneClass_SVDD
from src.models.config import Configuration as Cfg

DATASET = "cifar10"
IMG_DIM= 3072
IMG_HGT =32
IMG_WDT=32
IMG_CHANNEL=3
HIDDEN_LAYER_SIZE= 128
MODEL_SAVE_PATH = PROJECT_DIR + "/models/cifar10/OC_NN/"
REPORT_SAVE_PATH = PROJECT_DIR + "/reports/figures/cifar10/OC_NN/DeepSVDD/"
PRETRAINED_WT_PATH = ""
#LOSS_FUNCTION = "SOFT_BOUND_DEEP_SVDD"
# LOSS_FUNCTION = "ONE_CLASS_DEEP_SVDD"
LOSS_FUNCTION = "ONE_CLASS_NEURAL_NETWORK"

import os
os.chdir(PROJECT_DIR)
# RANDOM_SEED = [42,56,81,67,33,25,90,77,15,11]
RANDOM_SEED = [42]
AUC = []

for seed in RANDOM_SEED:  
  Cfg.seed = seed
  ocnn = OneClass_SVDD(DATASET,LOSS_FUNCTION,IMG_DIM, HIDDEN_LAYER_SIZE, IMG_HGT, IMG_WDT,IMG_CHANNEL, MODEL_SAVE_PATH, REPORT_SAVE_PATH,PRETRAINED_WT_PATH,seed)
 
  print("[INFO:] Testing with ALL other  Image Classes  as anomalies")
  ocnn.fit()
  print("==============PREDICTING THE LABELS ==============================")
  auc_score = ocnn.predict()
  AUC.append(auc_score)

print("===========TRAINING AND PREDICTING WITH OCSVDD============================")
print("AUROC computed ", AUC)
auc_roc_mean = np.mean(np.asarray(AUC))
auc_roc_std = np.std(np.asarray(AUC))
print ("AUROC =====", auc_roc_mean ,"+/-",auc_roc_std)
print("========================================================================")

Sample Results

MNIST

Example of the most normal (left) and most anomalous (right) test set examples per class on MNIST according to One Class Neural Networks and Robust Convolution Autoencoder (RCAE) anomaly scores.

CIFAR-10

Example of the most normal (left) and most anomalous (right) test set examples per class on CIFAR-10 according to One Class Neural Networks and Robust Convolution Autoencoder (RCAE) anomaly scores.

License

MIT

Disclosure

This implementation is based on the repository https://github.com/lukasruff/Deep-SVDD, which is licensed under the MIT license. The Deep SVDD repository is an implementation of the paper "Deep One-Class Classification by Ruff, Lukas and Vandermeulen, Robert A. and G{"o}rnitz, Nico and Deecke, Lucas and Siddiqui, Shoaib A. and Binder, Alexander and M{"u}ller, Emmanuel and Kloft, Marius

oc-nn_old's Issues

Problem with internal variables

while running the code, I got the following error:

---

ValueError: Variable r already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

File "tf_OneClass_CNN_model.py", line 45, in
r = tf.get_variable(name="r", dtype=tf.float32, shape=(), trainable=False)
File "sklearn_OCSVM_model.py", line 8, in
from tf_OneClass_CNN_model import func_get_ImageVectors
File "synthetic_models.py", line 1, in
#from sklearn_OCSVM_model import sklearn_OCSVM_linear,sklearn_OCSVM_rbf

---

The problem is that I am getting it while running both sections "Experiments: PFAM datasets" and "Experiments: Synthetic data"

Did anyone face the same problem? Do you, please, have any suggestions to fix the problem?
I am waiting forward to hearing from you

Code of trained models.

Dear Raghav:

Where is the code to obtain trained models, like "mnist_AE2_2_id_32_e_1000_encoder.model"?

Thanks, in advance.

loss is negative

I am training the model and the loss is negative number. Is it ok ?

oneClassNN

I tried to run the jupyter notebook and encountered this error in tflearn_OneClass_NN_model.py:

----> 8 from tflearn.layers.estimator import regression, oneClassNN
....
ImportError: cannot import name 'oneClassNN'

Indeed, I do not find this layer anywhere in tflearn (or searching for it via google for that matter). Where can I obtain the code for oneClassNN?

Data preparation

From your code it is not at all clear how you prepare the cifar-10 data for the modeling. For example,
in your comment of the code shown bellow, there is a comment "extract the class label from the image path and update the labels list". However, there is no explanation how you save the cifer-10 data into "image paths" in order to differentiate the labels. Moreover, for the training of the OC-NN model, only images with dogs should be used. However, from your code I don't see anywhere where you split the data into dogs and cats, and then save it into some "image path" to use it later.
Could you please clarify the exact steps how the cifar-10 data are prepared and saved to different "image paths" according to different labels? Going through load_dataset.py, none of the given functions doesn't do this.

For making auto-encoder, are all images in cifar-10 used, or only images with both dogs and cats, or only images with dogs? Btw, there is no code for making the auto-encoder model.

I find the code here very incomplete and I am very frustrated unsuccessfully trying to reproduce the results in the paper. Has anyone here succeeded to reproduced the claimed results?

imagePaths = sorted(list(paths.list_images(train_path)))
    # loop over the input training images
    image_dict = {}
    test_image_dict = {}
    i = 0
    for imagePath in imagePaths:
        # load the image, pre-process it, and store it in the data list
        image = cv2.imread(imagePath)
        image = cv2.resize(image, (side, side))
        image = img_to_array(image)
        data.append(image)
        image_dict.update({i: imagePath})
        i = i + 1

        # extract the class label from the image path and update the
        # labels list
        label = imagePath.split(os.path.sep)[-2]
        label = 1 if label == "dogs" else 0
        labels.append(label)
    # scale the raw pixel intensities to the range [0, 1]
    data = np.array(data)

y = keras.utils.to_categorical(labels, num_classes=2)

Question on CIFAR-10 results

I'm a little confused by the results published in your paper vs. the experiments you ran at KDD18. In the paper it seems that the OC-NN achieves essentially a perfect score on the unsupervised task of learning what a dog in CIFAR-10 looks like and telling it apart from a cat in CIFAR-10.

This is truly remarkable on it's own because current state-of-the-art CNNs that are a lot more complex than the proposed architecture don't reach this level of accuracy even if they have been trained on much larger datasets, much larger images and making it a binary classification which is the much easier task to solve.

In the Jupyter notebook from KDD18 the results are entirely different than the results reported in the paper and actually making the oc-nn the worst performing method tested.

Could you share your thoughts on this? Am I misunderstanding something?

Thanks!

customLoss error

I run the code ,but error. It is from keras.losses import customLoss, ImportError: cannot import name 'customLoss', what is it?

Encoder model architecture not as described in the paper

I want to reproduce results for cifar-10 data. I have downloaded the encoder model from the given link, cifar10_conv_3_id_32_e_1000_encoder.model. However, this model doesn't agree with the architecture that is described in the paper. In the paper it is said that the first and second layer have 16 filters, and the third and fourth layers have 32 layers each. According to the summary this is clearly not the case. Please find bellow the summary of cifar10_conv_3_id_32_e_1000_encoder.model.

However, I am even more confused with the flattening and dense layer at the very end of the encoder model. This was not mentioned anywhere in the paper that the convolutional autoencoder contains such layers.

If I am misunderstanding something, could you please point me in the right direction?

---

Layer (type) Output Shape Param #

input_1 (InputLayer) (None, 32, 32, 3) 0

---

conv2d_1 (Conv2D) (None, 32, 32, 32) 896

---

max_pooling2d_1 (MaxPooling2 (None, 16, 16, 32) 0

---

conv2d_2 (Conv2D) (None, 16, 16, 64) 18496

---

max_pooling2d_2 (MaxPooling2 (None, 8, 8, 64) 0

---

conv2d_3 (Conv2D) (None, 8, 8, 128) 73856

---

max_pooling2d_3 (MaxPooling2 (None, 4, 4, 128) 0

---

conv2d_4 (Conv2D) (None, 4, 4, 256) 295168

---

max_pooling2d_4 (MaxPooling2 (None, 2, 2, 256) 0

---

flatten_1 (Flatten) (None, 1024) 0

---

dense_1 (Dense) (None, 32) 32800

Total params: 421,216
Trainable params: 0
Non-trainable params: 421,216