hsahuja111 / reconstructing-training-data-from-trained-neural-network

This repository has research paper implementation which reconstructs training data.

Reconstructing-Training-Data-From-Trained-Neural-Network

This repository contains implementation of a research paper - Reconstruction Training Data from Trained Neural Network. I have tried to explain this paper below but to go through each theoritical proof you can check the original paper which is present in this repository named Research-Paper.pdf.

Lets firstly understand the basic idea of this paper

Overview of the paper

Usually, what we do is train a Neural Network using some training data and then leverage it to predict the output for some unknown data. But, this paper has done some great work to extract the training data back from the trained model.

Lets understand this more concisely,

Input to us is a Trained MLP model. By trained MLP model we actually mean that we have the weights/parameters saved with us. Now ,just by using this we have to generate the training data used while training our model.

Maths Behind

In the actual training process of a Neural Network we apply gradient descent on the Loss function we got. This loss function is actually the difference between the predicted and original. We keep on iterating till the time we got minimum error.

But here, What will be our Loss functions??

The intuition of it lies in the fact that every MLP model although seems to be generic but often has some bias. In case of Binary MLP model trained with Gradient descent, it converges to the same solution as given by SVM margin maximising. Hence from the equation given by SVM Margin Maximing problem we have derived 2 loss functions for our problem. The 3rd Loss function is based upon generic thinking.

So this is the SVM equation:

So here there are two

3 Loss functions used

1.Parameter Loss

2.Lambda Loss

3.Range Loss

Say, our training data were gray scale image so the pixel range will lies from -1 to 1. So we are penalising if we our resultant images goes outside of this range.

Suppose these Loss functions are L1,L2,L3 respectively.Total Loss will be weighted sum of L1,L2,L3.

Datasets

1. TOY DATASET

We have initially tried on the toy dataset. The process involved selecting 10 fully red images and 10 fully blue images to train a Binary MLP model. Subsequently, 100 randomly initialized images were obtained, and the objective now is to regenerate the original images, half of which should be red and the other half blue.

2. MNIST DATASET A pre-trained Binary MLP model was selected, which was trained using a dataset of 500 samples, comprising 250 samples of odd digits and the remaining 250 of even digits. The aim now is to generate these images back using a new set of 1000 samples.

Process Involved and Results

INPUT --> Parameters of Trained Model , Randomly initialised input image , parameters , lambda

Firstly, we have created a Binary MLP model and loaded the weights of the trained Binary MLP Model. You can find .pth file in this repository which has parameters(weights and biases).

CRUX

"Now, in the traditional process we apply gradient descent and tries to find the optimal value of weights and biases which would minimise the loss function. But here we have to find the optimal value of the pixels which would minimise the 3 loss functions explained above."