Deep-Unsupervised-Learning

Pytorch implementatiions of the Homeworks in course CS294-158

Here you can find PyTorch implementations of a Masked Autoencoder for Distribution Estimation, PixelCNN, RealNVP and other architectures that are used in course.

Homework 1

For the first homework there is a PyTorch implementation of PixelCNN-MADE

That is a auto-regressive model, which mixes ideas from the two papers Pixel Recurrent Neural Networks (2016) and MADE: Masked Autoencoder for Distribution Estimation (2015) to produce colored 28x28 MNIST digits. Pixel intensities have been quantized to 2 bits (i.e. four intensities for each color channel). The dataset can be downloaded from here.

Training data

I have used 60,000 quantized 28x28 images from the colored MNIST dataset. Here's a few samples from the training set:

Examples

After 50 epochs of training, a network consisting of 12 residual blocks - see Deep Residual Learning for Image Recognition (2015) - followed by a 3-layer MADE generates samples like the following:

You are very welcome to extend the code however you like. If you produce anything cool, be sure to let me know!

Homework 2

For the second homework we implement RealNVP coupling layers for modelling flows

Training data

Here I have used 5000 datapoints sampled from this distribution:

Example

After 500 epochs we can sample the following face

with this fancy latent space

and density plot

We also implement RealNVP(https://arxiv.org/abs/1605.08803) to achive these results:

Homework 3

We implement a VAE with a gated shortcur connection(https://arxiv.org/pdf/1612.08083) and train it on the SVHN dataset.

The final results look like this:

Homework 4

In the 4th homework we implement the Wasserstein GAN(https://arxiv.org/abs/1704.00028) and draw inspiration from the architecture used in SN-GAN(https://arxiv.org/abs/1802.05957)

This we for 80 K iteration and the training curves and mean inception score look like the following: