Pix2Pix: Denoising and Super Resolution (x4)

Overall about Pix2Pix

Image-to-Image Translation with Conditional Adversarial Networks

Link paper: Image-to-Image Translation with Conditional Adversarial Networks

Pix2Pix GAN paper was published back in 2016 by Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros. It was later revised in 2018. Pix2Pix is a deep learning model that aims to learn a mapping between an input image and an output image using a conditional generative adversarial network (cGAN)

Conditional GAN

The above shows an example of training a conditional GAN to map edges→photo.
The discriminator, D, learns to classify between fake (synthesized by the generator) and real {edge, photo} tuples.
The generator, G, learns to fool the discriminator.
Unlike an unconditional GAN, both the generator and discriminator observe the input edge map.

That is, conditional GANs learn a mapping from observed image x and random noise vector z, to y.

Conditional-Adversarial Loss

L1 Loss

Final Loss

Pix2Pix: Network Architectures

Both generator and discriminator use modules of the form convolution-BatchNorm-ReLu.

PatchGAN

Applications

There are many applications of a pix2pix network. These include the following:

To convert pixel level segmentation into real images
To convert day images into night images and vice versa
To convert satellite areal images into map images
To convert sketches into photos
To convert black and white images into colored images and vice versa

Limitations

Pix2Pix is a supervised learning algorithm. This means that it requires a dataset of paired examples. This is a major limitation as it is not always possible to obtain paired examples.