janhenriklambrechts / one-sentence-ml-papers

A collection of awesome papers in Deep Learning, with a one sentence summary.

Awesome Deep Learning Papers 📰

Feel free to browse my collection of summaries of various Deep Learning Topics. Main difference with other repos that have summaries of DL papers is that my README includes one sentence summaries of every paper discussed. I have found this incredibly helpful to remember a paper or assess if the paper is worth reading (again).

Ordered by topic 🗂️:

• GANs 🖼️
• Style Transfer 🎨
• Data Augmentation ✨
• Robustness 🛡️
• Deploying ML ☁️
• Continual Learning / Tranfer Learning ♻️
• Dynamic Inference 🐙
• Vision Architectures 👁️
• Deep Reinforcement Learning 🤖
• Neural Architecture Search 🔍

GANs 🖼️

Title	Authors	In One Sentence	Summary	Date	Link	Conference
Training Generative Adversarial Networks with Limited Data	Tero Karras et al.	Proposes a adaptive discriminator augmentation mechanism that significantly stabilizes limited data regimes	Summary	01/12/2020	Paper	NeurIPS 2020
Few-short Image Generation with Elastic Weight Consolidation	Yijun Li et al.	Proposes an algorithm that generates high-quality samples of different target domains	Summary	01/12/2020	Paper	NeurIPS 2020
A Style-Based Generator Architecture for Generative Adversarial Networks	Tero Karras et al.	Introduces a novel generator architecture based on style transfer, leading to better quality and control of generation	Summary	07/11/2016	Paper	CVPR 2019
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks	Alec Radford et al.	When we add some architectural constraints to GANs they provide good image representation that can be used for supervised learning.	Summary	07/11/2016	Paper	ICLR 2016

Style Transfer 🎨

Title	Authors	In One Sentence	Summary	Date	Link	Conference
Adaptive Style Transfer in Real-Time with Adaptive Instance Normalization	Xun Huang et al.	Introduces an AdaIN layer that allows for arbitraty style transfer in real-time	Summary	30/07/2017	Paper	ICCV 2017
Image Style Transfer Using Convolutional Neural Networks	Leon Gatys et al.	Introduces an algorithm that can separate and recombine style of an image and image content	Summary	07/11/2016	Paper	CVPR 2016

Data Augmentation ✨

Title	Authors	In One Sentence	Summary	Date	Link	Conference
Fast AutoAugment	Sungbin Lim et al.	Density matching can speed up AutoAugment search time while achieving comparable performance	Summary	25/05/2019	Paper	NeurIPS 2019
RandAugment: Practical Automated Data Augmentation with a Reduced Search Space	Ekin D. Cubuk, Barret Zoph et al.	Searching for a single distortion magnitude can do as well or improve previous automated augmentation strategies	Summary	11/04/2019	Paper	NeurIPS 2020
AutoAugment: Learning Augmentation Strategies from Data	Ekin D. Cubuk et al.	Meta-learning data augmentation for new SOTA on classification	Summary	11/04/2019	Paper	ArXiv

Robustness 🛡️

Title	Authors	In One Sentence	Summary	Date	Link	Conference
Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation	Raphel Gontijo Lopes et al.	Combining Cutout and Gaussian noise in a single data augmentation scheme improves robustness while not incurring accuracy drop.	Summary	06/06/2019	Paper	ICML 2019 Workshop
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations	Dan Hendrycks et al.	Establishes a benchmark for image classifier robustness to common corruptions.	Summary	25/05/2019	Paper	ICLR 2019

Deploying ML ☁️

Title	Authors	In One Sentence	Summary	Date	Link	Conference
Machine Learning: The High-Interest Credit Card of Technical Debt	D. Sculley et al.	Highlights risk factors and design patterns to be avoided in building ML systems	Summary	18/11/2014	Paper	NeurIPS 2014 Workshop
Challenges in Deploying ML: A Survey of Case Studies	Andrei Paleyes et al.	A survey of deploying ML solutions for different use cases	Summary	18/11/2020	Paper	NeurIPS 2020 Workshop

Continual Learning / Transfer Learning ♻️

Title	Authors	In One Sentence	Summary	Date	Link	Conference
A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks	Eden Belouadah et al.	Provides an evaluation framework for Incremental Learning Algos across visual tasks	Summary	15/12/2020	Paper	Neural Networks
Three scenarios for continual learning	Gido van de Ven et al.	Establishes and classifies earlier continual learning scenarios	Summary	15/04/2019	Paper	Arxiv
Few-Shot Class Incremental Learning	Xiaoyu Tao et al.	Representing knowledge using a network can learn and preserve the topology of feature manifold formed by different classes	Summary	24/04/2020	Paper	CVPR 2020
Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting	Sayna Ebrahimi et al.	Use saliency maps for experience replay to reduce catastrophic forgetting even more	Summary	25/01/2021	Paper	ICLR 2021
Mutual Alignment Transfer Learning	Markus Wulfmeister et al.	Demonstrates that auxilary rewards can bridge the gap between simulation and real-world robots	Summary	26/09/2017	Paper	CoRL 2017
NetTailor: Tuning the Architecture, Not just the Weights	Pedro Morgado et al.	Tuning the architecture, not just the weights	Summary	29/06/2019	Paper	CVPR 2019
Progressive Neural Networks	Andrei A. Rusu, Neil C. Rabinowitz et al.	Grow lateral connections to features learned by frozen weights	Summary	01/01/2016	Paper	Arxiv
Compacting, Picking and Growing for Unforgetting Continual Learning	Steven C.Y. Hung et al.	Leveraging Pruning and Progressive Networks Expansion can deal with catastrophic forgetting while retaining model compactness.	Summary	18/06/2019	Paper	NeurIPS 2019
PathNet: Evolution Channels Gradient Descent in Super Neural Networks	Chrisantha Fernando et al.	Genetic Algorithm that replicates and mutates earlier paths for faster transfer learning	Summary	19/11/2019	Paper	ArXiv
Contextual Transformation Networks for Online Continual Learning	Quang Pham et al.	Online Continual Learning method to model task-specific features for fixed architecture methods	Summary	25/01/2021	Paper	ICLR 2021
Overcoming catastrophic forgetting in neural networks	James Kirkpatrick et al.	Slowing down learning on weights important for earlier tasks overcomes catastrophic forgetting	Summary	25/01/2017	Paper	Arxiv
Continual Learning with Hypernetworks	Johannes von Oswald et al.	To overcome gigantic network growth for new tasks, we build task-conditioned hypernetworks	Summary	12/02/2020	Paper	ICLR 2020

Dynamic Inference 🐙

Title	Authors	In One Sentence	Summary	Date	Link	Conference
Universally Slimmable Networks and Improved Training Techniques	Jiahui Yu et al.	Makes Slimmable networks able to run at arbitrary widths	Summary	Paper	ICCV 2019
Anytime Inference with Distilled Hierarchical Neural Ensembles	Andria Ruiz et al.	Dynamically allocating multiple models in the ensemble and training them with distillation provides good dynamic inference	Summary	14/12/2020	Paper	AAAI 2020

Vision Architectures 👁️

Title	Authors	In One Sentence	Summary	Date	Link	Conference
GhostNet: More Features from Cheap Operations	Kai Han et al.	Transforming little feature maps with simple linear operations to many feature maps works very well	Summary	13/03/2020	Paper	ArXiv
RepVGG: Making VGG-style ConvNets Great Again	Xiaohan Ding et al.	Decoupling of training-time and inference-time architecture can lead to inference speedup of 83% over ResNet-50 while reaching 80+% on ImageNet	Summary	11/01/2021	Paper	ArXiv
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark	Xiaohua Zhai et al.	A unified evaluation for general visual representations by combining a bunch of vision tasks into one	Summary	21/02/2020	Paper	ArXiv

Deep Reinforcement Learning 🤖

Title	Authors	In One Sentence	Summary	Date	Link	Conference
Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model	Tree search and learned model can master unknown and complex environments	Summary	19/11/2019	Paper	ArXiv

Neural Architecture Search

Title	Authors	In One Sentence	Summary	Date	Link	Conference
AutoSlim: Towards One-Shot Architecture Search for Channel Numbers	Train a single slimmable network and then greedily slim the layer with least accuracy drop for one-shot NAS	Summary	01/06/2019	Paper	ArXiv