artix41 / awesome-transfer-learning

Best transfer learning and domain adaptation resources (papers, tutorials, datasets, etc.)

Awesome Transfer Learning

A list of awesome papers and cool resources on transfer learning, domain adaptation and domain-to-domain translation in general! As you will notice, this list is currently mostly focused on domain adaptation (DA) and domain-to-domain translation, but don't hesitate to suggest resources in other subfields of transfer learning.

Note: this list is not actively maintained anymore, but I still accept pull requests, so please don't hesitate if you want to contribute with newer resources

Table of Contents

• Tutorial and Blogs
• Papers
  • Surveys
  • Deep Transfer Learning
    • Fine-tuning approach
    • Feature extraction (embedding) approach
    • Policy transfer for RL
    • Few-shot transfer learning
    • Meta transfer learning
    • Applications
  • Unsupervised Domain Adaptation
    • Theory
    • Adversarial methods
    • Optimal Transport
    • Embedding methods
    • Kernel methods
    • Autoencoder approach
    • Subspace learning
    • Self-ensembling methods
    • Other
  • Semi-supervised Domain Adaptation
    • General methods
    • Subspace learning
    • Copulas methods
  • Few-shot Supervised Domain Adaptation
    • Adversarial methods
    • Embedding methods
  • Applied Domain Adaptation
    • Physics
• Datasets
  • Image-to-image
  • Text-to-text
  • Other
• Results
  • Digits transfer
• Challenges
• Libraries
• Books

Tutorials and Blogs

• Transfer Learning − Machine Learning's Next Frontier
• A Little Review of Domain Adaptation in 2017
• A Comprehensive Hands-on Guide to Transfer Learning with Real-World Applications in Deep Learning

Papers

Papers are ordered by theme and inside each theme by publication date (submission date for arXiv papers). If the network or algorithm is given a name in a paper, this one is written in bold before the paper's name.

Surveys

• A Survey on Transfer Learning (2009)
• Transfer Learning for Reinforcement Learning Domains: A Survey (2009)
• A Survey of transfer learning (2016)
• Domain Adaptation for Visual Applications: A Comprehensive Survey (2017)
• Deep Visual Domain Adaptation: A Survey (2018)

Deep Transfer Learning

Transfer of deep learning models.

Fine-tuning approach

• Do Better ImageNet Models Transfer Better? (2018)
• Using Pre-Training Can Improve Model Robustness and Uncertainty (2019)
• Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation (2020)
• Regularizing CNN Transfer Learning with Randomised Regression (2020)

Feature extraction (embedding) approach

• CNN Features off-the-shelf: an Astounding Baseline for Recognition (2014)
• Taskonomy: Disentangling Task Transfer Learning (2018)

Multi-task learning

• Learning without forgetting (2016)

Policy transfer for RL

• Reinforcement Learning to Train Ms. Pac-Man Using Higher-order Action-relative Inputs (2013)

Few-shot transfer learning

• Zero-Shot Transfer Learning for Event Extraction (2017)
• Learning a Deep Embedding Model for Zero-Shot Learning (2017)
• Zero-Shot Object Detection (2018)
• LSTD: A Low-Shot Transfer Detector for Object Detection (2018)
• Multidomain Document Layout Understanding using Few Shot Object Detection (2018)
• One-Shot Unsupervised Cross Domain Translation (2018)

Meta transfer learning

• Transfer Learning via Learning to Transfer (2018)

Applications

Medical imaging:

• Deep Convolutional Neural Networks forComputer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning (2016)
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? (2017)
• Comparison of deep transfer learning strategies for digital pathology (2018)

Robotics

• A Deep Convolutional Neural Network for Location Recognition and Geometry Based Information (2018)

Anomaly Detection

• Multiresolution Knowledge Distillation for Anomaly Detection (2020)

Unsupervised Domain Adaptation

Transfer between a source and a target domain. In unsupervised domain adaptation, only the source domain can have labels.

Theory

General

• A theory of learning from different domains (2010)
• The Role of Minimal Complexity Functions in Unsupervised Learning of Semantic Mappings (2018)
• A Theory of Label Propagation for Subpopulation Shift (2021)

Multi-source

• Domain Adaptation with Multiple Sources (2008)
• Algorithms and Theory for Multiple-Source Adaptation (2018)

Adversarial methods

Learning a latent space

• DANN: Domain-Adversarial Training of Neural Networks (2015)
• JAN: Deep Transfer Learning with Joint Adaptation Networks (2016)
• CoGAN: Coupled Generative Adversarial Networks (2016)
• DRCN: Deep Reconstruction-Classification Networks for Unsupervised Domain Adaptation (2016)
• DSN: Domain Separation Networks (2016)
• ADDA: Adaptative Discriminative Domain Adaptation (2017)
• GenToAdapt: Generate To Adapt: Aligning Domains using Generative Adversarial Networks (2017)
• WDGRL: Wasserstein Distance Guided Representation Learning for Domain Adaptation (2017)
• CyCADA: CyCADA: Cycle-Consistent Adversarial Domain Adaptation (2017)
• DIRT-T: A DIRT-T Approach to Unsupervised Domain Adaptation (2017)
• DupGAN: Duplex Generative Adversarial Network for Unsupervised Domain Adaptation (2018)
• MSTN: Learning Semantic Representations for Unsupervised Domain Adaptation (2018)
• Emerging Disentanglement in Auto-Encoder Based Unsupervised Image Content Transfer (2019)
• DTA: Drop to Adapt: Learning Discriminative Features for Unsupervised Domain Adaptation (2019)
• iDANN: Incremental Unsupervised Domain-Adversarial Training of Neural Networks (2020)
• Cross-stained Segmentation from Renal Biopsy Images Using Multi-level Adversarial Learning (2020)

Image-to-Image translation

• DIAT: Deep Identity-aware Transfer of Facial Attributes (2016)
• Pix2pix: Image-to-Image Translation with Conditional Adversarial Networks (2016)
• DTN: Unsupervised Cross-domain Image Generation (2016)
• SimGAN: Learning from Simulated and Unsupervised Images through Adversarial Training (2016) (2016)
• PixelDA: Unsupervised Pixel–Level Domain Adaptation with Generative Adversarial Networks (2016)
• UNIT: Unsupervised Image-to-Image Translation Networks (2017)
• CycleGAN: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (2017)
• DiscoGAN: Learning to Discover Cross-Domain Relations with Generative Adversarial Networks (2017)
• DualGAN: DualGAN: Unsupervised Dual Learning for Image-to-Image Translation (2017)
• SBADA-GAN: From source to target and back: symmetric bi-directional adaptive GAN (2017)
• DistanceGAN: One-Sided Unsupervised Domain Mapping (2017)
• pix2pixHD: High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2018)
• I2I: Image to Image Translation for Domain Adaptation (2017)
• MUNIT: Multimodal Unsupervised Image-to-Image Translation (2018)
• LSTNet: Unsupervised Latent Space Translation Network(2020)

Multi-source adaptation

• StarGAN: StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation (2017)
• XGAN: XGAN: Unsupervised Image-to-Image Translation for Many-to-Many Mappings (2017)
• BicycleGAN : Toward Multimodal Image-to-Image Translation (2017)
• Label Efficient Learning of Transferable Representations across Domains and Tasks (2017)
• ComboGAN: ComboGAN: Unrestrained Scalability for Image Domain Translation (2017)
• AugCGAN: Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data (2018)
• RadialGAN: RadialGAN: Leveraging multiple datasets to improve target-specific predictive models using Generative Adversarial Networks (2018)
• MADA: Multi-Adversarial Domain Adaptation (2018)
• MDAN: Multiple Source Domain Adaptation with Adversarial Learning (2018)

Temporal models (videos)

• Model F: Unsupervised Domain Adaptation for Face Recognition in Unlabeled Videos (2017)
• RecycleGAN: Recycle-GAN: Unsupervised Video Retargeting (2018)
• Vid2vid: Video-to-Video Synthesis (2018)
• Temporal Smoothing (TS): Everybody Dance Now (2018)
• TA³N: Temporal Attentive Alignment for Large-Scale Video Domain Adaptation (2019)

Optimal Transport

• OT: Optimal Transport for Domain Adaptation (2015)
• Theoretical Analysis of Domain Adaptation with Optimal Transport (2016)
• JDOT: Joint distribution optimal transportation for domain adaptation (2017)
• Monge map learning: Large Scale Optimal Transport and Mapping Estimation (2017)
• JCPOT: Optimal Transport for Multi-source Domain Adaptation under Target Shift (2018)
• DeepJDOT: DeepJDOT: Deep Joint distribution optimal transport for unsupervised domain adaptation (2018)

Embedding methods

• Unsupervised Domain Adaptation for Zero-Shot Learning (2015)
• DA_assoc : Associative Domain Adaptation (2017)

Kernel methods

• SurK: Covariate Shift in Hilbert Space: A Solution via Surrogate Kernels (2015)
• DAN: Learning Transferable Features with Deep Adaptation Networks (2015)
• RTN: Unsupervised Domain Adaptation with Residual Transfer Networks (2016)
• Easy DA: A Simple Approach for Unsupervised Domain Adaptation (2016)

Autoencoder approach

• MCAE: Learning Classifiers from Synthetic Data Using a Multichannel Autoencoder (2015)
• SMCAE: Learning from Synthetic Data Using a Stacked Multichannel Autoencoder (2015)

Subspace Learning

• SGF: Domain Adaptation for Object Recognition: An Unsupervised Approach (2011)
• GFK: Geodesic Flow Kernel for Unsupervised Domain Adaptation (2012)
• SA: Unsupervised Visual Domain Adaptation Using Subspace Alignment (2015)
• CORAL: Return of Frustratingly Easy Domain Adaptation (2015)
• Deep CORAL: Deep CORAL: Correlation Alignment for Deep Domain Adaptation (2016)
• ILS: Learning an Invariant Hilbert Space for Domain Adaptation (2016)
• Log D-CORAL: Correlation Alignment by Riemannian Metric for Domain Adaptation (2017)
• GCA: A Unified Framework for Domain Adaptation using Metric Learning on Manifolds (2018)

Self-Ensembling methods

• MT: Self-ensembling for domain adaptation (2017)

Other

• Adapting Visual Category Models to New Domains (2010)
• AdaBN: Revisiting Batch Normalization for Practical Domain Adaptation (2016)
• AutoDIAL: Automatic Domain Alignment Layers (2017)
• Fully Test-time Adaptation by Entropy Minimization (2020)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift (2020)
• Improving robustness against common corruptions by covariate shift adaptation (2020)
• IAST: Instance Adaptive Self-Training for Unsupervised Domain Adaptation (2020)
• Meta Self-Learning: Meta Self-Learning for Multi-Source Domain Adaptation: A Benchmark (2021)

Semi-supervised Domain Adaptation

All the source points are labelled, but only few target points are.

General methods

• da+lap-sim : Semi-Supervised Domain Adaptation with Instance Constraints (2013)

Subspace learning

• EA++: Co-regularization Based Semi-supervised Domain Adaptation (2010)
• SDASL: Semi-supervised Domain Adaptation with Subspace Learning for Visual Recognition (2015)

Copulas methods

• NPRV: Semi-Supervised Domain Adaptation with Non-Parametric Copulas (2013)

Few-shot Supervised Domain Adaptation

Only a few target examples are available, but they are labelled

Adversarial methods

• FADA: Few-Shot Adversarial Domain Adaptation (2017)
• Augmented-Cyc: Augmented Cyclic Adversarial Learning for Domain Adaptation (2018)

Embedding methods

• CCSA: Unified Deep Supervised Domain Adaptation and Generalization (2017)

Applied Domain Adaptation

Domain adaptation applied to other fields

Physics

• Learning to Pivot with Adversarial Networks (2016)
• Identifying Quantum Phase Transitions with Adversarial Neural Networks (2017)
• Automated discovery of characteristic features of phase transitions in many-body localization (2017)

Audio Processing

• Autoencoder-based Unsupervised Domain Adaptation for Speech Emotion Recognition (2014)
• Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation (2018)
• Semi-Supervised Monaural Singing Voice Separation With a Masking Network Trained on Synthetic Mixtures (2019)

Datasets

Image-to-image

• MNIST vs MNIST-M vs SVHN vs Synth vs USPS: digit images
• GTSRB vs Syn Signs : traffic sign recognition datasets, transfer between real and synthetic signs.
• NYU Depth Dataset V2: labeled paired images taken with two different cameras (normal and depth)
• CelebA: faces of celebrities, offering the possibility to perform gender or hair color translation for instance
• Office-Caltech dataset: images of office objects from 10 common categories shared by the Office-31 and Caltech-256 datasets. There are in total four domains: Amazon, Webcam, DSLR and Caltech.
• Cityscapes dataset: street scene photos (source) and their annoted version (target)
• UnityEyes vs MPIIGaze: simulated vs real gaze images (eyes)
• CycleGAN datasets: horse2zebra, apple2orange, cezanne2photo, monet2photo, ukiyoe2photo, vangogh2photo, summer2winter
• pix2pix dataset: edges2handbags, edges2shoes, facade, maps
• RaFD: facial images with 8 different emotions (anger, disgust, fear, happiness, sadness, surprise, contempt, and neutral). You can transfer a face from one emotion to another.
• VisDA 2017 classification dataset: 12 categories of object images in 2 domains: 3D-models and real images.
• Office-Home dataset: images of objects in 4 domains: art, clipart, product and real-world.
• DukeMTMC-reid and Market-1501: two pedestrian datasets collected at different places. The evaluation metric is based on open-set image retrieval.

Text-to-text

• Amazon review benchmark dataset: sentiment analysis for four kinds (domains) of reviews: books, DVDs, electronics, kitchen
• ECML/PKDD Spam Filtering: emails from 3 different inboxes, that can represent the 3 domains.
• 20 Newsgroup: collection of newsgroup documents across 6 top categories and 20 subcategories. Subcategories can play the role of the domains, as describe in this article.

Other

• Meta Self-Learning: Meta Self-Learning for Multi-Source Domain Adaptation: A Benchmark (2021)

Results

The results are indicated as the prediction accuracy (in %) in the target domain after adapting the source to the target. For the moment, they only correspond to the results given in the original papers, so the methodology may vary between each paper and these results must be taken with a grain of salt.

Digits transfer (unsupervised)

Source Target	MNIST MNIST-M	Synth SVHN	MNIST SVHN	SVHN MNIST	MNIST USPS	USPS MNIST
SA	56.90	86.44	?	59.32	?	?
DANN	76.66	91.09	?	73.85	?	?
iDANN	96.67	91.95	36.49	84.50	?	?
CoGAN	?	?	?	?	91.2	89.1
DRCN	?	?	40.05	81.97	91.80	73.67
DSN	83.2	91.2	?	82.7	?	?
DTN	?	?	90.66	79.72	?	?
PixelDA	98.2	?	?	?	95.9	?
ADDA	?	?	?	76.0	89.4	90.1
UNIT	?	?	?	90.53	95.97	93.58
GenToAdapt	?	?	?	92.4	95.3	90.8
SBADA-GAN	99.4	?	61.1	76.1	97.6	95.0
DAassoc	89.47	91.86	?	97.60	?	?
CyCADA	?	?	?	90.4	95.6	96.5
I2I	?	?	?	92.1	95.1	92.2
DIRT-T	98.7	?	76.5	99.4	?	?
DeepJDOT	92.4	?	?	96.7	95.7	96.4
DTA	?	?	?	99.4	99.5	99.1
LSTNet	?	?	?	?	97.61	97.01

Challenges

• Visual Domain Adaptation Challenge (VisDA)
• Open AI Retro Contest

Libraries

• Domain Adaptation: Salad (Semi-supervised Adaptive Learning Across Domains)

Books

• Transfer Learning in Action | GitHub Repo
• Transfer Learning for Natural Language Processing
• Hands-On Transfer Learning with Python | GitHub Repo
• Transfer Learning in Action

awesome-transfer-learning's Issues

Please add following paper related to Domain Adaptation(Accepted IJCNN19, Oral)

Hi @artix41
Please add the following paper related to Domain Adaptation. It has been accepted in IJCNN2019 Oral.
Paper: https://arxiv.org/abs/1904.01341
Project page: https://vinodkkurmi.github.io/DiscriminatorDomainAdaptation/
Code: https://github.com/vinodkkurmi/DiscriminatorDomainAdaptation

Thanks

Adding GCA paper (ECML 2018) to the list

I shall be glad if you could add the paper published at ECML 2018 https://link.springer.com/chapter/10.1007/978-3-030-10928-8_50 in your list.

The algorithm name is "GCA".
The title of the paper is "A Unified Framework for Domain Adaptation Using Metric Learning on Manifolds".

It should logically come in the section Subspace Learning under Unsupervised Domain Adaptation.

congrats !!! good effort to have transfer-learning/DA library. I am interested to contribute

Good work

You've done excellent work in transfer learning.
In fact, I also have a repo on transfer learning: https://github.com/jindongwang/transferlearning . It is basically in Chinese. If you are interested, I hope we can contribute together!

Asking for domain adaptation method suggestion

Suppose I train a ML model to classify human face (from direct frontal angle in a day environment) as either male or female on a large labeled dataset. Let call the model X. Then later I collect more data with different camera angle, e.g. the camera is now looking at the face with 45 degrees in a night environment, I labeled a few samples for this new dataset.
If I want to do transfer learning from model X to the new dataset, what's the best approach? My goal is to predict new dataset accurately with few labeled samples. Because the cost of labeling is high in my problem. I don't care about the accuracy of the first dataset. I only care about the accuracy of the new dataset.

E.g. I know that you could just re-train only the last layer from the model X and freeze early layers for the new dataset, but maybe that approach is for datasets with a similar distribution like ImageNet images and cats/dogs images?
ImageNet models surely have seen many kinds of real-world objects so it's able to classify cats and dogs when transfer learning.

But for my case, both datasets differ slightly in distribution/domain, that is the day/night environment and angle. And the model has seen only the frontal angle for its entire life. How can it adapt to a 45-degree angle the best?

If you know any technique/tutorial/paper that might work, let me know. It would be best if you have experience with it.

New NIPS Papers on Domain Adaptation

• Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
• Unsupervised Image-to-Image Translation Using Domain-Specific Variational Information Bound
• Adversarial Multiple Source Domain Adaptation
• Co-regularized Alignment for Unsupervised Domain Adaptation
• Revisiting (ϵ,γ,τ)-similarity learning for domain adaptation

Should be added once proceedings are published.

Can you add the following papers to your list ?

References -

[1] https://www.sciencedirect.com/science/article/abs/pii/S0952197618301088

[2] https://ieeexplore.ieee.org/abstract/document/8451152

[3] https://link.springer.com/chapter/10.1007/978-3-030-01424-7_34

New ICLR Submissions

Just as a quick link list, here is a list of ICLR Submissions using the keyword "Domain Adaptation". I guess waiting for the reviews makes sense before including them in the reading list.

Unsupervised DA

• A Collaborative Low-Rank Reconstructive Layer for Domain Shift Problems 
• Confidence Regularized Self-Training 
• Variational Domain Adaptation 
• Cosine similarity-based Adversarial process 
• DOMAIN ADAPTATION VIA DISTRIBUTION AND REPRESENTATION MATCHING: A CASE STUDY ON TRAINING DATA SELECTION VIA REINFORCEMENT LEARNING 
• Unsupervised Domain Adaptation for Distance Metric Learning 

Open set

• Learning Factorized Representations for Open-Set Domain Adaptation
• Reducing Overconfident Errors outside the Known Distribution 

Translation Based

• Augmented Cyclic Adversarial Learning for Domain Adaptation 
• Exemplar Guided Unsupervised Image-to-Image Translation with Semantic Consistency

Adversarial Adaptation

• IMPROVING ADVERSARIAL DISCRIMINATIVE DOMAIN ADAPTATION 

Dataset based

• Dataset Distillation 
• Morpho-MNIST: Quantitative Assessment and Diagnostics for Representation Learning 

Unsupervised

• Domain Adaptation for Structured Output via Disentangled Patch Representations 

Applications

• ADVERSARIAL DOMAIN ADAPTATION FOR STABLE BRAIN-MACHINE INTERFACES 

Multidomain Learning/Domain Generalization

• Multi-Domain Adversarial Learning 
• GradMix: Multi-source Transfer across Domains and Tasks 
• The Natural Language Decathlon: Multitask Learning as Question Answering 
• Semi-supervised Learning with Multi-Domain Sentiment Word Embeddings 
• Meta Domain Adaptation: Meta-Learning for Few-Shot Learning under Domain Shift 
• Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach 

Adv Examples as Domain Shift

• Improving the Generalization of Adversarial Training with Domain Adaptation 
• SPIGAN: Privileged Adversarial Learning from Simulation 

Unsorted

• Stacking for Transfer Learning 
• NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES 
• REVISTING NEGATIVE TRANSFER USING ADVERSARIAL LEARNING 
• Regularized Learning for Domain Adaptation under Label Shifts 

LSTD

Hi,
Thank you for your review of ressources in transfer learning.
I recommend to add the paper "LSTD: A Low-Shot Transfer Detector for Object Detection" (https://arxiv.org/abs/1803.01529).
You can find the implementation in Caffe here.
It uses a mix of SSD & Faster R-CNN but the interest is more in the regularization techniques the authors have created.
Also there is an implementation in Pytorch but it's buggy and seems dead.
To be complete, there is an application of the LSTD for "document layout analysis"/"document segmentation" using text mining for the classifier.

Finally, there is a model called "Zero-Shot Object Detection" that could be interesting but I didn't read yet.