simonvandenhende / awesome-multi-task-learning

A list of multi-task learning papers and projects.

Awesome Multi-Task Learning

This page contains a list of papers on multi-task learning for computer vision. Please create a pull request if you wish to add anything. If you are interested, consider reading our recent survey paper.

Multi-Task Learning for Dense Prediction Tasks: A Survey

Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai and Luc Van Gool.

Workshop

📢 📢 📢 We organized a workshop on multi-task learning at ICCV 2021 (Link).

• Jan 13: The recordings of our invited talks are now available on Youtube.

Table of Contents:

• Survey papers
• Datasets
• Architectures
  • Encoder-based
  • Decoder-based
  • Other
• Neural Architecture Search
• Optimization strategies
• Transfer learning

Survey papers

• Vandenhende, S., Georgoulis, S., Van Gansbeke, W., Proesmans, M., Dai, D., & Van Gool, L. Multi-Task Learning for Dense Prediction Tasks: A Survey, T-PAMI, 2020. [PyTorch]

• Ruder, S. An overview of multi-task learning in deep neural networks, ArXiv, 2017.

• Zhang, Y. A survey on multi-task learning, ArXiv, 2017.

• Gong, T., Lee, T., Stephenson, C., Renduchintala, V., Padhy, S., Ndirango, A., ... & Elibol, O. H. A comparison of loss weighting strategies for multi task learning in deep neural networks, IEEE Access, 2019.

Datasets

The following datasets have been regularly used in the context of multi-task learning:

• NYUDv2
• Cityscapes
• PASCAL
• Taskonomy
• KITTI
• SUN RGB-D
• BDD100K

Architectures

Encoder-based architectures

• Misra, I., Shrivastava, A., Gupta, A., & Hebert, M. Cross-stitch networks for multi-task learning, CVPR, 2016. [PyTorch]

• Gao, Y., Ma, J., Zhao, M., Liu, W., & Yuille, A. L. Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction, CVPR, 2019. [Tensorflow] [PyTorch]

• Liu, S., Johns, E., & Davison, A. J. End-to-end multi-task learning with attention, CVPR, 2019. [PyTorch]

Decoder-based architectures

• Bilen, H., Vedaldi, A. Integrated perception with recurrent multi-task neural networks, NeurIPS, 2016.

• Xu, D., Ouyang, W., Wang, X., & Sebe, N. Pad-net: Multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing, CVPR, 2018.

• Zhang, Z., Cui, Z., Xu, C., Jie, Z., Li, X., & Yang, J. Joint task-recursive learning for semantic segmentation and depth estimation, ECCV, 2018.

• Ruder, S., Bingel, J., Augenstein, I., & Søgaard, A. Latent multi-task architecture learning, AAAI, 2019.

• Zhang, Z., Cui, Z., Xu, C., Yan, Y., Sebe, N., & Yang, J. Pattern-affinitive propagation across depth, surface normal and semantic segmentation, CVPR, 2019.

• Zhou, L., Cui, Z., Xu, C., Zhang, Z., Wang, C., Zhang, T., & Yang, J. Pattern-Structure Diffusion for Multi-Task Learning, CVPR, 2020.

• Vandenhende, S., Georgoulis, S., & Van Gool, L. MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning, ECCV, 2020. [PyTorch]

Other

• Yang, Y., & Hospedales, T. Deep multi-task representation learning: A tensor factorisation approach, ICLR, 2017.

• Kokkinos, Iasonas. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory, CVPR, 2017.

• Rebuffi, S. A., Bilen, H., & Vedaldi, A. Learning multiple visual domains with residual adapters, NIPS, 2017.

• Long, M., Cao, Z., Wang, J., & Philip, S. Y. Learning multiple tasks with multilinear relationship networks, NIPS, 2017.

• Meyerson, E., & Miikkulainen, R. Beyond shared hierarchies: Deep multitask learning through soft layer ordering, ICLR, 2018.

• Rosenbaum, C., Klinger, T., & Riemer, M. Routing networks: Adaptive selection of non-linear functions for multi-task learning, ICLR, 2018.

• Mallya, A., Davis, D., & Lazebnik, S. Piggyback: Adapting a single network to multiple tasks by learning to mask weights, ECCV, 2018.

• Rebuffi, S. A., Bilen, H., & Vedaldi, A. Efficient parametrization of multi-domain deep neural networks, CVPR, 2018.

• Maninis, K. K., Radosavovic, I., & Kokkinos, I. Attentive single-tasking of multiple tasks, CVPR, 2019. [PyTorch]

• Kanakis, M., Bruggemann, D., Saha, S., Georgoulis, S., Obukhov, A., & Van Gool, L. Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference, ECCV, 2020.

• Wang, Q., Ke, J., Greaves, J., Chu, G., Bender, G., Sbaiz, L., Go, A., Howard, A., Yang, F., Yang, M.H. & Gilbert, J. Multi-path Neural Networks for On-device Multi-domain Visual Classification, WACV, 2021.

• Bruggemann, D., Kanakis, M., Obukhov, A., Georgoulis, S., & Van Gool, L. Exploring Relational Context for Multi-Task Dense Prediction, ArXiv, 2021.

• Li, W. H., Liu, X., & Bilen, H. Universal Representation Learning from Multiple Domains for Few-shot Classification, ICCV, 2021.

• Li, W. H., Liu, X., & Bilen, H. Learning Multiple Dense Prediction Tasks from Partially Annotated Data, ArXiv, 2021.

Neural Architecture Search

• Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., & Feris, R. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification, CVPR, 2017.

• Bragman, F. J., Tanno, R., Ourselin, S., Alexander, D. C., & Cardoso, J. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels, ICCV, 2019.

• Newell, A., Jiang, L., Wang, C., Li, L. J., & Deng, J. Feature partitioning for efficient multi-task architectures, ArXiv, 2019.

• Guo, P., Lee, C. Y., & Ulbricht, D. Learning to Branch for Multi-Task Learning, ICML, 2020.

• Standley, T., Zamir, A. R., Chen, D., Guibas, L., Malik, J., & Savarese, S. Which Tasks Should Be Learned Together in Multi-task Learning?, ICML, 2020.

• Vandenhende, S., Georgoulis, S., De Brabandere, B., & Van Gool, L. Branched multi-task networks: deciding what layers to share, BMVC, 2020.

• Bruggemann, D., Kanakis, M., Georgoulis, S., & Van Gool, L. Automated Search for Resource-Efficient Branched Multi-Task Networks, BMVC, 2020.

• Sun, X., Panda, R., & Feris, R. AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning, NIPS, 2020.

Optimization strategies

• Kendall, A., Gal, Y., & Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics, CVPR, 2018.

• Zhao, X., Li, H., Shen, X., Liang, X., & Wu, Y. A modulation module for multi-task learning with applications in image retrieval, ECCV, 2018.

• Chen, Z., Badrinarayanan, V., Lee, C. Y., & Rabinovich, A. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks, ICML, 2018.

• Sener, O., & Koltun, V. Multi-task learning as multi-objective optimization, NIPS, 2018. [PyTorch]

• Liu, P., Qiu, X., & Huang, X. Adversarial multi-task learning for text classification, ACL, 2018.

• Guo, M., Haque, A., Huang, D. A., Yeung, S., & Fei-Fei, L. Dynamic task prioritization for multitask learning, ECCV, 2018.

• Lin, X., Zhen, H. L., Li, Z., Zhang, Q. F., & Kwong, S. Pareto multi-task learning, NIPS, 2019.

• Suteu, M., & Guo, Y. Regularizing Deep Multi-Task Networks using Orthogonal Gradients, ArXiv, 2019.

• Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., & Finn, C. Gradient surgery for multi-task learning, NIPS, 2020. [Tensorflow]

• Chen, Z., Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H., Chai, Y., & Anguelov, D. Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout, NIPS, 2020.

• Li, W. H., & Bilen, H. Knowledge Distillation for Multi-task Learning, ECCV-Workshop, 2020. [PyTorch]

• Borse, S., Wang, Y., Zhang, Y., & Porikli, F. InverseForm: A Loss Function for Structured Boundary-Aware Segmentation, CVPR 2021.

• Vasu P., Saxena S., Tuzel O. Instance-Level Task Parameters: A Robust Multi-task Weighting Framework, ArXiv, 2021.

Transfer learning & Domain Adaptation

• Cui, Y., Song, Y., Sun, C., Howard, A., & Belongie, S. Large scale fine-grained categorization and domain-specific transfer learning, CVPR, 2018.

• Zamir, A. R., Sax, A., Shen, W., Guibas, L. J., Malik, J., & Savarese, S. Taskonomy: Disentangling task transfer learning, CVPR, 2018. [PyTorch]

• Achille, A., Lam, M., Tewari, R., Ravichandran, A., Maji, S., Fowlkes, C. C., ... & Perona, P. Task2vec: Task embedding for meta-learning, ICCV, 2019. [PyTorch]

• Dwivedi, K., & Roig, G. Representation similarity analysis for efficient task taxonomy & transfer learning, CVPR, 2019. [PyTorch]

• Saha, S., Obukhov, A., Paudel, D. P., Kanakis, M., Chen, Y., Georgoulis, S., & Van Gool, L. Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation, CVPR, 2021.

Robustness

• Mao, C., Gupta, A., Nitin, V., Ray, B., Song, S., Yang, J., & Vondrick, C. Multitask Learning Strengthens Adversarial Robustness, ECCV, 2020. [PyTorch]

• Zamir, A. R., Sax, A., Cheerla, N., Suri, R., Cao, Z., Malik, J., & Guibas, L. J. Robust Learning Through Cross-Task Consistency, CVPR, 2020.

• Georgescu, M. I., Barbalau, A., Ionescu, R. T., Khan, F. S., Popescu, M., & Shah, M. Anomaly Detection in Video via Self-Supervised and Multi-Task Learning, CVPR, 2021.

Other

• Eftekhar, A.,Sax, A., Malik, J., Zamir, A. Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets From 3D Scans, ICCV, 2021.

awesome-multi-task-learning's Issues

New MTL papers from our group

Hi there,

We're a MLSys group working on multi-task learning at the University of Massachusetts Amherst. Recently we have some new works completed. Would you mind taking a look at these papers and adding them to your repo list? Thank you!

1. Rethinking hard-parameter sharing in multi-domain learning, accepted by ICME 2022 (acceptance rate 29%).
2. AutoMTL: A Programming Framework for Automated Multi-Task Learning. A NAS-based MTL infrastructure.
3. A Tree-Structured Multi-Task Model Recommender, accepted by AutoML-Conf 2022 (acceptance rate 19.2%).

Best,
Lijun Zhang