iq-scm / vit-adapter

[ICLR 2023 Spotlight] Vision Transformer Adapter for Dense Predictions

ViT-Adapter

The official implementation of the paper "Vision Transformer Adapter for Dense Predictions".

Paper | Blog in Chinese | Slides | Colab Notebook (thanks @IamShubhamGupto, @dudifrid)

News

• 2023/04/14: 🚀 ViT-Adapter is used in EVA and DINOv2!
• 2023/01/21: Our paper is accepted by ICLR 2023!
• 2023/01/17: We win the champion of WSDM Cup 2023 Toloka VQA Challenge using ViT-Adapter.
• 2022/10/20: ViT-Adapter is adopted by Zhang et al. and they ranked 1st in the UVO Challenge 2022.
• 2022/08/22: ViT-Adapter is adopted by BEiT-3 and created new SOTA of 62.8 mIoU on ADE20K.
• 2022/06/09: ViT-Adapter-L achieves 60.4 box AP and 52.5 mask AP on COCO test-dev without Objects365.
• 2022/06/04: Code and models are released.
• 2022/05/12: ViT-Adapter-L reaches 85.2 mIoU on Cityscapes test set without coarse data.
• 2022/05/05: ViT-Adapter-L achieves the SOTA on ADE20K val set with 60.5 mIoU!

Highlights

• ViT-Adapter supports various dense prediction tasks, including object detection, instance segmentation, semantic segmentation, visual grounding, panoptic segmentation, etc.
• This codebase includes many SOTA detectors and segmenters to achieve top performance, such as HTC++, Mask2Former, DINO.

results.mp4

Abstract

This work investigates a simple yet powerful dense prediction task adapter for Vision Transformer (ViT). Unlike recently advanced variants that incorporate vision-specific inductive biases into their architectures, the plain ViT suffers inferior performance on dense predictions due to weak prior assumptions. To address this issue, we propose the ViT-Adapter, which allows plain ViT to achieve comparable performance to vision-specific transformers. Specifically, the backbone in our framework is a plain ViT that can learn powerful representations from large-scale multi-modal data. When transferring to downstream tasks, a pre-training-free adapter is used to introduce the image-related inductive biases into the model, making it suitable for these tasks. We verify ViT-Adapter on multiple dense prediction tasks, including object detection, instance segmentation, and semantic segmentation. Notably, without using extra detection data, our ViT-Adapter-L yields state-of-the-art 60.9 box AP and 53.0 mask AP on COCO test-dev. We hope that the ViT-Adapter could serve as an alternative for vision-specific transformers and facilitate future research. The code and models will be released.

Method

Catalog

• Support flash attention
• Support faster deformable attention
• Segmentation checkpoints
• Segmentation code
• Detection checkpoints
• Detection code
• Initialization

Citation

If this work is helpful for your research, please consider citing the following BibTeX entry.

@article{chen2022vitadapter,
  title={Vision Transformer Adapter for Dense Predictions},
  author={Chen, Zhe and Duan, Yuchen and Wang, Wenhai and He, Junjun and Lu, Tong and Dai, Jifeng and Qiao, Yu},
  journal={arXiv preprint arXiv:2205.08534},
  year={2022}
}

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.