all-for-code / imted

imTED: Integrally Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection

Code of Integrally Migrating Pre-trained Transformer Encoder-decoders for Visual Object Detection.

The code is based on mmdetection, please refer to get_started.md and MMDET_README.md to set up the environment and prepare the data.

Config Files and Performance and Trained Weights

We provide 9 configuration files in the configs directory.

Config File	Backbone	Epochs	Box AP	Mask AP	Download
imted_faster_rcnn_vit_small_3x_coco	ViT-S	36	48.2		model
imted_faster_rcnn_vit_base_3x_coco	ViT-B	36	52.9		model
imted_faster_rcnn_vit_large_3x_coco	ViT-L	36	55.4		model
imted_mask_rcnn_vit_small_3x_coco	ViT-S	36	48.7	42.7	model
imted_mask_rcnn_vit_base_3x_coco	ViT-B	36	53.3	46.4	model
imted_mask_rcnn_vit_large_3x_coco	ViT-L	36	55.5	48.1	model
imted_faster_rcnn_vit_base_2x_base_training_coco	ViT-B	24	50.6		model
imted_faster_rcnn_vit_base_2x_finetuning_10shot_coco	ViT-B	108	23.0		model
imted_faster_rcnn_vit_base_2x_finetuning_30shot_coco	ViT-B	108	30.4		model

MAE Pre-training

The pre-trained model is trained with the official MAE code. For ViT-S, we use a 4-layer decoder with dimension 256 for 800 epochs of pre-training. For ViT-B, we use an 8-layer decoder with dimension 512 for 1600 epochs of pre-training. Pre-trained weights can be downloaded from the official MAE weight. For ViT-L, we use an 8-layer decoder with dimension 512 for 1600 epochs of pre-training. Pre-trained weights can be downloaded from the official MAE weight.

Last Step of Preparation

For all experiments, remember to modify the path of pre-trained weights in the configuration files, e.g. configs/imted/imted_faster_rcnn_vit_small_3x_coco.py.

For few-shot experiments, please refer to FsDet for data preparation. Remember to modify the path of json in the configuration files, e.g. configs/imted/few_shot/imted_faster_rcnn_vit_base_2x_base_training_coco.py.

Evaluating with 1 GPU

tools/dist_test.sh "path/to/config/file.py" "path/to/trained/weights.pth" 1 --eval bbox

Training with 8 GPUs

tools/dist_train.sh "path/to/config/file.py" 8 

If you have any questions with our code/paper, feel free to contact me through issue or e-mail.