The official PyTorch implementation of our ECCV 2022 paper:
AiATrack: Attention in Attention for Transformer Visual Tracking
Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang, Junsong Yuan
[ECVA Open Access] [ArXiv Preprint] [Trained Models] [Raw Results] [Transformer Tracking]
Transformer trackers have achieved impressive advancements recently, where the attention mechanism plays an important role. However, the independent correlation computation in the attention mechanism could result in noisy and ambiguous attention weights, which inhibits further performance improvement. To address this issue, we propose an attention in attention module (named AiA), which enhances appropriate correlations and suppresses erroneous ones by seeking consensus among all correlation vectors. Our AiA module can be readily applied to both self-attention blocks and cross-attention blocks to facilitate feature aggregation and information propagation for visual tracking. Moreover, we propose a streamlined Transformer tracking framework (dubbed AiATrack), by introducing efficient feature reuse and target-background embeddings to make full use of temporal references. Experiments show that our tracker achieves state-of-the-art performance on several tracking benchmarks while running at a real-time speed.
The proposed AiATrack sets state-of-the-art results on 8 widely used benchmarks. Using ResNet-50 pre-trianed on ImageNet-1k, we can get:
The proposed AiATrack can run at 38fps (frames per second) on a single NVIDIA GeForce RTX 2080 Ti.
It takes nearly two days to train our model on 8 NVIDIA GeForce RTX 2080 Ti (each of which has 11GB GPU memory).
The proposed AiATrack has 15.79M (million) model parameters.
Trained Models (containing the model we trained on four datasets and the model we trained on GOT-10k only) [download zip file]
Raw Results (containing raw tracking results on the datasets we benchmarked in the paper) [download zip file]
Download and unzip these two zip files under AiATrack project path, then both of them can be directly used by our code.
-
Our experiments are conducted with Ubuntu 18.04 and CUDA 10.1.
-
-
Clone our repository to your local project directory.
-
Download the training datasets (LaSOT, TrackingNet, GOT-10k, COCO2017) and testing datasets (NfS, OTB, UAV123) to your disk, the organized directory should look like:
--LaSOT/ |--airplane |... |--zebra --TrackingNet/ |--TRAIN_0 |... |--TEST --GOT10k/ |--test |--train |--val --COCO/ |--annotations |--images --NFS30/ |--anno |--sequences --OTB100/ |--Basketball |... |--Woman --UAV123/ |--anno |--data_seq
-
Edit the PATH in
lib/test/evaluation/local.py
andlib/train/adim/local.py
to the proper absolute path.
-
-
We use conda to manage the environment.
conda create --name aiatrack python=3.6 conda activate aiatrack sudo apt-get install ninja-build sudo apt-get install libturbojpeg bash install.sh
Note that your PyTorch version must be
pytorch <= 1.10.1
to successfully compile PreciseRoIPooling since <THC/THC.h> has been removed inpytorch 1.11
. -
-
Multiple GPU training by DDP (suppose you have 8 GPU)
python tracking/train.py --mode multiple --nproc 8
-
Single GPU debugging (too slow, not recommended for training)
python tracking/train.py
-
For GOT-10k evaluation, remember to set
--config baseline_got
.
-
-
-
Make sure you have prepared the trained model.
-
On large-scale benchmarks:
-
LaSOT
python tracking/test.py --dataset lasot python tracking/test.py --dataset lasot_ext
Then evaluate the raw results using the official MATLAB toolkit.
-
TrackingNet
python tracking/test.py --dataset trackingnet python lib/test/utils/transform_trackingnet.py --tracker_name aiatrack --cfg_name baseline
Then upload
test/tracking_results/aiatrack/baseline/trackingnet_submit.zip
to the online evaluation server. -
GOT-10k
python tracking/test.py --param baseline_got --dataset got10k_test python lib/test/utils/transform_got10k.py --tracker_name aiatrack --cfg_name baseline_got
Then upload
test/tracking_results/aiatrack/baseline_got/got10k_submit.zip
to the online evaluation server.
-
-
On small-scale benchmarks:
-
NfS30, OTB100, UAV123
python tracking/test.py --dataset nfs python tracking/test.py --dataset otb python tracking/test.py --dataset uav python tracking/analysis_results.py
As previous works did, the frames where the target object doesn't exist will be excluded during the analysis.
-
-
For multiple threads inference, just add
--threads 40
aftertracking/test.py
(suppose you want to use 40 threads in total). -
To show the immediate prediction results during inference, modify
settings.show_result = True
inlib/test/evaluation/local.py
(may have bugs if you try this on a remote sever). -
Please refer to STARK+Alpha-Refine for VOT integration and DETR Tutorial for correlation map visualization.
-
❤️❤️❤️Our idea is implemented base on the following projects. We really appreciate their wonderful open-source works!
- STARK [related paper]
- PyTracking [related paper]
- DETR [related paper]
- PreciseRoIPooling [related paper]
If any parts of our paper and code help your research, please consider citing us and giving a star to our repository.
@inproceedings{gao2022aiatrack,
title={AiATrack: Attention in Attention for Transformer Visual Tracking},
author={Gao, Shenyuan and Zhou, Chunluan and Ma, Chao and Wang, Xinggang and Yuan, Junsong},
booktitle={European Conference on Computer Vision},
pages={146--164},
year={2022},
organization={Springer}
}
If you have any questions or concerns, feel free to open issues or directly contact me through the ways on my GitHub homepage. Suggestions and collaborations are also highly welcome!