mcg-nju / videomae-action-detection

[NeurIPS 2022 Spotlight] VideoMAE for Action Detection

VideoMAE for Action Detection (NeurIPS 2022 Spotlight) [Arxiv]

VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Zhan Tong, Yibing Song, Jue Wang, Limin Wang
Nanjing University, Tencent AI Lab

This repo contains the supported code and scripts to reproduce action detection results of VideoMAE. The code of pre-training is available in original repo.

📰 News

[2023.1.16] Code and pre-trained models are available now!

🚀 Main Results

✨ AVA 2.2

Method	Extra Data	Extra Label	Backbone	#Frame x Sample Rate	mAP
VideoMAE	Kinetics-400	✗	ViT-S	16x4	22.5
VideoMAE	Kinetics-400	✓	ViT-S	16x4	28.4
VideoMAE	Kinetics-400	✗	ViT-B	16x4	26.7
VideoMAE	Kinetics-400	✓	ViT-B	16x4	31.8
VideoMAE	Kinetics-400	✗	ViT-L	16x4	34.3
VideoMAE	Kinetics-400	✓	ViT-L	16x4	37.0
VideoMAE	Kinetics-400	✗	ViT-H	16x4	36.5
VideoMAE	Kinetics-400	✓	ViT-H	16x4	39.5
VideoMAE	Kinetics-700	✗	ViT-L	16x4	36.1
VideoMAE	Kinetics-700	✓	ViT-L	16x4	39.3

🔨 Installation

Please follow the instructions in INSTALL.md.

➡️ Data Preparation

Please follow the instructions in DATASET.md for data preparation.

⤴️ Fine-tuning with pre-trained models

The fine-tuning instruction is in FINETUNE.md.

📍Model Zoo

We provide pre-trained and fine-tuned models in MODEL_ZOO.md.

☎️ Contact

Zhan Tong: [email protected]

👍 Acknowledgements

Thanks to Lei Chen for support. This project is built upon MAE-pytorch, BEiT and AlphAction. Thanks to the contributors of these great codebases.

🔒 License

The majority of this project is released under the CC-BY-NC 4.0 license as found in the LICENSE file. Portions of the project are available under separate license terms: pytorch-image-models are licensed under the Apache 2.0 license. BEiT is licensed under the MIT license.

✏️ Citation

If you think this project is helpful, please feel free to leave a star⭐️ and cite our paper:

@inproceedings{tong2022videomae,
  title={Video{MAE}: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training},
  author={Zhan Tong and Yibing Song and Jue Wang and Limin Wang},
  booktitle={Advances in Neural Information Processing Systems},
  year={2022}
}

@article{videomae,
  title={VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training},
  author={Tong, Zhan and Song, Yibing and Wang, Jue and Wang, Limin},
  journal={arXiv preprint arXiv:2203.12602},
  year={2022}
}

videomae-action-detection's Issues

How can I use pre-trained model to detect action directly?

Thank you for your job!

How can I use pre-trained model to detect action directly?

Looking forward to your reply.

Input resolution of train and validation

Hi authors,

I found that the input video resolution was set to 16x224x224 while the video resolution used during the validation was 16x256x352.

I might read it wrong, but I wonder how much the video resolution could affect the validation accuracy. Could you please provide results validated with 224x224?

Could you provide AVA2.2 finetuned weights for ViT-L and ViT-H?

@wanglimin @yztongzhan

Thanks for your great job on video action detection. I use ViT-B finetuned weights test on my own video. The performance is so-so. Could you provide AVA2.2 finetuned weights for ViT-L and ViT-H?

Thanks a lot

What's the "extra label" setting in AVA 2.2 results

Hello,

May I kindly ask what's the "extra label" setting in AVA 2.2 results? It brings the results better by a lot.

AVA model checkpoint for ViT-L & ViT-H ?

Do you plan to release ViT-L/H model checkpoint for AVA dataset?

How to test?

Thanks for your work

After fine-tuning, the performance of the model on the verification set is obtained, but how to get the accuracy of the model on the test set? Is there a demo file or process for reference?