whwu95 / mvfnet

【AAAI'2021】MVFNet: Multi-View Fusion Network for Efficient Video Recognition

MVFNet: Multi-View Fusion Network for Efficient Video Recognition (AAAI 2021)

Overview

We release the code of the MVFNet (Multi-View Fusion Network). The core code to implement the Multi-View Fusion Module is codes/models/modules/MVF.py.

[Mar 24, 2021] We has released the code of MVFNet.

[Dec 20, 2020] MVFNet has been accepted by AAAI 2021.

• Prerequisites
• Data Preparation
• Model Zoo
• Testing
• Training

Prerequisites

All dependencies can be installed using pip:

python -m pip install -r requirements.txt

Our experiments run on Python 3.7 and PyTorch 1.5. Other versions should work but are not tested.

Download Pretrained Models

• Download ImageNet pre-trained models

cd pretrained
sh download_imgnet.sh

• Download K400 pre-trained models

Please refer to Model Zoo.

Data Preparation

Please refer to DATASETS.md for data preparation.

Model Zoo

Architecture	Dataset	T x interval	Top-1 Acc.	Pre-trained model	Train log	Test log
MVFNet-ResNet50	Kinetics-400	4x16	74.2%	Download link	Log link	Log link
MVFNet-ResNet50	Kinetics-400	8x8	76.0%	Download link	Miss	Log link
MVFNet-ResNet50	Kinetics-400	16x4	77.0%	Download link	Log link	Log link
MVFNet-ResNet101	Kinetics-400	4x16	76.0%	Download link	Log link	Log link
MVFNet-ResNet101	Kinetics-400	8x8	77.4%	Download link	Log link	Log link
MVFNet-ResNet101	Kinetics-400	16x4	78.4%	Download link	Log link	Log link

Testing

• For 3 crops, 10 clips, the processing of testing

# Dataset: Kinetics-400
# Architecture: R50_8x8 ACC@1=76.0%
bash scripts/dist_test_recognizer.sh configs/MVFNet/K400/mvf_kinetics400_2d_rgb_r50_dense.py ckpt_path 8 --fcn_testing

Training

This implementation supports multi-gpu, DistributedDataParallel training, which is faster and simpler.

• For example, to train MVFNet-ResNet50 on Kinetics400 with 8 gpus, you can run:

bash scripts/dist_train_recognizer.sh configs/MVFNet/K400/mvf_kinetics400_2d_rgb_r50_dense.py 8

• We also provide the script to train MVFNet on Kinetics400 with multiple machines (e.g., 2 machines and 16 GPUs).

# For first machine, --master_addr is the ip of your first machine
bash scripts/dist_train_multinode_1.sh configs/MVFNet/K400/mvf_kinetics400_2d_rgb_r50_dense.py 8

# For second machine, --master_addr is still the ip of your first machine
bash scripts/dist_train_multinode_2.sh configs/MVFNet/K400/mvf_kinetics400_2d_rgb_r50_dense.py 8

Acknowledgements

We especially thank the contributors of the mmaction codebase for providing helpful code.

License

This repository is released under the Apache-2.0. license as found in the LICENSE file.

Citation

If you think our work is useful, please feel free to cite our paper 😆 :

@inproceedings{wu2020MVFNet,
  author    = {Wu, Wenhao and He, Dongliang and Lin, Tianwei and Li, Fu and Gan, Chuang and Ding, Errui},
  title     = {MVFNet: Multi-View Fusion Network for Efficient Video Recognition},
  booktitle = {AAAI},
  year      = {2021}
}

Contact

For any question, please file an issue or contact

Wenhao Wu: [email protected]

mvfnet's Issues

UCF101

Hi, Can you provide UCF101 configs?

0it [00:00, ?it/s]

Hello, when processing frames extracted from kinetics400 dataset, what is the cause of the above error in running code?

Is this right for the test configuration?

Hi I noticed your great job for action recognition from AAAI 2021.
And I am trying to get the test results as yours on Kinetics400. After I have processed all the test videos to get the frames, I found that there is no annotation processing for kinetics400 test set up, neither in your configuration file.
Could you share the test annotation for Kinetics400 and explain why using validation for test?

MVFNet/configs/MVFNet/K400/mvf_kinetics400_2d_rgb_r50_dense.py

Line 58 in ed33622

ann_file_test = 'datalist/kinetics400/val_ffmpeg_fps30.txt'

MVFNet/configs/MVFNet/K400/mvf_kinetics400_2d_rgb_r50_dense.py

Line 145 in ed33622

data_root=data_root_val,

ann_file_test = 'datalist/kinetics400/val_ffmpeg_fps30.txt'
...
test=dict(
        type=dataset_type,
        ann_file=ann_file_test,
        data_root=data_root_val,
        pipeline=test_pipeline, 
        test_mode=True,
        modality='RGB',
        filename_tmpl='img_{:05}.jpg'    ))

Thanks a lot!

MVF constructed by nn.Conv3d？

Hi，
Is MVF block constructed by nn.Conv3d？
It can by constructed by nn.Conv2d？

About online recognition

Thank you for your great work.
My question is that the mvf module needs to use convolution among multi-view dimensions,especially contains T dimension. If we want to apply the model into online recognition, it is difficult to store too many history frames. So how to apply it to the online recognition?Thank you.

kinetic trainlist

Thank you for your great work.
Do you have the annnotation of the kinetic train.csv?
Thanks for your help!

Why do you override DistributedDataParallel and DistributedSampler ?

multi nodes need to set datasets in each machine？

do i need to copy datasets to each machine when i train the model with multi node?
thanks