sjpollard / slowfast

PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

Note

To get this working I needed to do the following:

conda install python=3.8

pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html

python -m pip install detectron2 -f   https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.10/index.html

pip install simplejson

conda install av -c conda-forge

pip install opencv-python

git clone https://github.com/facebookresearch/pytorchvideo.git
cd ../pytorchvideo
pip install -e .
cd ../SlowFast

pip install 'git+https://github.com/facebookresearch/fairscale'

pip install psutil

pip install scipy

pip install pandas

pip install pillow==9

pip install scikit-learn

pip install setuptools==59.5.0

I also needed to swap DATA.DECODING_BACKEND to be pyav and swap out slowfast/datasets/decoder.py for the fix in this pull request.

PySlowFast

PySlowFast is an open source video understanding codebase from FAIR that provides state-of-the-art video classification models with efficient training. This repository includes implementations of the following methods:

• SlowFast Networks for Video Recognition
• Non-local Neural Networks
• A Multigrid Method for Efficiently Training Video Models
• X3D: Progressive Network Expansion for Efficient Video Recognition
• Multiscale Vision Transformers
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• MViTv2: Improved Multiscale Vision Transformers for Classification and Detection
• Masked Feature Prediction for Self-Supervised Visual Pre-Training
• Masked Autoencoders As Spatiotemporal Learners
• Reversible Vision Transformers

Introduction

The goal of PySlowFast is to provide a high-performance, light-weight pytorch codebase provides state-of-the-art video backbones for video understanding research on different tasks (classification, detection, and etc). It is designed in order to support rapid implementation and evaluation of novel video research ideas. PySlowFast includes implementations of the following backbone network architectures:

• SlowFast
• Slow
• C2D
• I3D
• Non-local Network
• X3D
• MViTv1 and MViTv2
• Rev-ViT and Rev-MViT

Updates

• We now Reversible Vision Transformers. Both Reversible ViT and MViT models released. See projects/rev.
• We now support MAE for Video. See projects/mae for more information.
• We now support MaskFeat. See projects/maskfeat for more information.
• We now support MViTv2 in PySlowFast. See projects/mvitv2 for more information.
• We now support A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning. See projects/contrastive_ssl for more information.
• We now support Multiscale Vision Transformers on Kinetics and ImageNet. See projects/mvit for more information.
• We now support PyTorchVideo models and datasets. See projects/pytorchvideo for more information.
• We now support X3D Models. See projects/x3d for more information.
• We now support Multigrid Training for efficiently training video models. See projects/multigrid for more information.
• PySlowFast is released in conjunction with our ICCV 2019 Tutorial.

License

PySlowFast is released under the Apache 2.0 license.

Model Zoo and Baselines

We provide a large set of baseline results and trained models available for download in the PySlowFast Model Zoo.

Installation

Please find installation instructions for PyTorch and PySlowFast in INSTALL.md. You may follow the instructions in DATASET.md to prepare the datasets.

Quick Start

Follow the example in GETTING_STARTED.md to start playing video models with PySlowFast.

Visualization Tools

We offer a range of visualization tools for the train/eval/test processes, model analysis, and for running inference with trained model. More information at Visualization Tools.

Contributors

PySlowFast is written and maintained by Haoqi Fan, Yanghao Li, Bo Xiong, Wan-Yen Lo, Christoph Feichtenhofer.

Citing PySlowFast

If you find PySlowFast useful in your research, please use the following BibTeX entry for citation.

@misc{fan2020pyslowfast,
  author =       {Haoqi Fan and Yanghao Li and Bo Xiong and Wan-Yen Lo and
                  Christoph Feichtenhofer},
  title =        {PySlowFast},
  howpublished = {\url{https://github.com/facebookresearch/slowfast}},
  year =         {2020}
}