memory-attention-networks / mans

Memory Attention Networks, in IJCAI 2018

Memory Attention Networks (MANs)

Skeleton-based action recognition task is entangled with complex spatio-temporal variations of skeleton joints, and remains challenging for Recurrent Neural Networks (RNNs). In this work, we propose a temporal-then-spatial recalibration scheme to alleviate such complex variations, resulting in an end-to-end Memory Attention Networks (MANs) which consist of a Temporal Attention Recalibration Module (TARM) and a Spatio-Temporal Convolution Module (STCM). Specifically, the TARM is deployed in a residual learning module that employs a novel attention learning network to recalibrate the temporal attention of frames in a skeleton sequence. The STCM treats the attention calibrated skeleton joint sequences as images and leverages the Convolution Neural Networks (CNNs) to further model the spatialand temporal information of skeleton data. These two modules (TARM and STCM) seamlessly form a single network architecture that can be trained in an end-to-end fashion. MANs significantly boost the performance of skeleton-based action recognition and achieve the best results on four challenging benchmark datasets: NTU RGB+D, HDM05, SYSU-3D and UT-Kinect.

We provide a demo for MANs on NTU RGB+D dataset based on Keras.

Install

To run this demo, you should install these dependencies:

Keras 2.0.8
tensorflow 1.3.0

Run demo

python process_data.py
python train_MANs.py

The experimental results of the comparison algorithm are directly quoted from the corresponding papers.

Citation

if you find MANs useful in your research, please consider citing:

@inproceedings{Xie2018Memory,
  title={Memory Attention Networks for Skeleton-based Action Recognition},
  author={Xie, Chunyu and Li, Ce and Zhang, Baochang and Chen, Chen and Han, Jungong and Zou, Changqing and Liu, Jianzhuang},
  booktitle={International Joint Conference on Artificial Intelligence},
  pages={1639--1645},
  year={2018},
}

The paper link: Memory Attention Networks for Skeleton-based Action Recognition

Contact

[email protected]
Any discussions, suggestions and questions are welcome!

mans's Issues

Confusion regarding a variable

Hi, thanks for making your code public. In the train_MANs.py file, you have imported a class Models but haven't used it anywhere. Is this important or can the training be performed without it?

pretrained model needed?

would you mind sharing your pretrained model or demo for testing