exitudio / gaitmixer

Official repository for "GaitMixer: Skeleton-based Gait Representation Learning via Wide-spectrum Multi-axial Mixer"

GaitMixer

This repository contains the PyTorch code for:

GaitMixer: Skeleton-based Gait Representation Learning via Wide-spectrum Multi-axial Mixer The paper is accepted to ICASSP 2023.

Ekkasit Pinyoanuntapong, Ayman Ali, Pu Wang, Minwoo Lee, Chen Chen

Campare to Previous SOTA Skeleton-based Gait Recognition

We proposed 2 SOTA methods GaitFormer and GaitMixer in skeleton based gait recognition, improving from GaitGraph by 12% on average.

Method	NM	BG	CL	Mean
PoseGait	68.7	44.5	36.0	49.7
GaitGraph	87.7	74.8	66.3	76.3
GaitGraph2	82.0	73.2	63.6	72.9
GaitFormer (ours)	91.5	81.4	77.2	83.4
GaitMixer (ours)	94.9	85.6	84.5	88.3

Quick Start

First, create a virtual environment or install dependencies directly with:

conda env create -f environment.yml

Data preparation

Follow GaitGraph data preparation

Train

To train the model you can run the train.py script. Our paper presents 2 models to study behavior of self-attetion and large kernel depthwise separable convolution

1. GaitMixer (Spatial self-attention & temporal convolution)

python train.py casia-b ../data/casia-b_pose_train_valid.csv --valid_data_path ../data/casia-b_pose_test.csv

The pre-trained models is available at GaitMixer

2. GaitFormer (Spatial-temporal self-attention)

python train.py casia-b ../data/casia-b_pose_train_valid.csv --valid_data_path ../data/casia-b_pose_test.csv --model_type spatiotemporal_transformer

The pre-trained models is available at GaitFormer.

Check experiments/*.sh to see the example of other configurations used in the paper. By default, testing runs every 10 epochs. Use --test_epoch_interval 1 to change number of epochs per testing . See more training options in training doc

Main Results

Top-1 Accuracy per probe angle excluding identical-view cases for the provided models on CASIA-B dataset.

	0	18	36	54	72	90	108	126	144	162	180	mean
NM#5-6	94.4	94.9	94.6	96.3	95.3	96.3	95.3	94.7	95.3	94.7	92.2	94.9
BG#1-2	83.5	85.6	88.1	89.7	85.2	87.4	84.0	84.7	84.6	87.0	81.4	85.6
CL#1-2	81.2	83.6	82.3	83.5	84.5	84.8	86.9	88.9	87.0	85.7	81.6	84.5

Licence & Acknowledgement

GaitMixer itself is released under the MIT License (see LICENSE).

The following parts of the code are borrowed from other projects. Thanks for their wonderful work!

• GaitGraph: tteepe/GaitGraph
• PoseFormer: zczcwh/PoseFormer

---

https://ekkasit.com

Citing GaitMixer

If you find our work useful in your research, please consider citing:

@INPROCEEDINGS{10096917,
  author={Pinyoanuntapong, Ekkasit and Ali, Ayman and Wang, Pu and Lee, Minwoo and Chen, Chen},
  booktitle={ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, 
  title={Gaitmixer: Skeleton-Based Gait Representation Learning Via Wide-Spectrum Multi-Axial Mixer}, 
  year={2023},
  volume={},
  number={},
  pages={1-5},
  doi={10.1109/ICASSP49357.2023.10096917}}

gaitmixer's Issues

How to visualize the heat maps

Excuse me. Thanks for your work, but I wonder how to visualize the heat maps in your paper. Could you please show the code or explain the method about it.

Problems with training

Dear the authors:
I downloaded your code and trained it with 2 GTX1080Ti according to your readme document. But the training effect is much worse. Is there anything wrong with me? Attached is the training results.

           0        18        36        54  ...    144    162    180      mean

NM#5-6 0.937374 0.935354 0.948000 0.942424 ... 0.950 0.948 0.912 0.939596
BG#1-2 0.809091 0.829293 0.826263 0.835714 ... 0.816 0.840 0.787 0.811113
CL#1-2 0.780000 0.810000 0.819000 0.806122 ... 0.827 0.817 0.800 0.808069

About Mparams and GFLOPs of GaitMixer

Hi, it is a great work! I want to know the whole Mparams and GFLOPs of GaitMixer, including the pose estimator. Can you tell me? Thank you very much!!

how to test the acc on own data

Thank you for your outstanding work!I wanted to test the results with my own recorded data, but could not find a demo.

The difference in accuracy of reproduction.

Hi， I just download the data.zip from gaitgraph for casia-b_poses_train_valid.csv and casia-b_valid.csv, then use experiments/train.sh to train model, but the accuracy looks have a bit difference from paper. I have not adjusted any hyperparameters or network structure. How can I adjust it to reproduce this accuracy.
Looking forward to your early reply! Tks！

Program interrupt when multi-GPU training

Hi, it is great work! But I also needed some help. When I run train.py with multiple GPUs, (for example, the "--gpus" parameter is set to "0,1,2,3,4,5,6,7"), my program interrupts but returns no errors. I found that the interrupts occurred in the "loss.backward()" line of code. Can you give me some advice? Thank you very much!!