kenziyuliu / dgnn-pytorch

Unofficial PyTorch implementation of the CVPR'19 paper "Skeleton-Based Action Recognition with Directed Graph Neural Networks".

DGNN-PyTorch

An unofficial PyTorch implementation of the paper "Skeleton-Based Action Recognition with Directed Graph Neural Networks" in CVPR 2019.

NOTE: Experiment results are not being updated due to hardware limits.

• Paper: PDF
• Code is based on 2s-AGCN: GitHub

Dependencies

• Python >= 3.5
• scipy >= 1.3.0
• numpy >= 1.16.4
• PyTorch >= 1.1.0
• tensorboardX >= 1.8 (For logging)

Directory Structure

Most of the interesting stuff can be found in:

• model/dgnn.py: model definition of DGNN
• data_gen/: how raw datasets are processed into numpy tensors
• graphs/directed_ntu_rgb_d.py: graph definition for DGNN
• feeders/feeder.py: how datasets are read in
• main.py: general training/eval processes; graph freezing by disabling gradients; etc.

Downloading & Generating Data

NTU RGB+D

1. The NTU RGB+D dataset can be downloaded from here. We'll only need the Skeleton data (~ 5.8G).

2. After downloading, unzip it and put the folder nturgb+d_skeletons to ./data/nturgbd_raw/.

3. Generate the joint dataset first:

cd data_gen
python3 ntu_gen_joint_data.py

Specify the data location if the raw skeletons data are placed somewhere else. The default looks at ./data/nturgbd_raw/.

4. Then, in data_gen/, generate the bone dataset:

python3 ntu_gen_bone_data.py

5. Finally, generate the motion data from joints/bones:

python3 ntu_gen_motion_data.py

The generation scripts look for generated data in previous step. By default they look at ./data; change dir configs if needed.

Kinetics

(Currently, generating bone/motion data from Kinetics skeletons is not yet supported. Please feel free to add scripts based on kinetics_gendata.py)

1. Download the Kinetics dataset from ST-GCN repo (https://github.com/yysijie/st-gcn)
2. Generate joint data:

cd data_gen
python3 kinetics_gendata.py

3. Generate bone data: TODO, feel free to fork/submit PR :D
4. Generate motion data: TODO, feel free to fork/submit PR :D

Training

1st Stream: Spatial

To start training the network with the spatial stream, use the following command:

python3 main.py --config ./config/<dataset>/train_spatial.yaml

Here, <dataset> should be one of nturgbd-cross-subject, nturgbd-cross-view, or kinetics-skeleton depending on the dataset/task on which to train the model.

Note: At the moment, only nturgbd-cross-subject is supported. More config files will (hopefully) be added, or you could write your own config file using the existing ones for nturgbd-cross-subject.

2nd Stream: Motion

Similarly, to train on the motion stream data, do:

python3 main.py --config ./config/nturgbd-cross-subject/train_motion.yaml

and change the config file path for other datasets if needed.

Testing

Test individual streams

To test some model weights (by default saved in ./runs/), do:

python3 main.py --config ./config/<dataset>/test_spatial.yaml

Similarly, change the paths in config file, or change the config files (<dataset>) for different datasets as needed.

Ensemble results

Combine the generated scores with:

python ensemble.py --datasets <dataset>

where <dataset> is one of kinetics, ntu/xsub, ntu/xview

TODO

• Kinetics
  • Handling datasets
  • Config files

dgnn-pytorch's Issues

Requirement for NTU-RGBD datasets

Because the official website has been in error, can I get the NTU-RGBD dataset from baidu disk/ dropbox? email: [email protected]

_pickle.UnpicklingError: pickle data was truncated

When I run python main.py --config ./config/nturgbd-cross-view/train_joint.yaml.py ，gets the error：_pickle.UnpicklingError: pickle data was truncated！
Need your help！

您好，想请教一下关于网络的输出

我有点不太明白，网络是处理骨骼的三维坐标点转成的图数据，那输出为什么不是坐标点，怎么恢复成原来的坐标点形式？感谢！

What does "Time consumption: [Data] 1%, [Network] 99%" mean?

Thanks for your codes.
It takes me half an hour to train an epoch, and I get the "Time consumption: [Data] 1%, [Network] 99%" after an epoch.
Does it mean that I can somehow reduce the "Network"-time to accelerate training process?

请问您为什么你的关节预处理的结果 跟2sagcn的差别很大 不一样为什么

May I ask why your joint pretreatment results are very different from 2sagcn? Why?

Question about miss './run/'

To test some model weights (by default saved in ./runs/); However, there is no run folder in the project. Can you share the model weight; Thanks!!!!

How can I test on a video?

Hi，I tried to test a single video and output its corresponding action class. Can you tell me what should I do?Thank you .

FileNotFoundError: No such file or directory: '../data/kinetics_raw/kinetics_val_label.json'

您好，请问为什么会出现文件找不到的情况呀？我按照你的指令一步步执行，最后运行
python3 kinetics_gendata.py

出现了错误

why the training accuracy is so low ,about70%

the accuracy of spacial is about 70%,the accuracy of motion is lower 10%,this is just the training accuracy

How to output action labels during testing?

Hi，I want to output the corresponding action label for each video when testing. Which part of the code should I modify?Thank you very much.

Problem with fv.view(N, -1, V_node) function in dggn.py

Hello, I'm getting the following error while trying to run model training, with part of NTU RGB+D dataset:

Model total number of params: 4089320
[ Fri Feb 21 11:48:03 2020 ] Parameters:
{'work_dir': './work_dir/ntu/xsub/dgnn_spatial', 'model_saved_name': './runs/ntu_cs_dgnn_spatial', 'config': './config/nturgbd-cross-subject/train_spatial.yaml', 'phase': 'train', 'save_score': False, 'seed': 1, 'log_interval': 100, 'save_interval': 2, 'eval_interval': 5, 'print_log': True, 'show_topk': [1, 5], 'feeder': 'feeders.feeder.Feeder', 'num_worker': 64, 'train_feeder_args': {'joint_data_path': './data/ntu/xsub/train_data_joint.npy', 'bone_data_path': './data/ntu/xsub/train_data_bone.npy', 'label_path': './data/ntu/xsub/train_label.pkl', 'debug': False, 'random_choose': False, 'random_shift': False, 'random_move': False, 'window_size': -1, 'normalization': False}, 'test_feeder_args': {'joint_data_path': './data/ntu/xsub/val_data_joint.npy', 'bone_data_path': './data/ntu/xsub/val_data_bone.npy', 'label_path': './data/ntu/xsub/val_label.pkl'}, 'model': 'model.dgnn.Model', 'model_args': {'num_class': 60, 'num_point': 25, 'num_person': 2, 'graph': 'graph.directed_ntu_rgb_d.Graph'}, 'weights': None, 'ignore_weights': [], 'base_lr': 0.1, 'step': [60, 90], 'device': [0], 'optimizer': 'SGD', 'nesterov': True, 'batch_size': 1, 'test_batch_size': 1, 'start_epoch': 0, 'num_epoch': 60, 'weight_decay': 0.0005, 'freeze_graph_until': 10}
[ Fri Feb 21 11:48:03 2020 ] Training epoch: 1
[ Fri Feb 21 11:48:03 2020 ] Graphs are frozen at epoch 1
  0%|          | 0/60 [00:00<?, ?it/s]Traceback (most recent call last):
  File "D:/dev/DGNN-PyTorch/main.py", line 606, in <module>
    processor.start()
  File "D:/dev/DGNN-PyTorch/main.py", line 550, in start
    self.train(epoch, save_model=save_model)
  File "D:/dev/DGNN-PyTorch/main.py", line 398, in train
    output = self.model(batch_joint_data, batch_bone_data)
  File "D:\dev\DGNN-PyTorch\venv\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\dev\DGNN-PyTorch\model\dgnn.py", line 184, in forward
    fv, fe = self.l2(fv, fe)
  File "D:\dev\DGNN-PyTorch\venv\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\dev\DGNN-PyTorch\model\dgnn.py", line 123, in forward
    fv, fe = self.dgn(fv, fe)
  File "D:\dev\DGNN-PyTorch\venv\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\dev\DGNN-PyTorch\model\dgnn.py", line 85, in forward
    fv = fv.view(N, -1, V_node)
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
  0%|          | 0/60 [00:57<?, ?it/s]
Process finished with exit code 1

Is it problem with PyTorch version (I'm using 1.4.0) or may it be problem with size of dataset? I've tried to load the whole dataset, but then I have another issue with too big size of data.

Not really adaptive graph

Recently, I have go through the code and find that in this implementation, the number of the edges is still fixed. But if we input all the possible edges into the network of 9 layers, the memory will boom. I wonder whether there are some good ideas to use all possible edges and can stack many layers too.

Motion data is used in kinetics-dataset?

When I see config files, I found that there are two yamls : spatial and motion for NTU and joint and bone for Kinetics.
So, why do not use motion data in kinetics yaml config file?
But, in your paper, you compared three methods for both NTU and Kinetics : spatial, motion and fusion for both dataset.
Please explain to me. Thank you.

Problems on preprocess.py

How can I train on selected classes?

How can I train the DGNN on selected number of classes using NTU dataset instead in training it on all the data?

bone_data.npy file for Kinetics?

Could you provide me bone_data.npy file for kinetics data? Because when I extract this file, I got segmentation error.

请问一下 我跑出的准确率为什么特别低 才80左右？空间流

Excuse me, why is the accuracy rate of my running out so low? Only about 80? Space flow

How to decide to choose edge link using your scenario?

You chose edge link depends on your scenario. But why do you use this scenario to reduce edge links except for computational cost. Why not other links (total edge links for 16 joints can get 120 edge lines)?

你好，想请问一下运行代码碰到pickle data was truncated如何处理？

你好，想请问一下当我在运行python3 main.py --config ./config/nturgbd-cross-view/train_spatial.yaml的时候，出现pickle data was truncated的错误，请问该如何处理？

Why ensemble accuracies are unstable?

When I ensemble joint and bone score for every epoch, their accuracy is varying.
I mean although same epoch score, different accuracy result.
Why happened like that and how to fix it?

Can this model be trained and tested using 2D pose?

I hope to train and test this model with 2D pose data, so that I can better integrate openpose data

你好 kinetics-skeleton数据集是不是要用agcn.py

Hello, I'm getting the following error while trying to run model test, with part of NTU RGB+D dataset:

Can this code achieve the accuracy of the paper?

Can this code achieve the accuracy of the paper?

Thank you for your selfless dedication! Have you applied directed graphs to the Kinetics dataset?

Doubts regarding pre-processing of model

Hey, firstly thanks a lot for implementing and making the code public. I am trying to replicate the results on NTU 120 dataset but I am unable to get good accuracies for motion stream. I re-read the paper and found out that I had missed a pre-processing step mentioned below:

The body tracker of Kinect is prone to detecting
more than 2 bodies, some of which are objects. To filter the
wrong bodies, we first define the energy of each bodies as
the summation of the skeleton’s standard deviation across
each channel. We then select two bodies in each sample
according to their body energies. Subsequently, each sam-
ple is normalized and translated to the central perspective,
which is the same approach as that used earlier.

1. In your code for pre-processing of data, I can't find this line. So, is this required for replicating the results or we can still achieve great accuracies irrespective of this preprocessing step?

2. While testing the spatial and motion streams, is it important that we test on the model trained at the last epoch (50th ) or we can use the model checkpoint giving the best accuracy?