yuliangxiu / poseflow

PoseFlow: Efficient Online Pose Tracking (BMVC'18)

Pose Flow

Official implementation of Pose Flow: Efficient Online Pose Tracking .

Results on PoseTrack Challenge validation set:

1. Task2: Multi-Person Pose Estimation (mAP)

Method	Head mAP	Shoulder mAP	Elbow mAP	Wrist mAP	Hip mAP	Knee mAP	Ankle mAP	Total mAP
Detect-and-Track(FAIR)	67.5	70.2	62	51.7	60.7	58.7	49.8	60.6
AlphaPose	66.7	73.3	68.3	61.1	67.5	67.0	61.3	66.5

2. Task3: Pose Tracking (MOTA)

Method	Head MOTA	Shoulder MOTA	Elbow MOTA	Wrist MOTA	Hip MOTA	Knee MOTA	Ankle MOTA	Total MOTA	Total MOTP	Speed(FPS)
Detect-and-Track(FAIR)	61.7	65.5	57.3	45.7	54.3	53.1	45.7	55.2	61.5	Unknown
PoseFlow(DeepMatch)	59.8	67.0	59.8	51.6	60.0	58.4	50.5	58.3	67.8	8
PoseFlow(OrbMatch)	59.0	66.8	60.0	51.8	59.4	58.4	50.3	58.0	62.2	24

Latest Features

• Dec 2018: PoseFlow(General Version) released! Support ANY DATASET and pose tracking results visualization.
• Oct 2018: Support generating correspondence files with ORB(OpenCV), 3X FASTER and no need to compile DeepMatching library.

Requirements

• Python 2.7.13
• OpenCV 3.4.2.16
• OpenCV-contrib 3.4.2.16
• tqdm 4.19.8

Installation

1. Download PoseTrack Dataset from PoseTrack to AlphaPose/PoseFlow/posetrack_data/
2. (Optional) Use DeepMatching to extract dense correspondences between adjcent frames in every video, please refer to DeepMatching Compile Error to compile DeepMatching correctly

pip install -r requirements.txt

cd deepmatching
make clean all
make
cd ..

For Any Datasets (General Version)

1. Using AlphaPose to generate multi-person pose estimation results.

# pytorch version
python demo.py --indir ${image_dir}$ --outdir ${results_dir}$

# torch version
./run.sh --indir ${image_dir}$ --outdir ${results_dir}$

2. Run pose tracking

# pytorch version
python tracker-general.py --imgdir ${image_dir}$ 
                          --in_json ${results_dir}$/alphapose-results.json 
                          --out_json ${results_dir}$/alphapose-results-forvis-tracked.json
                          --visdir ${render_dir}$

# torch version
python tracker-general.py --imgdir ${image_dir}$ 
                          --in_json ${results_dir}$/POSE/alpha-pose-results-forvis.json 
                          --out_json ${results_dir}$/POSE/alpha-pose-results-forvis-tracked.json
                          --visdir ${render_dir}$

For PoseTrack Dataset Evaluation (Paper Baseline)

1. Using AlphaPose to generate multi-person pose estimation results on videos with format like alpha-pose-results-sample.json.
2. Using DeepMatching/ORB to generate correspondence files.

# Generate correspondences by DeepMatching
# (More Robust but Slower)
python matching.py --orb=0 

or

# Generate correspondences by Orb
# (Faster but Less Robust)
python matching.py --orb=1

3. Run pose tracking

python tracker-baseline.py --dataset=val/test  --orb=1/0

4. Evaluation

Original poseval has some instructions on how to convert annotation files from MAT to JSON.

Evaluate pose tracking results on validation dataset:

git clone https://github.com/leonid-pishchulin/poseval.git --recursive
cd poseval/py && export PYTHONPATH=$PWD/../py-motmetrics:$PYTHONPATH
cd ../../
python poseval/py/evaluate.py --groundTruth=./posetrack_data/annotations/val \
                    --predictions=./${track_result_dir}/ \
                    --evalPoseTracking --evalPoseEstimation

Citation

Please cite these papers in your publications if it helps your research:

@inproceedings{xiu2018poseflow,
  author = {Xiu, Yuliang and Li, Jiefeng and Wang, Haoyu and Fang, Yinghong and Lu, Cewu},
  title = {{Pose Flow}: Efficient Online Pose Tracking},
  booktitle={BMVC},
  year = {2018}
}

poseflow's Issues

Do i find a bug in the code?

firstly, thank you for your fantastic project, and i learning a lot from you task.
I read your code seriously, and i thank there have a bug in your code.
I think ret = fd.write("%d %d %d %d %f \n"%(kp1[m_n[0].queryIdx].pt[0], kp1[m_n[0].queryIdx].pt[**_0_**], kp2[m_n[0].trainIdx].pt[0], kp2[m_n[0].trainIdx].pt[1], m_n[0].distance)) in matching.py should be corrected toret = fd.write("%d %d %d %d %f \n"%(kp1[m_n[0].queryIdx].pt[0], kp1[m_n[0].queryIdx].pt[**_1_**], kp2[m_n[0].trainIdx].pt[0], kp2[m_n[0].trainIdx].pt[1], m_n[0].distance)).
"Yuan Fang", how do you see?

OKS VS. PCKh

Hi @YuliangXiu @Jeff-sjtu ,

May I ask where the below is used? Thx & regards,

PoseFlow/utils.py

Line 81 in 4b2f266

def compute_oks(anno, predict, delta):

Tracking and speed issues

I have a video from a sports game and I want to track players.
I used AlphaPose to generate pose data.
I have two issues with PoseFlow:

1. Speed.
   On my Windows PC it does not use GPU and a 23 sec video (1024x576, 24 fps) takes more than 10 hours to process (with Intel i5 4x4.2GHz CPU).
   Is this normal? Can it run on GPU? Would it run better on Linux with GPU?

For comparison, I wrote a simple script which does the same task with a simple algorithm, - it finds the closest person from previous frames, and if the distance is below threshold, it considers it the same person. It takes less than 1 second for the mentioned above video. And, actually, resulting data is not much worse than PoseFlow.

2. There're many issues with tracking. The basic one - often a person runs across the court, no jumps or anything, and suddenly it's another person. I run PoseFlow with default parameters, are there any parameters that can help?
   A worse issue is that often PoseFlow reassigns IDs ("idx") to different people. Part of a game a player # 2 is in the white shirt, in the left corner, another part it considers # 2 another player, in black shirt, in the right corner. It completely messes up my research .. Is there anything can be done?

Another question about usage, not that important, in the doc it is said that to use PoseFlow I need to download PoseTrack dataset, is this really required? It is 86Gb and I did not download it, PoseFlow works without it.

Thank you!

tracking result

Hi~,Thanks for your codes~.
I have a question about the tracking result.

after run the PoseFlow, I got the tracked keypoints with only 15 points. I found the new partsid is as follow, so id 6,7 were removed?
rmpe_part_ids = [0, 1, 2, 3, 4, 5, 10, 11, 12, 13, 14, 15, 8, 9]

could you tell me why?
Thank you

Tracked output JSON format?

I just want to confirm that the idx key is the value that persists across each image/frame to track individuals? As in, the person identified in frame 1 with the idx value of 1 is should be the same person identified in frame 2 to n with the idx value of 1? I don't see a spec for the JSON output format anywhere.

about the prediction

我有一些相关问题想请教一下,第一.我用alphapose-pytorch在我自己的数据上生成的关键点数据能否直接拿过来做姿态追踪的预测(我的数据是没有做标签,只是想做一下预测看一下算法的泛化性能)?如果是可行的话,那代码中的标签数据该怎么样设置呢?

How to evaluate posetrack dataset with `tracker.py`?

@YuliangXiu I want to evaluate pose track performance on posetrack-val2018 dataset and I've read your reply in this issue, where you just mentioned:

you can obtain the pose_coordinates from trainset and validset of PoseTrack dataset(json or mat files in posetrack_data/annotations). Otherwise, you need to generate pose_coordinates via Multi-Person Pose Estimation Framework, such as AlphaPose.

If I was right, then evaluating on posetrack dataset no need to do pose estimation beforehead?
Then I found this line in 'tracker.py'. Seems that it is a result json file generated by AlphaPose.

So here is my question:

1. tracker.py is a general solution for unlabeled data? and
2. If I want to evaluate the origin posetrack dataset, I need to modify tracker.py?

关于结果的请教

想问下您，我得到了结果的.json 文件，我怎么看或者保存的追踪过的结果图片呢？
还有就是能根据 alphapose处理的视频结果+poseflow 得到追踪的视频结果吗

about runtime

@YuliangXiu first of all, thank you for sharing your code that give me a chance to learn about it. In you paper you said "our method can track poses at 10 FPS" , could you tell me your GPU type about this paper??? thanks again, I am looking for your reply

three question about poseflow

hello, i have read the code and paper and i have three questions.
1.Can i use poseflow after alphapose of pytorch branch?
2.In your poseflow code, what is the drop threshold mean，it defaults to 2 and how to take the value？
3.I find that poseflow relies heavily on the results of alphapose，if alphapose does not detect keypoints in a frame, the result of tracking may be wrong.So do you have some ideas for correcting the keypoints of bigger error？
Please have a look，thank you

about tracking time

hi~
I have a question about the tracking time.
As you mentioned in paper , the tracking time is 10FPS, but when I run the demo code it is very slow, is it depended on the number of persons ?
Thanks~

JSON files for posetrack validation videos.

In line 66, of the tracker-baseline.py file, it reads a single file alpha-pose-results-val.json. So do we need to make a single file for all 50 videos from alphapose code?

notrack_json = "alpha-pose-results-{}.json".format(dataset) 

If so could you please clarify, how to do that, since alphapose only works for images in a single folder.

Installing in Jetson Nano?

I'm trying to install it in NVidia's Jetson Nano. But when I run the make clean all line, I get this error:

immintrin.h: No such file or directory
#include <immintrin.h>

How do I get past this?

Running Time Very Slow

Hi,

I tried your codes, for a 4min video (24fps), after I got results from alphapose, when I ran the poseflow, it's extremely slow. Even the time of loading alphapose is long, about 3mins. And the total tracking time is at least 2 hours... I wonder why this happens. You said the running time is 24fps for orb version. Please help. thanks a lot!

有一个幼稚的问题想要请教

你好，这两天拜读了你的这篇论文并看了你的代码。但有一个很简单的问题一直没有相通。希望能帮忙解答一下。不胜感激。
在 readme 的 Quick Start部分，首先需要运行alpha pose来生成一个video的每帧的human keypoint，然后运行 track.py。这里我一直没想明白。
alpha pose已经把每帧的human keypoint预测出来了。那么为什么还需要 track呢？直接把每帧预测出来的human keypoint绘制到屏幕上去，不就已经可以了吗

A question about pose track

About PoseTrack ECCV 2018 Challenge, how to generate the pose data just for testing Multi-Person Pose Tracking? Is it generated by Multi-Person Pose Estimation or just extracted pose_coordinate from /posetrack2018/posetrack_data/annotations/jsons? If only for evaluating Multi-Person Pose Tracking, should the original pose data be labeled?

How to use tracker.py on a video?

I already get the pose estimations of a video by AlphaPose according to the QuickStart. But I don't know how can I use tracker.py to get track ID with a JSON file and a video. Should I convert the video to images first?

疑问

你好，非常感谢你的代码的开源。我跑了一视频文件，发现关键点在pose-flow处理的坐标位置没有变化，我想请教一下如下问题


假定AlphaPose生成了一系列关键点json文件。
待解决问题：

1. Alpha Pose 生成的 json文件中每一帧的每个人只会有一个边框， 一套相应的关键点？

2. Pose Flow 处理json 文件

追踪匹配每个人在视频中连续性
是否调节了human bbox
是否调节了keypoints的位置

Google Colab demo with video input

Thank you for this repository.

I am trying to get a AlphaPose+PoseFlow workflow on videos to run on Google Colab. The AlphaPose part is fine - I'm getting the alphapose-results.json.

However, how do I run PoseFlow on it (since I don't have a image directory -just a video - and image_dir is a necessary argument for tracker-general.py)

My notebook is here.

Action recognition on own video

Hello,

Thank your for releasing the code for your paper. I'd like to be able to obtain action recognition results on my own video that contains a group of people walking around. Can you point me to any repos that use the results of this PoseFlow code to produce action recognition classification results?

Thank you,

question about the code and paper.

I read the code and related paper, and have a question :this code does not contain the process of Pose Flow NMS ?

Errors in building the DeepMatching Library

Can you please upload a generic version of the makefile or any specific step s to edit the makefile for compiling the deepmatching library?

fff versus non-fff weighting in Hungarian algorithm

From my understanding of

PoseFlow/utils.py

Line 157 in 4d2a78d

def best_matching_hungarian(all_cors, all_pids_info, all_pids_fff, track_vid_next_fid, weights, weights_fff, num, mag):

, this codebase performs pose tracking by matching bboxes & associated poses between frames based on a weighted sum of various distance metrics elaborated from object- and keypoint-level bbox IoUs as well as the shared ORB features contained within these bboxes.

It also seems like the bitartite matching graph is between a) all bboxes in frame t+1 and b) the most recent bboxes associated with each of the object tracks discovered thus far ( in frames <= t) based on https://github.com/MVIG-SJTU/AlphaPose/blob/9dafbb2259a36bf92df87042863eba7f543d3cf9/PoseFlow/utils.py#L93.

I was wondering why these distance metrics are weighted differently (

PoseFlow/tracker-general.py

Line 112 in 4d2a78d

weights = [1,2,1,2,0,0]

) depending on whether the boxes in b) were from frame t or not? It seems like the ORB-based matching criteria are given weight 0 for b) boxes from frame t and 1 otherwise.

代码内容与论文一致性

粗略看了代码，不论是general版本还baseline版本，计算过程都和论文不一样，想请教一下，代码是和论文不一致的吗？或者是我没有看到，可以麻烦指出论文实现部分在哪个位置吗？