zhiqiangdon / pose-adv-aug

Code for "Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation" (CVPR 2018)

Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation

Training code for the paper Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation, CVPR 2018

Overview

Traditional random augmentation has two limitations. It doesn't consider the individual difference of training samples when doing augmentation. And it is also independent of the training status of the target network. To tackle these problems, we design an agent to learn more effective data augmentation.

We model the training process as an adversarial learning problem. The agent (generator), conditioning on the individual samples and network status, tries to generate ''hard'' augmentations for the target network. The target network (discriminator), on the other hand, tries to learn better from the augmentations.

Prerequisites

This package has the following requirements:

• Python 2.7
• Pytorch 0.3.0.post4

Installing

Install pytorch:

pip install http://download.pytorch.org/whl/cu90/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl

Install torchvision, scipy, matplotlib, dominate and visdom:

pip install torchvision scipy matplotlib dominate visdom

Training

The training is divided into three stages. First, we pretrain the pose network for 10 epochs. Then we use the fixed pose network to pretrain the augmentation agent. Finally, we jointly optimize these two.

1. Pretrain the Pose Network

python stack-hg.py --gpu_id 0 --exp_id stack-2-hgs --vis_env stack-2-hgs --is_train true --bs 24

2. Pretrain the Augmentation Agent

Use the pose network to collect the scale and rotation distributions to train the agent:

python collect-scale-ditri.py --gpu_id 0 --exp_id stack-2-hgs --load_prefix_pose lr-0.00025-10.pth.tar --bs 10

python collect-rotation-ditri.py --gpu_id 0 --exp_id stack-2-hgs --load_prefix_pose lr-0.00025-10.pth.tar --bs 10

Pretrain the agent:

python pretrain-s-r-agent.py --gpu_id 0 --exp_id stack-2-hgs --load_prefix_pose lr-0.00025-10.pth.tar --bs 24

3. Jointly Train the Pose Network and Agent

python joint-train-pose-s-r-agent.py --gpu_id 0 --exp_id stack-2-hgs --load_prefix_pose lr-0.00025-10.pth.tar --load_prefix_sr lr-0.00025-1.pth.tar --vis_env stack-2-hgs-joint --is_train true --bs 24 

Citation

If you find this code useful in your research, please consider citing:

@inproceedings{peng2018jointly,
  title={Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation},
  author={Peng, Xi and Tang, Zhiqiang and Yang, Fei and Feris, Rogerio S and Metaxas, Dimitris},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2226--2234},
  year={2018}
}

pose-adv-aug's Issues

use

mpii-hr-lsp-normalizer.json origin?

Hi I was hoping if you could shed some light on were you got the json file? and if you are the one created it. Do you mind explaining a couple of things to me?

• how did you compute the scale for the Leeds dataset?
• What is the function/objective of the 'normalizer' field in the json file?

Thanks,

use another baseline like openpose?

你好，我的baseline是openpose，我想使用您的方法来进行对coco的自动的数据增强，请问您这个代码好移植吗？

RuntimeError: bool value of non-empty torch.ByteTensor objects is ambiguous

scale gaussian number is 7
scale means: [ -6.00000000e-01 -4.00000000e-01 -2.00000000e-01 1.11022302e-16
2.00000000e-01 4.00000000e-01 6.00000000e-01]
loading json file is done...
invalid center: [-26.0, 356.0]
image name: images-mpii/097266879.jpg
dataset: MPII
invalid center: [1718.0, -157.0]
image name: images-mpii/012545809.jpg
dataset: MPII
Traceback (most recent call last):
File "/media/z/CC/zxl/2Dhuman pose estimation-CX/pose-adv-aug/collect-scale-ditri.py", line 305, in
main()
File "/media/z/CC/zxl/2Dhuman pose estimation-CX/pose-adv-aug/collect-scale-ditri.py", line 84, in main
train_distri_path_2, hg, opt, is_train=True)
File "/media/z/CC/zxl/2Dhuman pose estimation-CX/pose-adv-aug/collect-scale-ditri.py", line 115, in collect_train_valid_data
grnd_distri_list = read_grnd_distri_from_txt(save_path)
File "/media/z/CC/zxl/2Dhuman pose estimation-CX/pose-adv-aug/collect-scale-ditri.py", line 290, in read_grnd_distri_from_txt
if k<0.1:
File "/home/z/anaconda2/envs/xlpy030/lib/python2.7/site-packages/torch/tensor.py", line 152, in bool
" objects is ambiguous")
RuntimeError: bool value of non-empty torch.ByteTensor objects is ambiguous
total training images: 25925（错误处）

Process finished with exit code 1

'Namespace' object has no attribute 'resume_prefix_pose'

Hello!

Thanks for sharing this great work! I would like to implement it in my research. But, I had faced an error after installing prerequisites.
Traceback (most recent call last): File "stack-hg.py", line 269, in <module> main() File "stack-hg.py", line 30, in main opt = TrainOptions().parse() File "$HOME/pose-adv-aug/options/base_options.py", line 62, in parse if self.opt.resume_prefix_pose != '': AttributeError: 'Namespace' object has no attribute 'resume_prefix_pose'
I think the error is internal, I guess. Can you shred some light on this issue? Thanks in advance..

About training dataset

Hi @zhiqiangdon
Thank your for sharing your nice work.
I want to cite your work and need some details.

In your paper
"Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation",
Is the table 3's result trained using the MPII+LSP dataset or Only LSP dataset?

why is ZeroDivisionError: float division by zero when i train LSP?

Traceback (most recent call last):
File "/media/pose-adv-aug/collect-scale-ditri.py", line 632, in
main()
File "/media/pose-adv-aug/collect-scale-ditri.py", line 393, in main
train_distri_path_2, hg, opt, is_train=True)
File "/media/pose-adv-aug/collect-scale-ditri.py", line 428, in collect_train_valid_data
collect_data(collect_data_loader, dataset, hg, save_path)
File "/media/pose-adv-aug/collect-scale-ditri.py", line 537, in collect_data
grnd_pts, normalizer, rotation)
File "/media/pose-adv-aug/pylib/Evaluation.py", line 366, in per_person_pckh
dists = calc_dists(pred_pts, grnd_pts, normalizers, use_zero=True)
File "/media/pose-adv-aug/pylib/Evaluation.py", line 291, in calc_dists
dists[c, n] = torch.dist(preds[n, c, :], target[n, c, :]) / normalize[n]
ZeroDivisionError: float division by zero