🚩 [Update] The face part of PyMAF-X has been updated. See face-only evaluation results on the NoW benchmark.
Hongwen Zhang · Yating Tian · Yuxiang Zhang · Mengcheng Li · Liang An · Zhenan Sun · Yebin Liu
Project Page | Video | Paper
Frame by frame reconstruction. Video clipped from here.
Reconstruction result on a COCO validation image.
Click Here for More Results
- Python 3.8
conda create --no-default-packages -n pymafx python=3.8
conda activate pymafx
- PyTorch tested on version 1.9.0
conda install pytorch==1.9.0 torchvision==0.10.0 cudatoolkit=11.1 -c pytorch -c conda-forge
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
- other packages listed in
requirements.txt
pip install -r requirements.txt
smpl_downsampling.npz & mano_downsampling.npz
- Run the following script to fetch necessary files.
bash fetch_data.sh
SMPL & SMPL-X model files
- Collect SMPL/MANO/FLAME/SMPL-X model files. Rename model files and put them into the
./data/smpldirectory.
Download the partial_mesh files and put it into the
./data/partial_meshdirectory.
Download the pre-trained model and put it into the
./data/pretrained_modeldirectory.
After collecting the above necessary files, the directory structure of ./data is expected as follows.
./data
├── J_regressor_extra.npy
├── smpl_mean_params.npz
├── smpl_downsampling.npz
├── mano_downsampling.npz
├── flame_downsampling.npy
├── partial_mesh
│ └── ***_vids.npz
├── pretrained_model
│ └── PyMAF-X_model_checkpoint_v1.1.pt
└── smpl
├── FLAME2020
│ ├── FLAME_NEUTRAL.pkl
│ ├── flame_dynamic_embedding.npy
│ └── flame_static_embedding.pkl
├── MANO_RIGHT.pkl
├── SMPLX_NEUTRAL_2020.npz
├── SMPL_NEUTRAL.pkl
└── model_transfer
├── MANO_SMPLX_vertex_ids.pkl
├── SMPL-X__FLAME_vertex_ids.npy
└── smplx_to_smpl.pkl
You can first give it a try on Google Colab using the notebook we have prepared, which is no need to prepare the environment yourself:
Run the demo code.
python -m apps.demo_smplx --image_folder examples/coco_images --detection_threshold 0.3 --pretrained_model data/pretrained_model/PyMAF-X_model_checkpoint_v1.1.pt --misc TRAIN.BHF_MODE full_body MODEL.PyMAF.HAND_VIS_TH 0.1
python -m apps.demo_smplx --vid_file examples/dancer_short.mp4 --pretrained_model data/pretrained_model/PyMAF-X_model_checkpoint_v1.1.pt --misc TRAIN.BHF_MODE full_body MODEL.PyMAF.HAND_VIS_TH 0.1
Results will be saved at ./output. You can set different hyperparamters in the scripts, e.g., --detection_threshold for the person detection threshold and MODEL.PyMAF.HAND_VIS_TH for the hand visibility threshold.
To perform training, we need to collect preprocessed files of training datasets first. The pseudo SMPL-X labels (with keys of 'xpose'/'xshape') can be downloaded at here. Please also refer to PyMAF for more details about training. Example usage:
python -m apps.train --regressor pymaf_net --train_data h36m_coco_itw --eval_every 10 --save_every 20 --train_data h36m_coco_itw --misc TRAIN.BATCH_SIZE 64 MODEL.PyMAF.AUX_SUPV_ON True MODEL.PyMAF.TRANS.USE_ATT True MODEL.PyMAF.TRANS.ATT_HEAD 1 MODEL.PyMAF.TRANS.ATT_FEAT_IDX 2 MODEL.MESH_MODEL smplx TRAIN.USE_EFT True MODEL.PyMAF.USE_CAM_FEAT True LOSS.SHAPE_W 0.6 MODEL.PyMAF.BACKBONE res50 POSE_RES_MODEL.PRETR_SET coco
If this work is helpful in your research, please cite the following papers.
@article{pymafx2023,
title={PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images},
author={Zhang, Hongwen and Tian, Yating and Zhang, Yuxiang and Li, Mengcheng and An, Liang and Sun, Zhenan and Liu, Yebin},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
year={2023}
}
@inproceedings{pymaf2021,
title={PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop},
author={Zhang, Hongwen and Tian, Yating and Zhou, Xinchi and Ouyang, Wanli and Liu, Yebin and Wang, Limin and Sun, Zhenan},
booktitle={Proceedings of the IEEE International Conference on Computer Vision},
year={2021}
}
Part of the code is borrowed from the following projects, including DaNet, SPIN, VIBE, SPEC, MeshGraphormer, PIFu, DensePose, HMR, HRNet, pose_resnet. Many thanks to their contributions.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent