lizhe00 / posevocab

Code of [SIGGRAPH 2023] "PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling"

PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling

SIGGRAPH 2023

Zhe Li, Zerong Zheng, Yuxiao Liu, Boyao Zhou, Yebin Liu

Tsinghua Univserity

Projectpage · Paper · Video

Introduction

We propose PoseVocab, a novel pose encoding method that encodes dynamic human appearances under various poses for human avatar modeling.

teaser.mp4

Installation

Clone this repo, then run the following scripts.

cd ./utils/posevocab_custom_ops
python setup.py install
cd ../..

SMPL-X & Pretrained Models

• Download SMPL-X files, place pkl files to ./smpl_files/smplx.
• Download pretrained models, unzip it to ./pretrained_models.

Run on THuman4.0 Dataset

Dataset Preparation

• Download THuman4.0 dataset. Let's take "subject00" as an example, and denote the root data directory as SUBJECT00_DIR.
• Specify the data directory and training frame list in gen_data/main_preprocess.py, then run the following scripts.

cd ./gen_data
python main_preprocess.py
cd ..

Training

Note: In the first training stage, our method reconstructs depth maps for the depth-guided sampling in the next stages. If you want to skip the first stage, you can download our provided depth maps from this link, unzip it to SUBJECT00_DIR/depths, and directly run python main.py -c configs/subject00.yaml -m train until the network converges.

• Stage 1: training to obtain depth maps. Set end_epoch in configs/subject00.yaml#L15 to 10.

python main.py -c configs/subject00.yaml -m train

• Stage 2: render depth maps.

python main.py -c configs/subject00.yaml -m render_depth_sequences

• Stage 3: continue training. Set start_epoch in configs/subject00.yaml#L14 to 11, end_epoch in configs/subject00.yaml#L15 to 100, and prev_ckpt in configs/subject00.yaml#L12 to ./results/subject00/epoch_latest.

python main.py -c configs/subject00.yaml -m train

Testing

Download testing poses from this link, unzip them to somewhere, denoted as TESTING_POSE_DIR.

• Specify prev_ckpt in configs/subject00.yaml#L78 as the pretrained model ./pretrained_models/subject00 or the trained one by yourself.
• Specify data_path in configs/subject00.yaml#L60 as the testing pose path, e.g., TESTING_POSE_DIR/thuman4/pose_01.npz.
• Run the following script.

python main.py -c configs/subject00.yaml -m test

• The output results can be found in ./test_results/subject00.

License

MIT License. SMPL-X related files are subject to the license of SMPL-X.

Citation

If you find our code or paper is useful to your research, please consider citing:

@inproceedings{li2023posevocab,
  title={PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling},
  author={Li, Zhe and Zheng, Zerong and Liu, Yuxiao and Zhou, Boyao and Liu, Yebin},
  booktitle={ACM SIGGRAPH Conference Proceedings},
  year={2023}
}

posevocab's Issues

Inference time per frame

Hi, thank you for the awesome work! I would like to ask what is the inference time of it? Can it achieve real-time performance?

hello, is the possible convert the input 3d keypoints to SMPL rotations?

hello, is the possible convert the input 3d keypoints to SMPL rotations?

Run on ZJU-MoCap Dataset

Will you provide the scripts about running on ZJU-MoCap?

question about dataset

Hi!!! This is a wonderful work!
And more, in your paper, you mentioned the following dataset:THuman4.0,DeepCap and ZJU-MoCap,but you only gave a description of how to use the THuman4.0 dataset. Will you publish the processing methods of other datasets, or will the other datasets be treated the same as the THuman 4.0 datasets?
Best wish!!

live_bounds parameter seems incorrect and cannot be correctly bound

Great job. I tried the CMU dataset and did SMPL fitting well, but during training, I couldn't do it. After checking, I found that live_ Bounds cannot be drawn correctly. Can you tell me what this is related to? Camera calibration?
my error：
File "/project-ssd/PoseVocab/dataset/dataset_mv_rgb_slrf.py", line 257, in getitem
nerf_random = nerf_util.sample_randomly_for_nerf_rendering(color_img, mask_img, depth_img, self.extr_mats[view_idx], self.intr_mats[view_idx], live_bounds, unsample_region_mask = boundary_mask_img, **ray_sampling['random'])
File "/project-ssd/PoseVocab/utils/nerf_util.py", line 293, in sample_randomly_for_nerf_rendering
sampled_inside_uv = inside_uv[np.random.choice(inside_uv.shape[0], inside_sample_num, replace = False)]
File "mtrand.pyx", line 909, in numpy.random.mtrand.RandomState.choice
ValueError: a must be greater than 0 unless no samples are taken

Release Date

How soon will this model be released for learning

Python environment for the code

Hi @lizhe00 , thanks a lot for releasing the code!

Could you share the python environment for this repo?

Question about the evaluation steps for the Tab.1

Dear author,

Thx for your great work about human avatar modeling. However, I con only find testing code in your repo. Could you pls tell the detailed steps/settings for your evaluation results in Tab.1? like, the frame ids and camera ids used to compute PSNR/SSIM/... for both training and novel poses.