yw0208 / w-hmr

W-HMR: Human Mesh Recovery in World Space with Weak-supervised Camera Calibration and Orientation Correction

W-HMR: Human Mesh Recovery in World Space with Weak-supervised Camera Calibration and Orientation Correction

Features

W-HMR is a human body pose and shape estimation method in world space. It predicts the SMPL body model in both camera and world coordinates for a monocular image. [Camera Calibration] predicts the focal length for better mesh-image alignment. [Orientation Correction] make recovered body reasonable in world space.

This implementation:

• has the demo code for W-HMR implemented in PyTorch.

News 🚩

[March 21, 2024] Release codes and pretrained weights for demo.

[March 26, 2024] Pre-processed labels are available now.

[April 24, 2024] Fix some import bugs and loading error.

[April 24, 2024] Release more necessary files and preprocessed labels.

TODOs

• Release demo codes.

• Release pre-processed labels.

• Release evluation codes.

• Release training codes.

Getting Started

Requirements

W-HMR has been implemented and tested on Ubuntu 18.04 with python == 3.8.

Install the requirements following environment.yml.

💃 If you have any difficulty configuring your environment or meet any bug, please let me know. I'd be happy to help!

Running the Demo

W-HMR

First, you need to download the required data (i.e our trained model and SMPL model parameters) from Google Drive or Baidu Netdisk. It is approximately 700MB. Unzip it and put it in the repository root. Then, running the demo is as simple as:

python demo/whmr_demo.py --image_folder data/sample_images --output_folder output/sample_images

Sample demo output:

On the right, they are the output in camera and world coordinate. We put them in world space and generate ground planes to show how accurate the orientation is.

Training

Training instructions will follow after publication.

Pre-processed Dataset Labels

All the data used in our paper is publicly available. You can just download them from their official website following our dataset introduction in the paper.

But for your convenience, I provide some download links for pre-processed labels here.

The most important source is PyMAF. You can download the pre-processed labels of 3DPW, COCO, LSP, MPII and MPI-INF-3DHP, which include pseudo 3D joint label fitted by EFT.

We also use some augmented data from CLIFF and HMR 2.0.

I also processed some dataset labels (e.g. AGORA and HuMMan), you can download them from Google Drive or Baidu Pan. You should unzip the dataset_extras.zip and put these files in the ./data/dataset_extras/ folder.

As for training W-HMR for global mesh recovery, I add pseudo-labels of global pose to some datasets. You can download them from Google Drive or Baidu Netdisk. Then unzip it and put files in the ./data/dataset_extras/ folder.

Acknowledgments

Part of the code is borrowed from the following projects, including ,PyMAF, AGORA, PyMAF-X, PARE, SPEC, MeshGraphormer, 4D-Humans, VitPose. Many thanks to their contributions.

Citation

If you find this repository useful, please consider citing our paper and lightning the star:

@article{yao2023w,
        title={W-HMR: Human Mesh Recovery in World Space with Weak-supervised Camera Calibration and Orientation Correction},
        author={Yao, Wei and Zhang, Hongwen and Sun, Yunlian and Tang, Jinhui},
        journal={arXiv preprint arXiv:2311.17460},
        year={2023}
}

w-hmr's Issues

ViTPose incorrect path

Hi
Something wrong with import path: here you import from models.ViTPose with capital T, but the corresponding path in the project is with small t. As below the path to a file is with capital T too, I think just renaming VitPose to ViTPose will resolve the issue.

Real position in camera space

Excellent work. Does this approach accurately provide the actual 3D positions of the SMPL model? For instance, methods like 4D-humans and Pymaf are known to be imprecise in outputting depth information. Could you recommend an advanced approach for achieving real position 3D human reconstruction utilizing monocular video footage? Your expertise and insights on this matter would be immensely appreciated.

some absolute paths to ckpt files

there are 5 checkpoints in the project, that are loaded by an absolute path:
this vitpose checkpoint, this smpl model, this pymaf checkpoint, this very important place and this not so important place.

Amazing work! When will the code released?

Can you specify a roadmap for code releasing?

ratio of training dataset

hello,
What is the approximate ratio among training datasets?
Ex) Human 3.6m 0.2, 3DPW 0.05, MPII3D 0.05, MPII 0.05, AGORA 0.2, COCO 0.15, SPEC-SYN 0.1 , and HuMMan 0.2.
thank you great work!

The training model is private and cannot be downloaded.

Unable to load EGL library

I am using window11 , anaconda environment with python 3.8.

utils\renderer_cam.py", line 130, in render_overlay_image
renderer = pyrender.OffscreenRenderer(

The above line produces an exception

OpenGL\platform\egl.py", line 86, in EGL
raise ImportError("Unable to load EGL library", *err.args)
ImportError: ('Unable to load EGL library', "Could not find module 'EGL' (or one of its dependencies). Try using the full path with constructor syntax.", 'EGL', None)
Installed pyopengl,pyopengl-accelerate

I am able to get pkl file with --no_render param but i need the .obj file
Please help me rectify the issue