mywordwin / mobilehand

MobileHand: Real-time 3D Hand Shape and Pose Estimation from Color Image

MobileHand: Real-time 3D Hand Shape and Pose Estimation from Color Image

This repository contains the sample code for the paper MobileHand: Real-time 3D Hand Shape and Pose Estimation from Color Image

Paper | Video | Results on STB Dataset B1 Random | Results on STB Dataset B1 Counting

If you find our code or paper useful, please consider citing

@inproceedings{MobileHand:2020,
  title = {MobileHand: Real-time 3D Hand Shape and Pose Estimation from Color Image},
  author = {Guan Ming, Lim and Prayook, Jatesiktat and Wei Tech, Ang},
  booktitle = {27th International Conference on Neural Information Processing (ICONIP)},
  year = {2020}
}

Installation

The simplest way to run our implementation is to use anaconda.

You can create an anaconda environment called mobilehand with

conda env create -f environment.yaml
conda activate mobilehand

Next, you will need to download the MANO right hand model

• Go to MANO project page
• Click on Sign In and register for your account
• Download Models & Code (mano_v1_2.zip)
• Unzip and copy the file mano_v1_2/models/MANO_RIGHT.pkl into the mobilehand/model folder

The following steps could be ignored by installing the latest version of Chumpy 0.70 which supports Python 3.

To allow the use of MANO model in Python 3 environment, we will need to remove Chumpy objects from the original `MANO_RIGHT.pkl` model. The following steps are adapted from [smplx repo](https://github.com/vchoutas/smplx/blob/master/tools/README.md): * In a Python 2 virtual environment with [Chumpy](https://github.com/mattloper/chumpy) and [tqdm](https://github.com/tqdm/tqdm) installed ``` conda create -n py27 python=2.7 conda activate py27 pip install chumpy pip install tqdm ```

• Run the following command to remove any Chumpy objects and it will create a new file MANO_RIGHT_NEW.pkl:

python model/clean_ch.py --input-models model/MANO_RIGHT.pkl --output-folder model/

Demo

Change directory to the folder mobilehand/code/

cd code/

To test on a sample image from the STB dataset run:

python demo.py --dataset stb

To test on a sample image from the FreiHAND dataset run:

python demo.py --dataset freihand

Sample results

• STB dataset
  

• FreiHAND dataset
  

Real-time demo

To test on a sample video.mp4 file run:

python realtime.py

To test from your own camera or video file, you can uncomment/edit lines 24 and 25 of realtime.py

References for dataset

[2017 ICIP] A Hand Pose Tracking Benchmark from Stereo Matching. [PDF] [Project] [Code]

Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Yang

[ICCV 2019] FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images. [PDF] [Project] [Code]

Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, Thomas Brox

References on 3D hand shape and pose estimation from color image

[CVPR 2019] Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering. [PDF]

Seungryul Baek, Kwang In Kim, Tae-Kyun Kim

[CVPR 2019] 3D Hand Shape and Pose from Images in the Wild. [PDF] [Code]

Adnane Boukhayma, Rodrigo de Bem, Philip H.S. Torr

[CVPR 2019] 3D Hand Shape and Pose Estimation from a Single RGB Image. [PDF] [Project] [Code] (Oral)

Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, Junsong Yuan

[CVPR 2019] Learning joint reconstruction of hands and manipulated objects. [PDF] [Code] [Code] [Project]

Yana Hasson, Gül Varol, Dimitris Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid

[ICCV 2019] End-to-end Hand Mesh Recovery from a Monocular RGB Image. [PDF] [Code]

Xiong Zhang*, Qiang Li*, Wenbo Zhang, Wen Zheng

[CVPR 2020] Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild. [PDF] [Project] (Oral)

Dominik Kulon, Riza Alp Güler, Iasonas Kokkinos, Michael Bronstein, Stefanos Zafeiriou

[CVPR 2020] Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data. [PDF] [Project] [Code]

Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, Feng Xu

References on other key methods that influence this work

[MVA 2019] Accurate Hand Keypoint Localization on Mobile Devices. [PDF] [Code]

Filippos Gouidis, Paschalis Panteleris, Iason Oikonomidis, Antonis Argyros

[CVPR 2018] End-to-end Recovery of Human Shape and Pose. [PDF] [Project] [Code]

Angjoo Kanazawa, Michael J Black, David W. Jacobs, Jitendra Malik

[SIGGRAPH ASIA 2017] Embodied Hands:Modeling and Capturing Hands and Bodies Together. [PDF] [Project]

Javier Romero, Dimitrios Tzionas, Michael J Black