mjones00 / romp

Monocular, One-stage, Regression of Multiple 3D People and their 3D positions & trajectories in camera & global coordinates. ROMP[ICCV21], BEV[CVPR22], TRACE[CVPR2023]

ROMP	BEV	TRACE
Monocular, One-stage, Regression of Multiple 3D People (ICCV21)	Putting People in their Place: Monocular Regression of 3D People in Depth (CVPR2022)	TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments (CVPR2023)
ROMP is a one-stage method for monocular multi-person 3D mesh recovery in real time.	BEV further explores multi-person depth relationships and supports all age groups.	TRACE further tracks specific subjects and recover their global 3D trajectory with dynamic cameras.
[Paper] [Video]	[Project Page] [Paper] [Video]	[Project Page] [Paper] [Video]
	[RelativeHuman Dataset]	[DynaCam Dataset]

We provide cross-platform API (installed via pip) to run ROMP & BEV on Linux / Windows / Mac.

Table of contents

• Table of contents
• News
• Getting started
  • Installation
  • Try on Google Colab
• How to use it
  • Please refer to this guidance for inference & export (fbx/glb/bvh).
  • Train
  • Evaluation
  • Docker usage
  • Bugs report
• Citation
• Acknowledgement

News

2023/06/17: First release of TRACE's inference code. Please refer to this instructions.
2022/06/21: Training & evaluation code of BEV is released. Please update the model_data.
2022/05/16: simple-romp v1.0 is released to support tracking, calling in python, exporting bvh, and etc.
2022/04/14: Inference code of BEV has been released in simple-romp v0.1.0.
2022/04/10: Adding onnx support, with faster inference speed on CPU/GPU.
Old logs

Getting started

Please use simple-romp for inference, the rest code is just for training.

Installation

pip install --upgrade setuptools numpy cython
pip install --upgrade simple-romp

For more details, please refer to install.md.

How to use it

Please refer to this guidance for inference & export (fbx/glb/bvh).

Train

For training, please refer to installation.md for full installation. Please prepare the training datasets following dataset.md, and then refer to train.md for training.

Evaluation

Please refer to romp_evaluation.md and bev_evaluation.md for evaluation on benchmarks.

Extensions

[Blender addon]: Yan Chuanhang created a Blender-addon to drive a 3D character in Blender using ROMP from image, video or webcam input.

[VMC protocol]: Vivien Richter implemented a VMC (Virtual Motion Capture) protocol support for different Motion Capture solutions with ROMP.

Docker usage

Please refer to docker.md

Bugs report

Welcome to submit issues for the bugs.

Contributors

This repository is maintained by Yu Sun.

ROMP has also benefited from many developers, including

• Peng Cheng : constructive discussion on Center map training.
• Marco Musy : help in the textured SMPL visualization.
• Gavin Gray : adding support for an elegant context manager to run code in a notebook.
• VLT Media and Vivien Richter : adding support for running on Windows & batch_videos.py.
• Chuanhang Yan : developing an addon for driving character in Blender.
• Tian Jin: help in simplified smpl and fast rendering (realrender).
• ZhengdiYu : helpful discussion on optimizing the implementation details.
• Ali Yaghoubian : add Docker file for simple-romp.

Citation

@InProceedings{TRACE,
    author = {Sun, Yu and Bao, Qian and Liu, Wu and Mei, Tao and Black, Michael J.},
    title = {{TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments}}, 
    booktitle = {CVPR}, 
    year = {2023}}
@InProceedings{BEV,
    author = {Sun, Yu and Liu, Wu and Bao, Qian and Fu, Yili and Mei, Tao and Black, Michael J},
    title = {{Putting People in their Place: Monocular Regression of 3D People in Depth}},
    booktitle = {CVPR},
    year = {2022}}
@InProceedings{ROMP,
    author = {Sun, Yu and Bao, Qian and Liu, Wu and Fu, Yili and Michael J., Black and Mei, Tao},
    title = {{Monocular, One-stage, Regression of Multiple 3D People}},
    booktitle = {ICCV},
    year = {2021}}

Acknowledgement

This work was supported by the National Key R&D Program of China under Grand No. 2020AAA0103800.
MJB Disclosure: https://files.is.tue.mpg.de/black/CoI_CVPR_2023.txt