Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans
Project Page | Video | Paper | Data
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans
Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou
CVPR 2021
Any questions or discussions are welcomed!
Please see INSTALL.md.
Please see docker/README.md.
Please see CUSTOM.
Please see INSTALL.md to download the dataset.
We provide the pretrained models at here.
We already provide some processed data. If you want to process more videos of People-Snapshot, you could use tools/process_snapshot.py.
You can also visualize smpl parameters of People-Snapshot with tools/vis_snapshot.py.
Take the visualization on female-3-casual
as an example. The command lines for visualization are recorded in visualize.sh.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/if_nerf/female3c/latest.pth
. -
Visualization:
- Visualize novel views of single frame
python run.py --type visualize --cfg_file configs/snapshot_f3c_demo.yaml exp_name female3c render_views 144
- Visualize views of dynamic humans with fixed camera
python run.py --type visualize --cfg_file configs/snapshot_f3c_perform.yaml exp_name female3c
- Visualize mesh
# generate meshes python run.py --type visualize --cfg_file configs/snapshot_f3c_mesh.yaml exp_name female3c train.num_workers 0 # visualize a specific mesh python tools/render_mesh.py --exp_name female3c --dataset people_snapshot --mesh_ind 226
-
The results of visualization are located at
$ROOT/data/render/female3c
and$ROOT/data/perform/female3c
.
Take the training on female-3-casual
as an example. The command lines for training are recorded in train.sh.
- Train:
# training python train_net.py --cfg_file configs/snapshot_f3c.yaml exp_name female3c resume False # distributed training python -m torch.distributed.launch --nproc_per_node=4 train_net.py --cfg_file configs/snapshot_f3c.yaml exp_name female3c resume False gpus "0, 1, 2, 3" distributed True
- Tensorboard:
tensorboard --logdir data/record/if_nerf
Please see INSTALL.md to download the dataset.
We provide the pretrained models at here.
The smpl parameters of ZJU-MoCap have different definition from the one of MPI's smplx.
- If you want to extract vertices from the provided smpl parameters, please use
zju_smpl/extract_vertices.py
. - The reason that we use the current definition is described at here.
It is okay to train Neural Body with smpl parameters fitted by smplx.
The command lines for test are recorded in test.sh.
Take the test on sequence 313
as an example.
- Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/if_nerf/xyzc_313/latest.pth
. - Test:
python run.py --type evaluate --cfg_file configs/latent_xyzc_313.yaml exp_name xyzc_313
Take the visualization on sequence 313
as an example. The command lines for visualization are recorded in visualize.sh.
-
Download the corresponding pretrained model and put it to
$ROOT/data/trained_model/if_nerf/xyzc_313/latest.pth
. -
Visualization:
- Visualize novel views of single frame
python run.py --type visualize --cfg_file configs/xyzc_demo_313.yaml exp_name xyzc_313 render_views 100
- Visualize views of dynamic humans with fixed camera
python run.py --type visualize --cfg_file configs/xyzc_perform_313.yaml exp_name xyzc_313 render_views 1
- Visualize views of dynamic humans with rotated camera
python run.py --type visualize --cfg_file configs/xyzc_perform_313.yaml exp_name xyzc_313 render_views 100
- Visualize mesh
# generate meshes python run.py --type visualize --cfg_file configs/latent_xyzc_mesh_313.yaml exp_name xyzc_313 train.num_workers 0 # visualize a specific mesh python tools/render_mesh.py --exp_name xyzc_313 --dataset zju_mocap --mesh_ind 0
-
The results of visualization are located at
$ROOT/data/render/xyzc_313
and$ROOT/data/perform/xyzc_313
.
Take the training on sequence 313
as an example. The command lines for training are recorded in train.sh.
- Train:
# training python train_net.py --cfg_file configs/latent_xyzc_313.yaml exp_name xyzc_313 resume False # distributed training python -m torch.distributed.launch --nproc_per_node=4 train_net.py --cfg_file configs/latent_xyzc_313.yaml exp_name xyzc_313 resume False gpus "0, 1, 2, 3" distributed True
- Tensorboard:
tensorboard --logdir data/record/if_nerf
If you find this code useful for your research, please use the following BibTeX entry.
@inproceedings{peng2021neural,
title={Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans},
author={Peng, Sida and Zhang, Yuanqing and Xu, Yinghao and Wang, Qianqian and Shuai, Qing and Bao, Hujun and Zhou, Xiaowei},
booktitle={CVPR},
year={2021}
}