Anh-Quan Cao,
Raoul de Charette
Inria, Paris, France.
If you find this work or code useful, please cite our paper and give this repo a star:
@InProceedings{cao2022scenerf,
author = {Cao, Anh-Quan and de Charette, Raoul},
title = {SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields},
publisher = {arxiv},
year = {2022},
}
Novel depths synthesis | 3D Reconstruction |
![]() | ![]() |
- Pretrained models will be released soon 🚨
- Create conda environment:
$ conda create -y -n scenerf python=3.7
$ conda activate scenerf
- This code was implemented with python 3.7, pytorch 1.7.1 and CUDA 10.2. Please install PyTorch:
$ conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.2 -c pytorch
- Install the dependencies:
$ cd scenerf/
$ pip install -r requirements.txt
- Install tbb
$ conda install -c bioconda tbb=2020.2
- Downgrade torchmetrics
$ pip install torchmetrics==0.6.0
- Finally, install scenerf:
$ pip install -e ./
-
To train and evaluate novel depths/views synthesis, please download on KITTI Odometry website the following data:
- Odometry data set (calibration files, 1 MB)
- Odometry data set (color, 65 GB)
- Odometry ground truth poses (4 MB)
- Velodyne laser data, 80 GB
-
To evaluate scene reconstruction, please download the SemanticKITTI voxel data (700 MB).
-
Create a folder to store preprocess data at
/path/to/kitti/preprocess/folder
. -
Store paths in environment variables for faster access (Note: folder 'dataset' is in /path/to/kitti):
$ export KITTI_PREPROCESS=/path/to/kitti/preprocess/folder $ export KITTI_ROOT=/path/to/kitti
-
Create folders to store training logs at /path/to/kitti/logdir.
-
Store in an environment variable:
$ export KITTI_LOG=/path/to/kitti/logdir
-
Train scenerf using 4 v100-32g GPUs with batch_size of 4 (1 item per GPU):
$ cd scenerf/ $ python scenerf/scripts/train.py \ --bs=4 --n_gpus=4 \ --enable_log=True \ --preprocess_root=$KITTI_PREPROCESS \ --root=$KITTI_ROOT \ --logdir=$KITTI_LOG \ --n_gaussians=4 --n_pts_per_gaussian=8 \ --max_epochs=50 --exp_prefix=Train
Create folders to store intermediate evaluation data at /path/to/evaluation/save/folder
and reconstruction data at /path/to/reconstruction/save/folder
.
$ export EVAL_SAVE_DIR=/path/to/evaluation/save/folder
$ export RECON_SAVE_DIR=/path/to/reconstruction/save/folder
Supposed we obtain the model from the training step at /path/to/model/checkpoint/last.ckpt
. We follow the steps below to evaluate the novel depths synthesis performance.
- Compute the depth metrics on all frames in each sequence, additionally grouped by the distance to the input frame.
$ cd scenerf/
$ python scenerf/scripts/evaluation/save_depth_metrics.py \
--eval_save_dir=$EVAL_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS \
--model_path=/path/to/model/checkpoint/last.ckpt
- Aggregate the depth metrics from all sequences.
$ cd scenerf/
$ python scenerf/scripts/evaluation/agg_depth_metrics.py \
--eval_save_dir=$EVAL_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS
Given the trained model at /path/to/model/checkpoint/last.ckpt
, the novel views synthesis performance is obtained as followed:
- Render an RGB image for every frame in each sequence.
$ cd scenerf/
$ python scenerf/scripts/evaluation/render_colors.py \
--eval_save_dir=$EVAL_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS \
--model_path=/path/to/model/checkpoint
- Compute the metrics, additionally grouped by the distance to the input frame.
$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_color.py --eval_save_dir=$EVAL_SAVE_DIR
- Generate novel views/depths for reconstructing scene.
$ cd scenerf/
$ python scenerf/scripts/reconstruction/generate_novel_depths.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS \
--model_path=/path/to/model/checkpoint \
--angle=10 --step=0.5 --max_distance=10.1
- Convert the novel views/depths to TSDF volume. Note: the angle, step, and max_distance should match the previous step.
$ cd scenerf/
$ python scenerf/scripts/reconstruction/depth2tsdf.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS \
--angle=10 --step=0.5 --max_distance=10.1
- Compute scene reconstruction metrics using the generated TSDF volumes.
$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_sr.py \
--recon_save_dir=$RECON_SAVE_DIR \
--root=$KITTI_ROOT \
--preprocess_root=$KITTI_PREPROCESS
The work was partly funded by French project SIGHT (ANR-20-CE23-0016) and was performed using HPC resources from GENCI–IDRIS (Grant 2021-AD011012808 and 2022-AD011012808R1). We thank Fabio Pizzati and Ivan Lopes for their kind proofreading.