mfahes / scenerf

SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields

SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields

Anh-Quan Cao, Raoul de Charette
Inria, Paris, France.

If you find this work or code useful, please cite our paper and give this repo a star:

@InProceedings{cao2022scenerf,
    author    = {Cao, Anh-Quan and de Charette, Raoul},
    title     = {SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields},
    publisher = {arxiv},
    year      = {2022},
}

Teaser

Novel depths synthesis	3D Reconstruction

Table of Content

• News
• Installation
• Datasets
• Training
• Evaluation
  • Novel depths synthesis
  • Novel views synthesis
  • Scene reconstruction
• Acknowledgment

News

• Pretrained models will be released soon 🚨

Installation

1. Create conda environment:

$ conda create -y -n scenerf python=3.7
$ conda activate scenerf

2. This code was implemented with python 3.7, pytorch 1.7.1 and CUDA 10.2. Please install PyTorch:

$ conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.2 -c pytorch

3. Install the dependencies:

$ cd scenerf/
$ pip install -r requirements.txt

4. Install tbb

$ conda install -c bioconda tbb=2020.2

5. Downgrade torchmetrics

$ pip install torchmetrics==0.6.0

6. Finally, install scenerf:

$ pip install -e ./

Dataset

1. To train and evaluate novel depths/views synthesis, please download on KITTI Odometry website the following data:

   • Odometry data set (calibration files, 1 MB)
   • Odometry data set (color, 65 GB)
   • Odometry ground truth poses (4 MB)
   • Velodyne laser data, 80 GB

2. To evaluate scene reconstruction, please download the SemanticKITTI voxel data (700 MB).

3. Create a folder to store preprocess data at /path/to/kitti/preprocess/folder.

4. Store paths in environment variables for faster access (Note: folder 'dataset' is in /path/to/kitti):

   $ export KITTI_PREPROCESS=/path/to/kitti/preprocess/folder
   $ export KITTI_ROOT=/path/to/kitti 
   

Training

1. Create folders to store training logs at /path/to/kitti/logdir.

2. Store in an environment variable:

   $ export KITTI_LOG=/path/to/kitti/logdir
   

3. Train scenerf using 4 v100-32g GPUs with batch_size of 4 (1 item per GPU):

   $ cd scenerf/
   $ python scenerf/scripts/train.py \
       --bs=4 --n_gpus=4 \
       --enable_log=True \
       --preprocess_root=$KITTI_PREPROCESS \
       --root=$KITTI_ROOT \
       --logdir=$KITTI_LOG \
       --n_gaussians=4 --n_pts_per_gaussian=8  \
       --max_epochs=50 --exp_prefix=Train
   

Evaluation

Create folders to store intermediate evaluation data at /path/to/evaluation/save/folder and reconstruction data at /path/to/reconstruction/save/folder.

$ export EVAL_SAVE_DIR=/path/to/evaluation/save/folder
$ export RECON_SAVE_DIR=/path/to/reconstruction/save/folder

Evaluate novel depths synthesis

Supposed we obtain the model from the training step at /path/to/model/checkpoint/last.ckpt. We follow the steps below to evaluate the novel depths synthesis performance.

1. Compute the depth metrics on all frames in each sequence, additionally grouped by the distance to the input frame.

$ cd scenerf/
$ python scenerf/scripts/evaluation/save_depth_metrics.py \
    --eval_save_dir=$EVAL_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS \
    --model_path=/path/to/model/checkpoint/last.ckpt

2. Aggregate the depth metrics from all sequences.

$ cd scenerf/
$ python scenerf/scripts/evaluation/agg_depth_metrics.py \
    --eval_save_dir=$EVAL_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS

Evaluate novel views synthesis

Given the trained model at /path/to/model/checkpoint/last.ckpt, the novel views synthesis performance is obtained as followed:

1. Render an RGB image for every frame in each sequence.

$ cd scenerf/
$ python scenerf/scripts/evaluation/render_colors.py \
    --eval_save_dir=$EVAL_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS \
    --model_path=/path/to/model/checkpoint

2. Compute the metrics, additionally grouped by the distance to the input frame.

$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_color.py --eval_save_dir=$EVAL_SAVE_DIR

Scene reconstruction

1. Generate novel views/depths for reconstructing scene.

$ cd scenerf/
$ python scenerf/scripts/reconstruction/generate_novel_depths.py \
    --recon_save_dir=$RECON_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS \
    --model_path=/path/to/model/checkpoint \
    --angle=10 --step=0.5 --max_distance=10.1

2. Convert the novel views/depths to TSDF volume. Note: the angle, step, and max_distance should match the previous step.

$ cd scenerf/
$ python scenerf/scripts/reconstruction/depth2tsdf.py \
    --recon_save_dir=$RECON_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS \
    --angle=10 --step=0.5 --max_distance=10.1

3. Compute scene reconstruction metrics using the generated TSDF volumes.

$ cd scenerf/
$ python scenerf/scripts/evaluation/eval_sr.py \
    --recon_save_dir=$RECON_SAVE_DIR \
    --root=$KITTI_ROOT \
    --preprocess_root=$KITTI_PREPROCESS

Acknowledgment

The work was partly funded by French project SIGHT (ANR-20-CE23-0016) and was performed using HPC resources from GENCI–IDRIS (Grant 2021-AD011012808 and 2022-AD011012808R1). We thank Fabio Pizzati and Ivan Lopes for their kind proofreading.