zonglin-tian / neural-pil

Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition - NeurIPS2021

Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition

Project Page | Video | Paper

Implementation for Neural-PIL. A novel method which decomposes multiple images into shape, BRDF and illumination with a split-sum preintegrated illumination network.

Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition
Mark Boss¹, Varun Jampani², Raphael Braun¹, Ce Liu³, Jonathan T. Barron², Hendrik P. A. Lensch^1
1University of Tübingen, ²Google Research, ³Microsoft Azure AI (work done at Google)

Setup

A conda environment is used for dependency management

conda env create -f environment.yml
conda activate neuralpil

Running

python train_neural_pil.py --datadir [DIR_TO_DATASET_FOLDER] --basedir [TRAIN_DIR] --expname [EXPERIMENT_NAME] --gpu [COMMA_SEPARATED_GPU_LIST]

Specific Arguments per Dataset

Most setup is handled by configurations files located in configs/neural_pil/.

Our Synthethic Scenes

--config configs/neural_pil/blender.txt

NeRF Synthethic Scenes

--config configs/neural_pil/nerf_blender.txt

Real-World

--config configs/neural_pil/real_world.txt

Often objects are captured in a spherical manner and the flag --spherify should be applied for those scenes.

Datasets

All datasets are taken from NeRD.

Run Your Own Data

Mainly camera poses and segmentation masks are required. For the poses the scripts from NeRF are used to prepare the scenes. The dataset then needs to be put in the following file structure:

images/
    [IMG_NAME_1].jpg
    [IMG_NAME_2].jpg
    ...
masks/
    [IMG_NAME_1].jpg
    [IMG_NAME_2].jpg
    ...
poses_bounds.npy

The poses_bounds.npy is generated from the LLFF script.

Evaluation

The train_neural_pil.py can be called with a --render_only flag and the --config flag pointing to the args.txt of the experiments folder.

Citation

@inproceedings{boss2021neuralpil,
  title         = {Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition},
  author        = {Boss, Mark and Jampani, Varun and Braun, Raphael and Liu, Ce and Barron, Jonathan T. and Lensch, Hendrik P.A.},
  booktitle     = {Advances in Neural Information Processing Systems (NeurIPS)},
  year          = {2021},
}