iamncj / nrhints

Official Code Release for SIGGRAPH 2023 Paper: Relighting Neural Radiance Fields with Shadow and Highlight Hints

Relighting Neural Radiance Fields with Shadow and Highlight Hints

Chong Zeng · Guojun Chen · Yue Dong · Pieter Peers · Hongzhi Wu · Xin Tong

SIGGRAPH 2023 Conference Proceedings

Project Page | Paper | arXiv | Data

---

Setup

Environment

The code is developed and tested on Linux servers with NVIDIA GPU(s). We support Python 3.8+ and PyTorch 1.11+. After getting a required Python environment, you can setup the rest of requirements by running:

git clone https://github.com/iamNCJ/NRHints.git
cd NRHints
pip install -r requirements.txt

Data

Our data is compatible with NeRF Blender Dataset, except that we have extra fields in each frame for point light position.

You can download our data here.

Usage

Configuration System

We use tyro for configuration management. Description to all configurations can be found by running python main.py -h.

Training

python3 main.py config:nr-hints --config.data.path /path/to/data/ --config.scene-name XXX

Refer to train_synthetic.sh and train_real.sh for training on synthetic and real data, respectively.

Note:

1. Our code automatically detects the number of GPUs and uses all of them for training. If you want to use a subset of GPUs, you can set the CUDA_VISIBLE_DEVICES environment variable.
2. For training on real captured scenes, we recommend turning on camera optimization by using config:nr-hints-cam-opt, which can significantly reduce the blurry effects. Since this is an improvement after the paper submission, details are described in the author's version.

Testing

python3 main.py config:nr-hints --config.data.path /path/to/data/ --config.scene-name XXX --config.evaluation-only True

Refer to eval_synthetic.sh and eval_real.sh for testing on synthetic and real data, respectively.

Our pretrained models can be downloaded here.

Data and Models

Real Captured Scenes

Object	Data	Pre-trained model
Cat	Link	Link
Cluttered Scene	Link	Link
Pixiu Statuette	Link	Link
Ornamental Fish	Link	Link
Cat on Decor	Link	Link
Cup and Fabric	Link	Link
Pikachu Statuette	Link	Link

Synthetic Rendered Scenes

Note: Our synthetic data rendering scripts are released at here.

Object	Data	Pre-trained model
Diffuse	Link	Link
Metallic	Link	Link
Glossy-Metal	Link	Link
Rough-Metal	Link	Link
Anisotropic-Metal	Link	Link
Plastic	Link	Link
Glossy-Plastic	Link	Link
Rough-Plastic	Link	Link
Short-Fur	Link	Link
Long-Fur	Link	Link
Translucent	Link	Link
Fur-Ball	Link	Link
Basket	Link	Link
Layered Woven Ball	Link	Link
Drums	Link	Link
Hotdog	Link	Link
Lego	Link	Link

You can use the script download_data.sh to download all data.

Citation

Cite as below if you find this repository is helpful to your project:

@inproceedings {zeng2023nrhints,
    title      = {Relighting Neural Radiance Fields with Shadow and Highlight Hints},
    author     = {Chong Zeng and Guojun Chen and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong},
    booktitle  = {ACM SIGGRAPH 2023 Conference Proceedings},
    year       = {2023}
}

Acknowledgement

Some code snippets are borrowed from NeuS and Nerfstudio. Thanks for these great projects.

nrhints's Issues

how to make a video

I tested cat pretrained, how to convert it into mp4 video

Why can the method obtain comparable results to Gao et al. using only 500 images?

Hi, thanks for your work and code, the results are pretty impressive!

I'd like to know why the proposed method can obtain comparable results to Gao et al. using only 500 training images. I think both methods are trained in a scene-specific fashion and use the same kinds of hints. I wonder what specific design in the proposed method makes it more data efficient?

Could you share your blender files used for data generation

Hi, thanks for the wonderful work!

I'm working on the reconstruction of reflected scenes, and I'm hoping for generating images that have the same camera position but with different light position. Your dataset is really good and comprehensive, so I am wondering if you could share your blender project?

Many thanks!

How to train the model using my own data?

Hello! Thank you for sharing the code.
I have a video recorded in a real-life scenario and have utilized COLMAP to calculate the camera parameters. However, I have trouble in getting the light position and light intensity. Would you kindly guide me on how to obtain them? Thank you very much!

How to import the ouput model to blender?

Hi!
I want to use the output model of nrhints in blender. But i found the output model has no color in blender, like this:

This is the first output model 'cat' in blender.
Did I do something wrong? I hope you can help me!

Best wishes.

How to visualize 3D model?

Hi, thanks for providing the source code.

I have been trying to viusalize the difference between tradional NeRF methods and Relighting hints with NeRF. I was able to get the 3D model with the SDF field for the Cat dataset, how can i visualize the model with hihglight and shadow hints?
I have attached a screencapture of the output model

Thanks