sanowar-raihan / nerf-meta

NeRF Meta-Learning with PyTorch

NeRF Meta Learning With PyTorch

nerf-meta is a PyTorch re-implementation of NeRF experiments from the paper "Learned Initializations for Optimizing Coordinate-Based Neural Representations". Simply by initializing NeRF with meta-learned weights, we can achieve:

• Photo Tourism
• View Synthesis from Single Image

Be sure to check out the original resources from the authors:

• [Paper]: https://arxiv.org/abs/2012.02189
• [Original Project Page]: https://www.matthewtancik.com/learnit
• [Official JAX Implementation]: https://github.com/tancik/learnit
• [Twitter Summary]: https://twitter.com/BenMildenhall/status/1335290097390895104

Environment

• Python 3.8
• PyTorch 1.8
• NumPy, imageio, imageio-ffmpeg

Photo Tourism

Starting from a meta-initialized NeRF, we can interpolate between camera pose, focal length, aspect ratio and scene appearance. The videos below are generated with a 5 layer only NeRF, trained for ~100k iterations.

BrandenburgGate.mp4

TreviFountain.mp4

Data

• Get image collection of different landmarks from image-matching-challenge
• Get poses and bounds from learnit google drive

Train and Evaluate

1. Train NeRF on a single landmark scene using Reptile meta-learning:

   python tourism_train.py --config ./configs/tourism/$landmark.json

2. Test Photo Tourism performance and generate an interpolation video of the landmark:

   python tourism_test.py --config ./configs/tourism/$landmark.json --weight-path $meta_weight.pth

View Synthesis from Single Image

Given a single input view, meta-initialized NeRF can generate a 360-degree video. The following ShapeNet video is generated with a class-specific NeRF (5 layers deep), trained for ~100k iterations.

ShapeNet.mp4

Data

• Get ShapeNet Data and splits files from learnit google drive

Train and Evaluate

1. Train NeRF on a particular ShapeNet class using Reptile meta-learning:

   python shapenet_train.py --config ./configs/shapenet/$shape.json

2. Optimize the meta-trained model on a single view and test on other held-out views. It also generates a 360 video for each test object:

   python shapenet_test.py --config ./configs/shapenet/$shape.json --weight-path $meta_weight.pth

Acknowledgments

I referenced several open-source NeRF and Meta-Learning code base for this implementation. Specifically, I borrowed/modified code from the following repositories:

• learnit
• JAX NeRF
• NeRF PyTorch Lightning
• learn2learn

Thanks to the authors for releasing their code.