peterzs / stylizing-3d-scene

Stylizing-3D-Scene

PyTorch implementaton of our WACV 2022 paper "Stylizing 3D Scene via Implicit Representation and HyperNetwork". You can visit our project website here.

In this work, we aim to address the 3D scene stylization problem - generating stylized images of the scene at arbitrary novel view angles.

Paper

Stylizing 3D Scene via Implicit Representation and HyperNetwork
Pei-Ze Chiang*, Meng-Shiun Tsai*, Hung-Yu Tseng, Wei-Sheng Lai, Wei-Chen Chiu
IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022.

Please cite our paper if you find it useful for your research.

@InProceedings{Chiang_2022_WACV,
    author    = {Chiang, Pei-Ze and Tsai, Meng-Shiun and Tseng, Hung-Yu and Lai, Wei-Sheng and Chiu, Wei-Chen},
    title     = {Stylizing 3D Scene via Implicit Representation and HyperNetwork},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {January},
    year      = {2022},
}

Environment Setting

• This code was developed with Python 3.6.12 & Pytorch 1.6.0 & CUDA 10.0

Dataset and Pretrained Weight

• Dataset: Tanks and Temples and Wikiart images
  NOTE: Download the datasets from here and unzip them under ./Tanks_and_Temples and ./wikiart/train.
• Pretrained Weights: Second stage
  NOTE: Download the pretrained weight from here and put it under ./logs/[scene_name]_second_stage/ (scene_name: Family, Francis, Horse, Playground, Truck)

Testing

1. First Stage:

CUDA_VISIBLE_DEVICES=0,1 python ddp_test_nerf.py --config configs/test_family_second.txt --render_splits test

2. Second Stage:

bash test_script_second.sh

Training

1. First Stage:

CUDA_VISIBLE_DEVICES=0,1 python ddp_train_nerf.py --config configs/train_family_first.txt

2. Second Stage:

CUDA_VISIBLE_DEVICES=0,1 python ddp_train_nerf.py --config configs/train_family_second.txt

Acknowledgments

Our code is based on NeRF++: Analyzing and Improving Neural Radiance Fields.
The implementation of Hypternetwork and Style-VAE are based on Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations and Adversarial Style Mining for One-Shot Unsupervised Domain Adaptation.
The implementation of Consistency metric(Temporal Warping Error) is borrowed from Learning Blind Video Temporal Consistency.