- Install
Anaconda
on your machine. - In root directory run
conda env create -f environment.yml
&conda activate relighting
- Redirect to
src/venders/
runsh ./scripts/download_trained_model.sh
to download the pretrained model - Redirect to
src/
runpython3 main.py
- You will see output images inside folder
outputs
(Warning: this only works for full-body(human) photos)
-
The pipline is inspired by Total Relighting: Learning to Relight Portraits for Background Replacement https://augmentedperception.github.io/total_relighting/total_relighting_paper.pdf (Google, 2021)
The implementation steps are different from the paper. -
The Pipeline:
Surface Normal
->Image Matting
->Albedo
->HDR filering
->Shading
-
(surfaceNormal method 1) Get normal map directly (sobel). http://citebay.com/how-to-cite/sobel-filter/
-
(surfaceNormal method 2) Get face/relative depth map, then derive normal map from the depth https://pytorch.org/hub/intelisl_midas_v2/ & download model https://nuigalwayie-my.sharepoint.com/:u:/g/personal/f_khan4_nuigalway_ie/EepkuVajAhdIjZoQm5Weyx4BjXcEZy-uw5OWxxMXq1WJPA?e=rv3aSY & https://github.com/khan9048/Facial_depth_estimation & https://courses.cs.washington.edu/courses/cse590b/02au/hdrc.pdf
-
(surfaceNormal method 3) calculate normal map, based on PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization∗ https://arxiv.org/pdf/2004.00452.pdf (some code and pre-trained models provided by the paper are used in our project)
-
calculate albedo map
-
handle HDR, (transform HDR to cube map). (Real Time Readering, 6.2.4) https://www.researchgate.net/publication/3572571_An_adaptive_Gaussian_filter_for_noise_reduction_and_edge_detection
-
shading & compositing
-
-
(Reminder: might need to change the order of some steps & mask normal map)