zeta1999 / few-shot-patch-based-training

The official implementation of our SIGGRAPH 2020 paper Interactive Video Stylization Using Few-Shot Patch-Based Training

Interactive Video Stylization Using Few-Shot Patch-Based Training

The official implementation of

Interactive Video Stylization Using Few-Shot Patch-Based Training
O. Texler, D. Futschik, M. Kučera, O. Jamriška, Š. Sochorová, M. Chai, S. Tulyakov, and D. Sýkora
[WebPage], [Paper], [BiBTeX]

Run

Download the testing-data.zip, and unzip. The _train folder is expected to be next to the _gen folder.

Pre-Trained models

If you want just quickly test the network, here are some pre-trained-models.zip. Unzip, and follow with the Generate step. Be sure to set the correct --checkpoint path when calling generate.py, e.g., _pre-trained-models/Zuzka2/model_00020.pth.

Train

To train the network, run the train.py See the example command below:

train.py --config "_config/reference_P.yaml" 
		 --data_root "Zuzka2_train" 
		 --log_interval 1000 
		 --log_folder logs_reference_P

Every 1000 (log_interval) epochs, train.py saves the current generator to logs_reference_P (log_folder), and it validates/runs the generator on _gen data - the result is saved in Zuzka2_gen/res__P

Generate

To generate the results, run generate.py.

generate.py --checkpoint "Zuzka2_train/logs_reference_P/model_00020.pth" 
	    --data_root "Zuzka2_gen"
	    --dir_input "input_filtered"
	    --outdir "Zuzka2_gen/res_00020" 
	    --device "cuda:0"

Installation

Tested on Windows 10, Python 3.7.8, CUDA 10.2. With the following python packages:

numpy                  1.19.1
opencv-python          4.4.0.40
Pillow                 7.2.0
PyYAML                 5.3.1
scikit-image           0.17.2
scipy                  1.5.2
tensorflow 	       1.15.3 (tensorflow is used only in the logger.py, I will remove this not-necessary dependency soon)
torch                  1.6.0
torchvision            0.7.0

TODO

• Add code for "interactive" usedcase as shown in the paper.
• Remove the dependency on tensorflow.

Credits

• This project started when Ondrej Texler was an intern at Snap Inc., and it was funded by Snap Inc. and Czech Technical University in Prague
• The main engineering forces behind this repository are Ondrej Texler, David Futschik, and Michal Kučera.

License

• The Patch-Based Training method is not patented, and we do not plan on patenting.
• However, you should be aware that certain parts of the code in this repository were written when Ondrej Texler and David Futschik were employed by Snap Inc.. If you find this project useful for your commercial interests, please, reimplement it.

Citing

If you find Interactive Video Stylization Using Few-Shot Patch-Based Training useful for your research or work, please use the following BibTeX entry.

@Article{Texler20-SIG,
    author    = "Ond\v{r}ej Texler and David Futschik and Michal Ku\v{c}era and Ond\v{r}ej Jamri\v{s}ka and \v{S}\'{a}rka Sochorov\'{a} and Menglei Chai and Sergey Tulyakov and Daniel S\'{y}kora",
    title     = "Interactive Video Stylization Using Few-Shot Patch-Based Training",
    journal   = "ACM Transactions on Graphics",
    volume    = "39",
    number    = "4",
    pages     = "73",
    year      = "2020",
}