shirgur / hp-vae-gan

Official repository for HP-VAE-GAN

Hierarchical Patch VAE-GAN

Official repository of the paper "Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample" (NeurIPS 2020)

Project | arXiv | Code

Real Videos

Fake Videos

Environment setting

Use commands in env.sh to setup the correct conda environment

Colab

An example for training and extracting samples for image generation. The same can be easily modified for video generation using *_video(s).py files. https://colab.research.google.com/drive/1SmxFVqUvEkU7pHIwyLUz4VM1AxoVU-ER?usp=sharing

Training Video

For training a single video, use the following command for example:

CUDA_VISIBLE_DEVICES=0 python train_video.py --video-path data/vids/air_balloons.mp4 --vae-levels 3 --checkname myvideotest --visualize

Common training options:

# Networks Hyper Parameters
--nfc                model basic # channels
--latent-dim         Latent dim size
--vae-levels         # VAE levels
--generator          generator mode

# Optimization hyper parameters
--niter              number of iterations to train per scale
--rec-weight         reconstruction loss weight
--train-all          train all levels w.r.t. train-depth

# Dataset
--video-path         video path (required)
--start-frame        start frame number
--max-frames         # frames to save
--sampling-rates     sampling rates

# Misc
--visualize     visualize using tensorboard

Training Image

For training a single video, use the following command for example:

CUDA_VISIBLE_DEVICES=0 python train_image.py --image-path data/imgs/air_balloons.jpg --vae-levels 3 --checkname myimagetest --visualize

Training baselines for video

For training a single video using SinGan re-implementation, use the following command:

CUDA_VISIBLE_DEVICES=0 python train_video_baselines.py --video-path data/vids/air_balloons.mp4 --checkname myimagetest --visualize --generator GeneratorSG --train-depth 1

Generating Samples

Use eval_*.py to generate samples from an "experiment" folder created during training. The code uses Glob package for multiple experiments evaluation, for example, the following line will generate 100 video samples for all trained movies:

python eval_video.py --num-samples 100 --exp-dir run/**/*/experiment_0

results are saved under run/**/*/experiment_0/eval

In order to extract gifs and images, use the extract_*.py files similarly:

python eval_video.py --max-samples 4 --exp-dir run/**/*/experiment_0/eval

results are saved under run/**/*/experiment_0/eval/gifs(images).

Citation

If you found this work useful, please cite.

@article{gur2020hierarchical,
  title={Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample},
  author={Gur, Shir and Benaim, Sagie and Wolf, Lior},
  journal={arXiv preprint arXiv:2006.12226},
  year={2020}
}

hp-vae-gan's Issues

CUDA error always at the same point

Hi, first of all thanks for sharing your code :)

I'm running COLAB notebook with a short video (2 secs). When I run

!python /content/hp-vae-gan/train_video_baselines.py --video-path /content/movie-2secs.mp4 --checkname mymovie--visualize --generator GeneratorSG --train-depth 1

I always get this error in step 10, no matter the number of iterations (I've lowered to 50 in this example):

RuntimeError: CUDA out of memory. Tried to allocate 270.00 MiB (GPU 0; 15.90 GiB total capacity; 14.64 GiB already allocated; 49.88 MiB free; 14.94 GiB reserved in total by PyTorch)
Training scale [10/10]: 0% 0/50

Is this an error? Can I tweak something to make this work? Many thanks in advance

how to train paint-to-image?

i like this project. tell me, what command can i use for test paint-to-image?

Evaluation code for SVFID

Thank you for opening the source of the excellent work! I noticed that this work first proposed the SVFID metric, however, this seems not included in the code. Could you kindly sharing the code for evaluating the SVFID?

GPU lost issue

Hi, first of all thanks for your nice project and paper.

When i try to train with some image, there is issue that GPU is lost.
The version of pytorch is 1.6.0
and i train with GTX 1080 with cuda 11.2 and TITAN RTX with cuda 11.4.

Sometimes train is finished with no error, but sometimes GPU is lost and train is stop.

Were there any issues like this?

colab

can you please add a google colab