pfnet-research / tgan

The implementation of Temporal Generative Adversarial Nets with Singular Value Clipping

Temporal Generative Adversarial Nets

The new version of TGAN has been published and the code is available: TGANv2.

This repository contains a collection of scripts used in the experiments of Temporal Generative Adversarial Nets with Singular Value Clipping.

Disclaimer: PFN provides no warranty or support for this implementation. Use it at your own risk. See license for details.

Results

Requirements

These scripts require the following python libraries.

• Chainer 2.0.0+
• h5py
• numpy
• pandas
• PIL
• PyYAML
• matplotlib

Note that they also require ffmpeg to produce a video from a set of images.

Usage

Datasets

In order to run our scripts, you need to prepare MovingMNIST and UCF-101 datasets as follows.

MovingMNIST

1. Download mnist_test_seq.npy from here.
2. Put it on path-to-tgans/data/mnist_test_seq.npy.

UCF-101

There are two ways to create an UCF-101 dataset for this script.

1. Transforms all the videos in the UCF-101 dataset to the images.
2. Resizes these images to the appropriate resolution, and concatenate them into as single hdf5 format represented as (time, channel, rows, cols). In this transformation we used make_ucf101.py in this repository. Note that this script also produces a config file that describes videos and these corresponding label information.
3. puts them on path-to-tgans/data.

Another way is to simply download these files; please download them from this url, and put them on the same directory.

Training

TGAN with WGAN and Singular Value Clipping

python train.py --config_path configs/moving_mnist/mnist_wgan_svd_zdim-100_no-beta-all_init-uniform-all.yml --gpu 0
python train.py --config_path configs/ucf101/ucf101_wgan_svd_zdim-100_no-beta.yml --gpu 0

TGAN (WGAN and weight clipping)

python train.py --config_path configs/moving_mnist/mnist_wgan_clip_zdim-100_no-beta-all_init-uniform-all.yml --gpu 0
python train.py --config_path configs/ucf101/ucf101_wgan_clip_zdim-100_no-beta.yml --gpu 0

TGAN (vanilla GAN)

python train.py --config_path configs/ucf101/ucf101_vanilla_zdim-100_no-beta.yml --gpu 0

Quantitative evaluation on UCF101 (2019/08/20)

We have uploaded mean2.npz on GitHub because there are many inquiries about the mean file in the UCF101. If you want to perform a quantitative evaluation, please download it from this url.

Citation

Please cite the paper if you are interested in:

@inproceedings{TGAN2017,
    author = {Saito, Masaki and Matsumoto, Eiichi and Saito, Shunta},
    title = {Temporal Generative Adversarial Nets with Singular Value Clipping},
    booktitle = {ICCV},
    year = {2017},
}

License

MIT License. Please see the LICENSE file for details.

tgan's Issues

pretrained models

Could you please provide the weights for the final trained models on the different datasets. Thanks.

mean2.npz file for inception score computation

You load files mean2.npz file in

evaluation/calc_inception.py (line 36)
and
evaluation/tile_prediction.py (line 48)

and substract it from generated videos before feeding into c3d net. It looks like some sort of normalization, but you do not explain anywhere in the code nor the computation of that mean file is anywhere in the code. Could you clarify what this mean is and why you are substracting it?

Also, is it possible to provide the file?

Thanks.

training procedure of WGAN

Hi, excellent work! I have a question that in origin WGAN paper and its code, in one iteration discriminator is trained several sub-iterations before generator is trained, but that's not the case in your code on weight clipping method. So is this method tried and it is useless?

Where is the condition TGAN?

Hi, the paper says that you also implemented Conditional TGAN which can take labels as the condition. Did you upload the code of conditional TGAN? I could not find it.

ValueError: unsupported pickle protocol: 4

Hi, I downloaded the ucf-101 dataset using the url provided. However, when I try to run the following command:
train.py --config_path configs/ucf101/ucf101_wgan_svd_zdim-100_no-beta.yml --gpu 0

the error 'ValueError: unsupported pickle protocol: 4' pops out.
which is caused by the code
self.conf = pd.read_pickle(config_path)

It seems that it is caused by the conflict of python version. Could you please fix the bug for python 2.7 ?

Inception Score on UCF101

Hi,
nice work. I have a small question about the inception score evaluation procedure.
As referenced in #8
the normalization file for C3D network was not provided with the code.
The proposed solution has a file, that was calculated for image resolution 112 * 112,
while you are using 128 * 171.

Could you, please, explain how it was calculated?
Is it taken from .binaryproto normalization file of Caffe C3D model, finetuned on UCF
https://github.com/facebook/C3D/tree/master/C3D-v1.0/examples/c3d_finetuning
or the initial model trained on Sports-1M
https://github.com/facebook/C3D/tree/master/C3D-v1.0/examples/c3d_feature_extraction ?
Do you perform any additional transformations?

Thank you in advance,
Vlad

chainer issues

Here's just a small snippet of the error chain: File "/home/bcichy/anaconda3/lib/python3.6/site-packages/chainer/dataset/dataset_mixin.py", line 67, in getitem
return self.get_example(index)
File "datasets/ucf_dataset.py", line 37, in get_example
x = self._crop_center(x)
File "datasets/ucf_dataset.py", line 26, in _crop_center
assert x.shape[2] == self.img_size
AssertionError

Is there a specific chainer version to run (+ cupy) because I don't see a requirements file for this project.

https://www.dropbox.com/sh/j9fsakeuvicpeo8/AAD6BVhbZRyi7NXaMfn6TO4da?dl=0

https://www.dropbox.com/sh/j9fsakeuvicpeo8/AAD6BVhbZRyi7NXaMfn6TO4da?dl=0
500 error

Is there any pretrained model?

Hello! Do you have any pretrained model that we can use to generate a few samples?
Thanks

tgan on golf videos

Hi, when I try to run tgan on golf videos two things are not working:

1. when I use the wgan_clip configuration the update updaters/tgan_updater_wgan_clip.py is not part of this repository

2. when I use the wgan_svd configuration I get the following error:

Traceback (most recent call last):
  File "train.py", line 188, in <module>
    sys.exit(main())
  File "train.py", line 183, in main
    trainer.run()
  File "/scratch_net/boyscouts/kratzwab/apps/python-2.7/lib/python2.7/site-packages/chainer/training/trainer.py", line 313, in run
    six.reraise(*sys.exc_info())
  File "..../apps/python-2.7/lib/python2.7/site-packages/chainer/training/trainer.py", line 299, in run
    update()
  File ".../apps/python-2.7/lib/python2.7/site-packages/chainer/training/updater.py", line 223, in update
    self.update_core()
  File "updaters/tgan_updater_wgan.py", line 107, in update_core
    for name, p in namedparams.items() if p.data.ndim >= 4]
AttributeError: 'numpy.ndarray' object has no attribute 'get'

Do you know how I can fix this?