On linux, tensorflow 1.13.0rc1/2
tried Tensorflow 1.14
here is what i get..
$ python3 runGan.py 4
Testing test case 4
Using TensorFlow backend.
Preparing train_data
[Config] Use random crop
[Config] Use random crop
[Config] Use random flip
Sequenced batches: 27610, sequence length: 10
Preparing validation_data
[Config] Use random crop
[Config] Use random crop
[Config] Use random flip
Sequenced batches: 4400, sequence length: 10
tData count = 27610, steps per epoch 6902
Finish building the network.
Scope generator:
Variable: generator/generator_unit/input_stage/conv/Conv/weights:0
Shape: [3, 3, 51, 64]
Variable: generator/generator_unit/input_stage/conv/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_1/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_1/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_1/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_1/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_2/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_2/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_2/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_2/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_3/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_3/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_3/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_3/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_4/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_4/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_4/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_4/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_5/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_5/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_5/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_5/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_6/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_6/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_6/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_6/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_7/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_7/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_7/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_7/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_8/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_8/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_8/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_8/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_9/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_9/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_9/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_9/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_10/conv_1/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_10/conv_1/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/resblock_10/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/resblock_10/conv_2/Conv/biases:0
Shape: [64]
Variable: generator/generator_unit/conv_tran2highres/conv_tran1/Conv2d_transpose/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/conv_tran2highres/conv_tran1/Conv2d_transpose/biases:0
Shape: [64]
Variable: generator/generator_unit/conv_tran2highres/conv_tran2/Conv2d_transpose/weights:0
Shape: [3, 3, 64, 64]
Variable: generator/generator_unit/conv_tran2highres/conv_tran2/Conv2d_transpose/biases:0
Shape: [64]
Variable: generator/generator_unit/output_stage/conv/Conv/weights:0
Shape: [3, 3, 64, 3]
Variable: generator/generator_unit/output_stage/conv/Conv/biases:0
Shape: [3]
total size: 843587
Scope fnet:
Variable: fnet/autoencode_unit/encoder_1/conv_1/Conv/weights:0
Shape: [3, 3, 6, 32]
Variable: fnet/autoencode_unit/encoder_1/conv_1/Conv/biases:0
Shape: [32]
Variable: fnet/autoencode_unit/encoder_1/conv_2/Conv/weights:0
Shape: [3, 3, 32, 32]
Variable: fnet/autoencode_unit/encoder_1/conv_2/Conv/biases:0
Shape: [32]
Variable: fnet/autoencode_unit/encoder_2/conv_1/Conv/weights:0
Shape: [3, 3, 32, 64]
Variable: fnet/autoencode_unit/encoder_2/conv_1/Conv/biases:0
Shape: [64]
Variable: fnet/autoencode_unit/encoder_2/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: fnet/autoencode_unit/encoder_2/conv_2/Conv/biases:0
Shape: [64]
Variable: fnet/autoencode_unit/encoder_3/conv_1/Conv/weights:0
Shape: [3, 3, 64, 128]
Variable: fnet/autoencode_unit/encoder_3/conv_1/Conv/biases:0
Shape: [128]
Variable: fnet/autoencode_unit/encoder_3/conv_2/Conv/weights:0
Shape: [3, 3, 128, 128]
Variable: fnet/autoencode_unit/encoder_3/conv_2/Conv/biases:0
Shape: [128]
Variable: fnet/autoencode_unit/decoder_1/conv_1/Conv/weights:0
Shape: [3, 3, 128, 256]
Variable: fnet/autoencode_unit/decoder_1/conv_1/Conv/biases:0
Shape: [256]
Variable: fnet/autoencode_unit/decoder_1/conv_2/Conv/weights:0
Shape: [3, 3, 256, 256]
Variable: fnet/autoencode_unit/decoder_1/conv_2/Conv/biases:0
Shape: [256]
Variable: fnet/autoencode_unit/decoder_2/conv_1/Conv/weights:0
Shape: [3, 3, 256, 128]
Variable: fnet/autoencode_unit/decoder_2/conv_1/Conv/biases:0
Shape: [128]
Variable: fnet/autoencode_unit/decoder_2/conv_2/Conv/weights:0
Shape: [3, 3, 128, 128]
Variable: fnet/autoencode_unit/decoder_2/conv_2/Conv/biases:0
Shape: [128]
Variable: fnet/autoencode_unit/decoder_3/conv_1/Conv/weights:0
Shape: [3, 3, 128, 64]
Variable: fnet/autoencode_unit/decoder_3/conv_1/Conv/biases:0
Shape: [64]
Variable: fnet/autoencode_unit/decoder_3/conv_2/Conv/weights:0
Shape: [3, 3, 64, 64]
Variable: fnet/autoencode_unit/decoder_3/conv_2/Conv/biases:0
Shape: [64]
Variable: fnet/autoencode_unit/output_stage/conv1/Conv/weights:0
Shape: [3, 3, 64, 32]
Variable: fnet/autoencode_unit/output_stage/conv1/Conv/biases:0
Shape: [32]
Variable: fnet/autoencode_unit/output_stage/conv2/Conv/weights:0
Shape: [3, 3, 32, 2]
Variable: fnet/autoencode_unit/output_stage/conv2/Conv/biases:0
Shape: [2]
total size: 1745506
The first run takes longer time for training data loading...
Save initial checkpoint, before any training
[testWhileTrain] step 0:
python3 main.py --output_dir ex_FRVSR06-21-16/train/ --summary_dir ex_FRVSR06-21-16/train/ --mode inference --num_resblock 10 --checkpoint ex_FRVSR06-21-16/model-0 --cudaID 0 --input_dir_LR ./LR/calendar/ --output_pre --output_name 000000000 --input_dir_len 10
Using TensorFlow backend.
input shape: [1, 144, 180, 3]
output shape: [1, 576, 720, 3]
Finish building the network
Loading weights from ckpt model
Frame evaluation starts!!
Traceback (most recent call last):
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node generator/generator_unit/input_stage/conv/Conv/Conv2D}}]]
[[{{node generator/Assign_1}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 263, in
output_frame = sess.run(outputs, feed_dict=feed_dict)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node generator/generator_unit/input_stage/conv/Conv/Conv2D (defined at /media/coles/New Volume/TecoGAN-master/lib/ops.py:55) ]]
[[node generator/Assign_1 (defined at main.py:211) ]]
Caused by op 'generator/generator_unit/input_stage/conv/Conv/Conv2D', defined at:
File "main.py", line 208, in
gen_output = generator_F(inputs_all, 3, reuse=False, FLAGS=FLAGS)
File "/media/coles/New Volume/TecoGAN-master/lib/frvsr.py", line 62, in generator_F
net = conv2(gen_inputs, 3, 64, 1, scope='conv')
File "/media/coles/New Volume/TecoGAN-master/lib/ops.py", line 55, in conv2
activation_fn=None, weights_initializer=tf.contrib.layers.xavier_initializer())
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1155, in convolution2d
conv_dims=2)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
return func(*args, **current_args)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1058, in convolution
outputs = layer.apply(inputs)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1227, in apply
return self.call(inputs, *args, **kwargs)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 530, in call
outputs = super(Layer, self).call(inputs, *args, **kwargs)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 554, in call
outputs = self.call(inputs, *args, **kwargs)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/keras/layers/convolutional.py", line 194, in call
outputs = self._convolution_op(inputs, self.kernel)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 966, in call
return self.conv_op(inp, filter)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 591, in call
return self.call(inp, filter)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 208, in call
name=self.name)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1026, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node generator/generator_unit/input_stage/conv/Conv/Conv2D (defined at /media/coles/New Volume/TecoGAN-master/lib/ops.py:55) ]]
[[node generator/Assign_1 (defined at main.py:211) ]]
Traceback (most recent call last):
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.CancelledError: Dequeue operation was cancelled
[[{{node load_frame_cpu/validation_data/shuffle_batch}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 391, in
results = sess.run(fetches)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 676, in run
run_metadata=run_metadata)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1171, in run
run_metadata=run_metadata)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1270, in run
raise six.reraise(*original_exc_info)
File "/home/coles/.local/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run
return self._sess.run(*args, **kwargs)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1327, in run
run_metadata=run_metadata)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1091, in run
return self._sess.run(*args, **kwargs)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.CancelledError: Dequeue operation was cancelled
[[node load_frame_cpu/validation_data/shuffle_batch (defined at /media/coles/New Volume/TecoGAN-master/lib/dataloader.py:270) ]]
Caused by op 'load_frame_cpu/validation_data/shuffle_batch', defined at:
File "main.py", line 284, in
rdata = frvsr_gpu_data_loader(FLAGS, useValidat)
File "/media/coles/New Volume/TecoGAN-master/lib/dataloader.py", line 297, in frvsr_gpu_data_loader
vald_batch_list, vald_num_image_list_HR_t_cur = loadHRfunc(valFLAGS, tar_size)
File "/media/coles/New Volume/TecoGAN-master/lib/dataloader.py", line 270, in loadHR
min_after_dequeue=FLAGS.video_queue_capacity, num_threads=FLAGS.queue_thread, seed = FLAGS.rand_seed)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/training/input.py", line 1346, in shuffle_batch
name=name)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/training/input.py", line 873, in _shuffle_batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/ops/data_flow_ops.py", line 488, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3645, in queue_dequeue_many_v2
timeout_ms=timeout_ms, name=name)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/coles/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()
CancelledError (see above for traceback): Dequeue operation was cancelled
[[node load_frame_cpu/validation_data/shuffle_batch (defined at /media/coles/New Volume/TecoGAN-master/lib/dataloader.py:270) ]]