clvrai / relation-network-tensorflow Goto Github PK

There are some errors when I want to restore the model from checkpoint.
I changed 'self.pretrain_saver.restore(self.session, self.ckpt_path)'
to ' self.saver.restore(self.session, self.ckpt_path) '
It worked.
Is that right?

The N_GRID value is not set in the generator.py, I've set it to 256 but I'm not sure. Also line 53 seems a typo x = idx_coor[i] % NGRID_ should be x = idx_coor[i] % N_GRID

Then when I try python trainer.py it raises ValueError('num_outputs should be int or long, got %s.', num_outputs) ValueError: ('num_outputs should be int or long, got %s.', 10) maybe it's related to my N_GRID guess... ;)

num_outputs is 10 <class 'numpy.int64' at 0x102e01418> a NumPy type but the TensorFlow library is expecting regular int or long type

As is mentioned,during the training phase, the samples including images, questions, ground truth answers, and predicted answers can be monitored on Tensorboard if the library Tensorflow Plot is installed,how could i operate to monitor this process???
I see tfplot and seaborn two modules in link"Tensorflow Plot",do i need to install seaborn either???
And also if i want to visualize the CNN internal network,what should i do???
Thanks for your guide!!!

when i run python evaler.py --dataset_path Sort-of-CLEVR_default --model rn --checkpoint_path train_dir/baseline-default-Sort-of-CLEVR_default_lr_0.00025-20170819-213535/model-10001
it cannot start the inference and evaluation,the errors are like this:
2017-09-06 23:04:15.889399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1052] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 660, pci bus id: 0000:01:00.0, compute capability: 3.0)
[2017-09-06 23:04:16,316] Using Model class : <class 'model_rn.Model'>
[2017-09-06 23:04:16,318] CONV
[2017-09-06 23:04:16,432] CONV/g_theta
[2017-09-06 23:04:20,278] f_phi
[2017-09-06 23:04:20,335] Successfully loaded the model.
WARNING:tensorflow:From evaler.py:96: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
2017-09-06 23:04:20.338655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1052] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 660, pci bus id: 0000:01:00.0, compute capability: 3.0)
[2017-09-06 23:04:20,407] Checkpoint path : train_dir/baseline-default-Sort-of-CLEVR_default_lr_0.00025-20170819-213535/model-10001
[2017-09-06 23:04:20,407] dataset: Sort-of-CLEVR_default
2017-09-06 23:04:20.725816: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key f_phi/fully_connected_2/weights not found in checkpoint
2017-09-06 23:04:20.726046: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key f_phi/fully_connected_2/weights not found in checkpoint
[[Node: save/RestoreV2_37 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_37/tensor_names, save/RestoreV2_37/shape_and_slices)]]
2017-09-06 23:04:20.726102: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_4/BatchNorm/beta not found in checkpoint
2017-09-06 23:04:20.728558: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_1/BatchNorm/beta not found in checkpoint
2017-09-06 23:04:20.731861: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_3/w not found in checkpoint
2017-09-06 23:04:20.732235: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key f_phi/fully_connected_2/biases not found in checkpoint
2017-09-06 23:04:20.732289: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_4/BatchNorm/gamma not found in checkpoint
2017-09-06 23:04:20.734357: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_3/biases not found in checkpoint
2017-09-06 23:04:20.734468: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_1/BatchNorm/gamma not found in checkpoint
2017-09-06 23:04:20.736531: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_4/BatchNorm/moving_mean not found in checkpoint
2017-09-06 23:04:20.737279: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key f_phi/fully_connected_1/weights not found in checkpoint
2017-09-06 23:04:20.739398: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_3/BatchNorm/moving_variance not found in checkpoint
2017-09-06 23:04:20.741029: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_1/BatchNorm/moving_variance not found in checkpoint
2017-09-06 23:04:20.742882: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key f_phi/fully_connected_1/biases not found in checkpoint
2017-09-06 23:04:20.743091: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_1/BatchNorm/moving_mean not found in checkpoint
2017-09-06 23:04:20.745361: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_3/BatchNorm/moving_mean not found in checkpoint
2017-09-06 23:04:20.745434: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_1/biases not found in checkpoint
2017-09-06 23:04:20.747760: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key f_phi/fully_connected/weights not found in checkpoint
2017-09-06 23:04:20.747760: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_1/w not found in checkpoint
2017-09-06 23:04:20.750858: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_3/BatchNorm/gamma not found in checkpoint
2017-09-06 23:04:20.751412: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key f_phi/fully_connected/biases not found in checkpoint
2017-09-06 23:04:20.752404: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_4/BatchNorm/moving_variance not found in checkpoint
2017-09-06 23:04:20.754160: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_2/BatchNorm/beta not found in checkpoint
2017-09-06 23:04:20.755551: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/g_theta/fully_connected_3/weights not found in checkpoint
2017-09-06 23:04:20.755632: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_3/BatchNorm/beta not found in checkpoint
2017-09-06 23:04:20.755763: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_4/biases not found in checkpoint
2017-09-06 23:04:20.758080: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_2/BatchNorm/gamma not found in checkpoint
2017-09-06 23:04:20.760821: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_2/w not found in checkpoint
2017-09-06 23:04:20.762303: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_4/w not found in checkpoint
2017-09-06 23:04:20.762356: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/g_theta/fully_connected_3/biases not found in checkpoint
2017-09-06 23:04:20.763147: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_2/BatchNorm/moving_mean not found in checkpoint
2017-09-06 23:04:20.763228: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_2/biases not found in checkpoint
2017-09-06 23:04:20.767768: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/g_theta/fully_connected_2/weights not found in checkpoint
2017-09-06 23:04:20.767895: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/conv_2/BatchNorm/moving_variance not found in checkpoint
2017-09-06 23:04:20.774584: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/g_theta/fully_connected/biases not found in checkpoint
2017-09-06 23:04:20.775900: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/g_theta/fully_connected_1/biases not found in checkpoint
2017-09-06 23:04:20.776379: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/g_theta/fully_connected_2/biases not found in checkpoint
2017-09-06 23:04:20.776488: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/g_theta/fully_connected/weights not found in checkpoint
2017-09-06 23:04:20.777465: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key CONV/g_theta/fully_connected_1/weights not found in checkpoint
Traceback (most recent call last):
File "evaler.py", line 223, in
main()
File "evaler.py", line 220, in main
evaler.eval_run()
File "evaler.py", line 123, in eval_run
self.saver.restore(self.session, self.checkpoint_path)
File "/home/han/tensorflow/_python_build/tensorflow/python/training/saver.py", line 1560, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/han/tensorflow/_python_build/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/han/tensorflow/_python_build/tensorflow/python/client/session.py", line 1118, in _run
feed_dict_tensor, options, run_metadata)
File "/home/han/tensorflow/_python_build/tensorflow/python/client/session.py", line 1315, in _do_run
options, run_metadata)
File "/home/han/tensorflow/_python_build/tensorflow/python/client/session.py", line 1334, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key f_phi/fully_connected_2/weights not found in checkpoint
[[Node: save/RestoreV2_37 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_37/tensor_names, save/RestoreV2_37/shape_and_slices)]]
[[Node: save/RestoreV2_4/_73 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_156_save/RestoreV2_4", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

Caused by op u'save/RestoreV2_37', defined at:
File "evaler.py", line 223, in
main()
File "evaler.py", line 217, in main
evaler = Evaler(config, dataset_test)
File "evaler.py", line 109, in init
self.saver = tf.train.Saver(max_to_keep=100)
File "/home/han/tensorflow/_python_build/tensorflow/python/training/saver.py", line 1140, in init
self.build()
File "/home/han/tensorflow/_python_build/tensorflow/python/training/saver.py", line 1172, in build
filename=self._filename)
File "/home/han/tensorflow/_python_build/tensorflow/python/training/saver.py", line 688, in build
restore_sequentially, reshape)
File "/home/han/tensorflow/_python_build/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
File "/home/han/tensorflow/_python_build/tensorflow/python/training/saver.py", line 247, in restore_op
[spec.tensor.dtype])[0])
File "/home/han/tensorflow/_python_build/tensorflow/python/ops/gen_io_ops.py", line 991, in restore_v2
shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
File "/home/han/tensorflow/_python_build/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/han/tensorflow/_python_build/tensorflow/python/framework/ops.py", line 3046, in create_op
op_def=op_def)
File "/home/han/tensorflow/_python_build/tensorflow/python/framework/ops.py", line 1604, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

NotFoundError (see above for traceback): Key f_phi/fully_connected_2/weights not found in checkpoint
[[Node: save/RestoreV2_37 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_37/tensor_names, save/RestoreV2_37/shape_and_slices)]]
[[Node: save/RestoreV2_4/_73 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_156_save/RestoreV2_4", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
And i also tried the other file(model-1.data-00000-of-00001,model-1.index,model-1.meta),it seems to occur the same problem,and when i omitted the checkpoint path,it occurs odd results:
[2017-09-06 23:11:59,187] Average accuracy of non-relational questions: 0.488599348534%
[2017-09-06 23:11:59,187] Average accuracy of relational questions: 12.1654501217%
[2017-09-06 23:11:59,187] Average accuracy: 5.17073170732%
[2017-09-06 23:11:59,187] Evaluation complete.
Please give me some tips about this,thank you~

Now, I've got an error with a None argument, in fact fetch is None at line 162 in trainer.py. It seems to be an error with the model but I'm not sure... also step is 0 ??? it looks a bit strange to me.

python trainer.py
[2017-06-24 20:09:54,919] Reading ./datasets/Sort-of-CLEVR_default/data.hy ...
[2017-06-24 20:09:54,929] Reading Done: ./datasets/Sort-of-CLEVR_default/data.hy
[2017-06-24 20:09:54,929] Reading ./datasets/Sort-of-CLEVR_default/data.hy ...
[2017-06-24 20:09:54,930] Reading Done: ./datasets/Sort-of-CLEVR_default/data.hy
[2017-06-24 20:09:54,930] Train Dir: ./train_dir/rn-default-Sort-of-CLEVR_default_lr_0.00025-20170624-200954
[2017-06-24 20:09:54,930] input_ops [inputs]: Using 8000 IDs from dataset
[2017-06-24 20:09:55,082] input_ops [inputs]: Using 2000 IDs from dataset
[2017-06-24 20:09:56,609] Using Model class : <class 'model_rn.Model' at 0x10ebe18d8>
[2017-06-24 20:09:56,614] CONV
[2017-06-24 20:09:56,984] CONV/g_theta
[2017-06-24 20:10:06,848] f_phi
[2017-06-24 20:10:07,013] Successfully loaded the model.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
[2017-06-24 20:11:30,513] dataset: Sort-of-CLEVR_default, learning_rate: 0.000250
[2017-06-24 20:11:30,514] Training Starts!
{'a': <tf.Tensor 'shuffle_batch:0' shape=(16, 10) dtype=float32>,
'id': <tf.Tensor 'shuffle_batch:1' shape=(16,) dtype=string>,
'img': <tf.Tensor 'shuffle_batch:2' shape=(16, 128, 128, 3) dtype=float32>,
'q': <tf.Tensor 'shuffle_batch:3' shape=(16, 11) dtype=float32>}
Traceback (most recent call last):
File "trainer.py", line 242, in
main()
File "trainer.py", line 239, in main
trainer.train()
File "trainer.py", line 130, in train
self.run_single_step(self.batch_train, step=s, is_train=True)
File "trainer.py", line 162, in run_single_step
fetch, feed_dict=self.model.get_feed_dict(batch_chunk, step=step)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 767, in run
run_metadata_ptr)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 952, in _run
fetch_handler = _FetchHandler(self._graph, fetches, feed_dict_string)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 408, in init
self._fetch_mapper = _FetchMapper.for_fetch(fetches)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 230, in for_fetch
return _ListFetchMapper(fetch)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 337, in init
self._mappers = [_FetchMapper.for_fetch(fetch) for fetch in fetches]
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 337, in
self._mappers = [_FetchMapper.for_fetch(fetch) for fetch in fetches]
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 227, in for_fetch
(fetch, type(fetch)))
TypeError: Fetch argument None has invalid type <class 'NoneType' at 0x10022ce60>

Could you please explain all the functions these .py files want to achieve,for example,you use generator.py to generate datasets with different shapes ,what does sort_of_clevr.py want to do???
And what about ops.py, input_ops.py, util.py,vqa_util.py???
Actually reading Class python is hard for me because I am a newer in this field and hope you would help me ,thank you very much~~~

You aplpy tf.nn.softmax inside your model and then apply tf.nn.softmax_cross_entropy_with_logits to that softmax:

self.all_preds = tf.nn.softmax(logits)
loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels)

however, the tensorflow documentation clearly states

"WARNING: This op expects unscaled logits, since it performs a softmax on logits internally for efficiency. Do not call this op with the output of softmax, as it will produce incorrect results."

This seems like a bug?

I meet some problems
when i run trainer.py,it ends like:
[2017-08-19 11:16:12,451] [train step 199991]
Exception AttributeError: AttributeError("'NoneType' object has no attribute 'ScopedTFStatus'",) in <bound method Context.del of <tensorflow.python.eager.context.Context object at 0x7febc2729210>> ignored
when i run eval.py,there shows:
W tensorflow/core/framework/op_kernel.cc:1192] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for ckpt_dir
So i guess train_dir/baseline-Sort-of-CLEVR_default_lr_0.001-20170619-040301/model-10001
doesn't exist,and i don't know why,please help me ~~~

As the comment for Q2 in generator.py says, the question is generated to check whether the shape is in the bottom of the image or not, but then the value of y is checked to be greater than half of the image.

# Q2: bottom?
if rep.y[i] > int(img_size/2):
    A[i*NUM_Q+1, NUM_COLOR+2] = True
else:
    A[i*NUM_Q+1, NUM_COLOR+3] = True

Shouldn't it be if rep.y[i] < int(img_size/2): ?

Hi, I'm trying to find a source for the Sort-of-CLEVR Dataset. The provided code in this repository seems to be what I'm looking for, but I need help understanding how to set it up for training, validation, and testing. Could you provide a brief example of how the included code could be used to generate a training, validation, and testing set, and then from this, iterate through these datasets in batches?

If I want to test on CLEVR, what is the difference of these codes,mainly modifying what files???
Thank you~

Then when I try python trainer.py it raises ValueError('num_outputs should be int or long, got %s.', num_outputs) ValueError: ('num_outputs should be int or long, got %s.', 10).

It seems not related to the generation process that is working now

I've checked and num_outputs 10 is <class 'numpy.int64' at 0x102e01418> a NumPy type int64 but the TensorFlow library is expecting regular int or long type.

Traceback (most recent call last):
File "trainer.py", line 242, in
main()
File "trainer.py", line 235, in main
dataset_train, dataset_test)
File "trainer.py", line 57, in init
self.model = Model(config)
File "/Users/claudecoulombe/git/Relation-Network-Tensorflow/model_rn.py", line 47, in init
self.build(is_train=is_train)
File "/Users/claudecoulombe/git/Relation-Network-Tensorflow/model_rn.py", line 131, in build
logits = f_phi(g, scope='f_phi')
File "/Users/claudecoulombe/git/Relation-Network-Tensorflow/model_rn.py", line 127, in f_phi
fc_3 = fc(fc_2, n, activation_fn=None, name='fc_3')
File "/Users/claudecoulombe/git/Relation-Network-Tensorflow/ops.py", line 42, in fc
output = slim.fully_connected(input, output_shape, activation_fn=activation_fn)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 177, in func_with_args
return func(*args, **current_args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1386, in fully_connected
raise ValueError('num_outputs should be int or long, got %s.', num_outputs)
ValueError: ('num_outputs should be int or long, got %s.', 10)

My configuration:

Python 3.6.0a2
TensorFlow 1.0.1
NumPy 1.12.1
PIL 1.1.7
matplotlib 1.5.1
h5py 2.7.0
progressbar 3.30.2

Is it possible that the TensorFlow version 1.0.1 which differs from the prerequisite 1.0.0 could explain the problem?

I see a lot of experiments in ReadMe.md,could you give me some instructions about how to achieve this???

In the file model_rn.py, can we have a better way to implement this double for loop? to speedup graph generation.

for i in range(d*d):
                   o_i = conv_4[:, int(i / d), int(i % d), :]
                   o_i = concat_coor(o_i, i, d)
                   for j in range(d*d):
                       o_j = conv_4[:, int(j / d), int(j % d), :]
                       o_j = concat_coor(o_j, j, d)
                       if i == 0 and j == 0:
                           g_i_j = g_theta(o_i, o_j, q, reuse=False)
                       else:
                           g_i_j = g_theta(o_i, o_j, q, reuse=True)
                       all_g.append(g_i_j)