shaohua0116 / multiview2novelview Goto Github PK

Hi,
I am trying to train your network using my own dataset. I want to finetune the model from your pretrained kitti checkpint, but it failed.

I am sorry to bother you, but I would like to consult you about some structural problems. Are you not using the Normalization layer in the whole network or only in the discriminator? I am looking forward to your reply.

Dear Author
Thanks for sharing your interesting work but i have the following questions:

1. What is the specs of the machine you used cause it takes days of running with me and did not finish, for example it is working since 3 days and reached to:
   [2019-03-24 17:43:27,001] [train step 145431] Loss: 4.35295 Pixel loss: 4.03669 Flow loss: 0.31627 (1.589 sec/batch, 5.036 instances/sec)
   and did not finish yet. so am asking what is the number of training steps?
2. the input and output for your network is just images no videos right?
3. I tried the following command :
   python trainer.py --batch_size 8 --dataset car --num_input 4
   but it gives the following error after reaching training step (number train step 4251), do you have any idea why?

[2019-03-23 05:36:24,763] [train step 4261] Loss: 2.96025 Pixel loss: 2.85450 Flow loss: 0.10575 (1.607 sec/batch, 2.489 instances/sec)
Traceback (most recent call last):
File "trainer.py", line 380, in
main()
File "trainer.py", line 377, in main
trainer.train()
File "trainer.py", line 193, in train
opt_gan=s > gan_start_step, is_train=True)
File "trainer.py", line 209, in run_single_step
batch_chunk = self.session.run(batch)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_0_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 4, current size 3)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

Caused by op u'shuffle_batch', defined at:
File "trainer.py", line 380, in
main()
File "trainer.py", line 374, in main
trainer = Trainer(config, dataset_train, dataset_test)
File "trainer.py", line 48, in init
dataset, self.batch_size, is_training=True)
File "/data/ehab/Multiview2NovelviewMaster/input_ops.py", line 76, in create_input_ops
min_after_dequeue=min_capacity,
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 1220, in shuffle_batch
name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 791, in _shuffle_batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 457, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 1342, in _queue_dequeue_many_v2
timeout_ms=timeout_ms, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): RandomShuffleQueue '_0_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 4, current size 3)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

4. i tried to get some figures from tensorboard, but how can i get numerical results published in tables in your paper?
   I really appreciate your time and your reply
   Regards

Hello @shaohua0116

Thank you for sharing such good work!

What are the best number of iterations for datasets after which you perform evaluations?

If this question already raised/explained and I missed it, then I am sorry for raising it again.
Thanks in advance

I might be wrong yet shouldn't the second conv2d input be _?

def residual_block(input, output_shape, is_train, info=False, k=3, s=1,
                   name="residual", activation_fn=lrelu, norm='batch'):
    with tf.variable_scope(name):
        with tf.variable_scope('res1'):
            _ = conv2d(input, output_shape, is_train, k_h=k, k_w=k, s=s,
                       activation_fn=activation_fn, norm=norm)
        with tf.variable_scope('res2'):
            _ = conv2d(input, output_shape, is_train, k_h=k, k_w=k, s=s,
                       activation_fn=None, norm=norm)
        _ = activation_fn(_ + input)
        if info: log.info('{} {}'.format(name, _.get_shape().as_list()))
    return _

Hello, I found a performance issue in the definition of build_loss, model.py, tf.expand_dims(target_image, axis=-1) will be created repeatedly during program execution, resulting in reduced efficiency. I think it should be created before the loop.

Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.

Thanks for sharing your code
I need help to find the line in your code that calculate the aggregate output mentioned in equation to determine estimated target image in page 7 between equation 5 and 6.
Thanks

Dear Sir
Thank you for sharing you work
I tried to use the following command but I got error loading the data as follow:
I use:

python trainer.py --batch_size 4 --dataset kitti --num_input 4 --checkpoint ～/Checkpoints_KITTI/model-1

I got the following:

Traceback (most recent call last):
File "trainer.py", line 380, in
main()
File "trainer.py", line 369, in main
dataset.create_default_splits(config.num_input)
File "/data/ehab/Multiview2NovelviewMaster/datasets/kitti.py", line 132, in create_default_splits
bound=bound)
File "/data/ehab/Multiview2NovelviewMaster/datasets/kitti.py", line 35, in init
self.data = h5py.File(file, 'r')
File "/usr/lib64/python2.7/site-packages/h5py/_hl/files.py", line 394, in init
swmr=swmr)
File "/usr/lib64/python2.7/site-packages/h5py/_hl/files.py", line 170, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 85, in h5py.h5f.open
IOError: Unable to open file (unable to open file: name = './datasets/kitti/data_kitti.hdf5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

Dear sir:
thanks for sharing your code!
but I meet some strange problems when trainning,

the image list is totally different from yours, but I do the trainning with the code you released, I didn't change anything~
Waiting for your reply

Dear Author
Thank you for sharing your code
I have a question about flow module using appearance flow network, in [23] they reported that they used Disocclusionaware Appearance Flow Network, Did you used DOAFN or just AFN?
Thanks

Hello dear authors,

In your source code, we can set the normalization parameter to 'batch', 'instance' normalization or the default 'None'.

Can I ask that is there a reason why normalization is off by default? Did you have the time to experiment with normalization as well? Didn't it contribute to better results?

Any information is greatly appreciated.
Thanks.

Hello,
I am trying to evaluate model from checkpoints provided by you.
I have referred to descriptions provided by you in Github.
Thus I used
python evaler.py --dataset kitti --data_id_list ./testing_tuple_lists/id_kitti.txt --checkpoint ./kitti_checkpoint/model-1 --loss True --write_summary True --summary_file log_kitti.txt --plot_image True --output_dir img_kitti .
However, I get an error: OutOfRangeError (see above for traceback): Read less bytes than requested

I would appreciate if anyone could help me.

Hello,
In the equation (5) in your paper, there is a element-wise square operator like this:

I believe this loss function was defined in this piece of code
l1_loss += tf.reduce_mean(loss_map * normalized_mask) * current_weight / \ (int(img.get_shape()[1]) * int(img.get_shape()[2])) * regularizer_weight

I wonder it should be like this
l1_loss += tf.reduce_mean((loss_map**2) * normalized_mask) * current_weight / \ (int(img.get_shape()[1]) * int(img.get_shape()[2])) * regularizer_weight

Can u verify it is correct ? If your code is correct then I am sorry for my question. I just confused when I see the loss like that. Thanks !

Dear Author
Thanks for sharing your interesting work but i have the following questions:

What is the specs of the machine you used cause it takes days of running with me and did not finish, for example it is working since 3 days and reached to:
[2019-03-24 17:43:27,001] [train step 145431] Loss: 4.35295 Pixel loss: 4.03669 Flow loss: 0.31627 (1.589 sec/batch, 5.036 instances/sec)
and did not finish yet. so am asking what is the number of training steps?
the input and output for your network is just images no videos right?
I tried the following command :
python trainer.py --batch_size 8 --dataset car --num_input 4
but it gives the following error after reaching training step (number train step 4251), do you have any idea why?
[2019-03-23 05:36:24,763] [train step 4261] Loss: 2.96025 Pixel loss: 2.85450 Flow loss: 0.10575 (1.607 sec/batch, 2.489 instances/sec)
Traceback (most recent call last):
File "trainer.py", line 380, in
main()
File "trainer.py", line 377, in main
trainer.train()
File "trainer.py", line 193, in train
opt_gan=s > gan_start_step, is_train=True)
File "trainer.py", line 209, in run_single_step
batch_chunk = self.session.run(batch)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
run_metadata_ptr)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_0_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 4, current size 3)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

Caused by op u'shuffle_batch', defined at:
File "trainer.py", line 380, in
main()
File "trainer.py", line 374, in main
trainer = Trainer(config, dataset_train, dataset_test)
File "trainer.py", line 48, in init
dataset, self.batch_size, is_training=True)
File "/data/ehab/Multiview2NovelviewMaster/input_ops.py", line 76, in create_input_ops
min_after_dequeue=min_capacity,
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 1220, in shuffle_batch
name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 791, in _shuffle_batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 457, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 1342, in _queue_dequeue_many_v2
timeout_ms=timeout_ms, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): RandomShuffleQueue '_0_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 4, current size 3)
[[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

i tried to get some figures from tensorboard, but how can i get numerical results published in tables in your paper?
I really appreciate your time and your reply
Regards

Hello sirs,
I am trying to implement your code with my own small dataset.
Is that okay if you can share how you create the dataset in case of ShapeNet or KITTI.

Dear Author
Thank you for sharing your code.
I have some questions about quantitative indicators and visualization results.
When I used the KITTI model you provided for the test, I found that the SSIM index was higher than the index shown in your paper. Moreover, when I used the code you provided to train KITTI, the final loss of single picture input would converge to 0.198, but the visual effect of the output picture was not as good as the result you provided. What is the reason?

Thanks for the code. Was wondering if you code also post the supplementary material for reference as well.

Hi,

I would like to start by thanking you for sharing your code and trained models. I'm working these days on an undergraduate project with a partner. We are trying to use your model on general data without your supplied poses data using SfM algorithms.

For our project, we tried to use the data from KITTI's official website and couldn't make it work on your trained model. We understood the problem was that your poses data was different than the data on KITTI's website.

Can you please explain how did you create your poses data from the data supplied on the KITTI website?

Hello dear authors,

In your dataset files (e.g. shapenet_car.py), a foreground mask is also concatenated to the image tensor. Can I know what is the purpose of it? Unfortunately, I can not find where it is used.

Is it related to calculating the loss? Maybe opting out the background pixels and only calculating loss on the object's pixels?

Any information is greatly appreciated.
Thanks.