zubata88 / mdgru Goto Github PK

When running the train+test experiment as described in
https://zubata88.github.io/mdgru/how_to_use.html#train-test
the code ran into the an error, apparently during evaluation phase.
Find the details below. I also attach the log file.
log.txt

INFO:eval:it: 2499, time: [i/o: 0.004388, processing: 4.636132, all: 4.64052], loss: 6.305851275101304e-05
INFO:eval:it: 2500, time: [i/o: 0.002093, processing: 4.635756, all: 4.637849], loss: 0.0106213837862$1477
INFO:eval:evaluating file /home/sanromag/DATA/WMH/mdgru/data/val/5183c7e2 of shape (176, 256, 256) wi$h w [64, 64, 64] and p [5, 5, 5]
Traceback (most recent call last):
  File "RUN_mdgru.py", line 122, in <module>
    run_mdgru()
  File "RUN_mdgru.py", line 118, in run_mdgru
    return runner.run()
  File "/home/sanromag/WORK/CODE/external/mdgru/mdgru/runner.py", line 423, in run
    self.train()
  File "/home/sanromag/WORK/CODE/external/mdgru/mdgru/runner.py", line 321, in train
    error = self.validation(showIt=self.save_validation_results, name=it)
  File "/home/sanromag/WORK/CODE/external/mdgru/mdgru/runner.py", line 221, in validation
    return_results=True)
File "/home/sanromag/WORK/CODE/external/mdgru/mdgru/eval/__init__.py", line 345, in test_all_available
    certainty[slicesa] *= np.arange(1.0 / (pp + 1), 1, 1.0 / (pp + 1)).reshape(reshapearr)
ValueError: cannot reshape array of size 6 into shape (5,1,1)

INFO🏃epoch 0/0: running for 100000 batches of 1:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: filter must be 4-dimensional: [1,7,7,16,32]
[[Node: train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/ExpandDims, train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/ExpandDims_1)]]
[[Node: train12_errorrepr/split_2/_93 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_14829_train12_errorrepr/split_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "RUN_mdgru.py", line 460, in
Runner(ev, experiments_postfix="_" + pf, **args_runner).run()
File "/home/antal/Programs/tensorflow-mdgru-seminew/runner/init.py", line 346, in run
self.train()
File "/home/antal/Programs/tensorflow-mdgru-seminew/runner/init.py", line 234, in train
loss = self.ev.train(batch_size=self.batch_size)
File "/home/antal/Programs/tensorflow-mdgru-seminew/eval/init.py", line 247, in train
_, loss, summary = self.sess.run(tasks, ph)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: filter must be 4-dimensional: [1,7,7,16,32]
[[Node: train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/ExpandDims, train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/ExpandDims_1)]]
[[Node: train12_errorrepr/split_2/_93 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_14829_train12_errorrepr/split_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/Conv2D', defined at:
File "RUN_mdgru.py", line 453, in
args_eval)
File "/home/antal/Programs/tensorflow-mdgru-seminew/eval/init.py", line 326, in init
super(LargeVolumeEvaluation, self).init(model, collectioninst, kw)
File "/home/antal/Programs/tensorflow-mdgru-seminew/eval/classification.py", line 12, in init
kw)
File "/home/antal/Programs/tensorflow-mdgru-seminew/eval/init.py", line 205, in init
self.model.optimize
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/init.py", line 19, in wrapper
setattr(self, attribute, function(self))
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/mdgru_classification.py", line 82, in optimize
rets = optimizer.minimize(self.cost, global_step=self.global_step)
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/init.py", line 19, in wrapper
setattr(self, attribute, function(self))
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/init.py", line 110, in cost
loss = tf.reduce_mean(self.costs)
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/init.py", line 19, in wrapper
setattr(self, attribute, function(self))
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/mdgru_classification.py", line 69, in costs
loss = tf.nn.softmax_cross_entropy_with_logits(logits=self.logits,
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/init.py", line 19, in wrapper
setattr(self, attribute, function(self))
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/mdgru_classification.py", line 46, in logits
h = self.mdgru_bb(h, self.dropout, mdgruc, fcc, name="{}".format(it + 1), istraining=self.training, **kw)
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/mdrnn/init.py", line 59, in mdgru_bb
mdgru = mdgruclass()
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/mdrnn/mdgru.py", line 125, in call
strides=copy(st))
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/mdrnn/mdgru.py", line 174, in add_cgru
swap_memory=self.swap_memory, initial_state=initial_state)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py", line 627, in dynamic_rnn
dtype=dtype)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py", line 824, in _dynamic_rnn_loop
swap_memory=swap_memory)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3224, in while_loop
result = loop_context.BuildLoop(cond, body, loop_vars, shape_invariants)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2956, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 2893, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py", line 3194, in
body = lambda i, lv: (i + 1, orig_body(*lv))
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py", line 795, in _time_step
(output, new_state) = call_cell()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/rnn.py", line 781, in
call_cell = lambda: cell(input_t, state)
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/crnn/cgru.py", line 81, in call
strides=self.strides, orthogonal_init=False)
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/crnn/init.py", line 166, in _convlinear
is_circular_convolution=self.periodic_convolution_h)
File "/home/antal/Programs/tensorflow-mdgru-seminew/model/crnn/init.py", line 106, in _convolution
return convolution_helper_padding_same(data, convolution_filter, filter_shape, strides)
File "/home/antal/Programs/tensorflow-mdgru-seminew/helper.py", line 99, in convolution_helper_padding_same
return tf.nn.convolution(inp, convolution_filter, "SAME", strides)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 780, in convolution
return op(input, filter)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 868, in call
return self.conv_op(inp, filter)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 520, in call
return self.call(inp, filter)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 204, in call
name=self.name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 193, in _conv1d
name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 497, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/util/deprecation.py", line 497, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/nn_ops.py", line 2464, in conv1d
data_format=data_format)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): filter must be 4-dimensional: [1,7,7,16,32]
[[Node: train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/ExpandDims, train12_errorrepr/1/mdgru/dim1/backward/rnn/while/CGRUCell/Gates/ConvLinear/convolution_1/ExpandDims_1)]]
[[Node: train12_errorrepr/split_2/_93 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_14829_train12_errorrepr/split_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Hello,
thanks for sharing this software. I am running now my train+test. While trying to only run traning according to documentation in
https://zubata88.github.io/mdgru/how_to_use.html#only-train
I stumbled upon a couple of errors:

• options --ignore_nifti_header and --onlytrain are not valid
• when removing these options, then an exception raises because the script is missing --locationtesting option, specifically:
  Exception: either tps or location has to be set

I am finally running the option train+test, that runs without problems, so far.
https://zubata88.github.io/mdgru/how_to_use.html#train-test

Thanks,
G.

for some reason, filter size for state is using the second and filter size for input is using the first filter size argument. fix this and make filter size changeable through RUN_mdgru.py as parameter

Major issue:
Dice-loss options are not passed on to model class in RUN_mdgru.py

Minor issues:
Logging handler are added only in the runner.py script, not allowing logging of debug info in RUN_mdgru.py
Logging creates duplicate log messages.
Functions scipy.misc.imsave and scipy.misc.imread are deprecated in scipy 1.0.0. Suggesting to use imageio.imwrite and imageio.imread instead.

test_each should be named val_each.
Or at least val_each should also exist as a parameter.

Choosing a negative padding value should not result in a negative overlap between the patches during validation/testing.

Hello, would it be possible for you to provide the ckpt for multiple sclerosis lesion segmentation?
Thank you in advance

..., which was copy pasted with the now released proper github library at https://github.com/spezold/mvloader.

Please implement a --testonly setup which is able to load multiple training ckpts.
For each ckpt it should test all relevant files in --locationtesting and save the results under the provided --optionname(s).
For --testonly at least one optionname should be provided. (--modelname=trainingname, --optionname==testingname)
--optionname(s) should have the same length as --ckpt(s).
if --optionname is singular, concatate the ckpt names to the optionname, provided the ckpt names are pairwise different.

Additionally, there should be a parameter whether to save all test results or only to store their scores in a .csv.

Hi,

I would like to reproduce the results from the paper Automated Segmentation of Multiple Sclerosis Lesions using Multi-Dimensional Gated Recurrent Units but I'm not sure how to set the parameters to get optimal results.

Would the following settings work well enough?

python3 RUN_mdgru.py --datapath path/to/samplestructure --locationtraining train_data \
--locationvalidation val_data \
--optionname onlytrainrun --modelname mdgrudef48 -w 64 64 64 -p 5 5 5 \
-f seq1.nii.gz seq2.nii.gz seq3.nii.gz -m lab.nii.gz --iterations 100000 \
--nclasses 2 --num_threads 8 --only_train --rotate 0.2 --scale 0.8 1.2 --deformation 0 2 --deformSigma 1 1 \
--add_e_bn True --resmdgru True --use_dropconnect_on_state True --dice_loss_label 0 1 --dice_loss_weight 0.8 --dice_autoweighted --dice_cc

I am particularly unsure of the --deformation 0 2 --deformSigma 1 1 and --dice_loss_weight 0.8 settings.
The --add_e_bn True --resmdgru True --use_dropconnect_on_state True seem to match the descriptions in the paper.

The README.md says that some changes were done to get the best results, so could I ever reach the same performance without touching the code?

My goal is to benchmark MS lesion segmentation methods (for the French Multiple Sclerosis Registry (OFSEP)), so I would like to know if this method competes with a method like nnUNet. Maybe you have an idea about it?

Thanks a lot for your code and (hopefully) your help 👍

f1 = DSC (dice).
introduce f0.5, f1 and f2.
During training, after testeach (valeach) not only log mean values but also min, median and max.
(--testeach should be --valeach)

ignore nifti header leads to wrong affine matrices, when nifti images are saved to disk that did not already have RL/AP/SI ordering of axes.

experiments/*/*/cache/temp-0.ckpt.number -> temp-number.ckpt

final.ckpt-number -> final-number.ckpt or just final.ckpt but I think final-number.ckpt is still helpfull

When loading a checkpoint file and continue training, the old validation_scores.csv should be copied and appended. Mind the iteration number.

The checkpoint file paths are wrong in validation_scores.csv.
In validation_scores.csv the checkpoint entries are empty after the first validation, and the second validation entries have checkpoint files of the first validation. And so on...
Probably self.checkpointfiles[0] is updated after appending the validation scores.