simple_bind error about cbst

[yzou2]

I have never met this problem. It seems the problem is caused by the mismatch of label and data. What's your mxnet version?

[Solacex]

I have never met this problem. It seems the problem is caused by the mismatch of label and data. What's your mxnet version?

I am using mxnet 1.3.0, just follow the instructions in the README and haven't changed the code.

[yzou2]

That's weird. Can you post all your environment configuration? Let might help.

It seems the label is reshaped to 1 x 26912. (928/8 * 1856/8 = 26912; 8 is the downsampling rate.) There should be something wrong in the dataloader/loss calculation part.

I just rerun the code on my computer. It is ok.

[Solacex]

`2019-09-26 09:56:43,246 Host start with arguments Namespace(backward_do_mirror=False, base_lr=None, batch_images=None, cache_images=None, check_start=1, check_step=4, crop_size=None, data_root='DATA_ROOT_GTA5', data_root_tgt='/home/guangrli/Data/cityscapes/', dataset='gta', dataset_tgt='cityscapes', debug=False, from_epoch=0, gpus='0', idx_round=0, init_src_port=0.3, init_tgt_port=0.15, kc_policy='cb', kvstore='local', log_file='gta_rna-a1_cls19_s8_ep-0000_train.log', lr_steps=None, lr_type=None, mine_id_address=None, mine_id_number=3, mine_port=0.5, mine_thresh=0.001, model='cityscapes_rna-a1_cls19_s8', no_mxnet_cudnn_autotune_default=True, origin_size=2048, origin_size_tgt=2048, output='gta2city/cbst-sp', phase='val', prefetch_threads=1, prefetcher='thread', scale_rate_range='0.7,1.3', seed_int=0, split='train', split_tgt='train', stop_epoch=None, test_flipping=True, test_scales='1850', test_steps=1, to_epoch=None, weight_decay=0.0005, weights='../Data/gta_rna-a1_cls19_s8_ep-0000.params', with_prior='True')
2019-09-26 09:56:43,247 Host and model specs {'feat_stride': 8, 'dataset': 'gta', 'classes': 19, 'net_type': 'rna', 'net_name': 'a1', 'lr_params': {'base': 0.1, 'type': 'fixed', 'args': None}, 'dataset_tgt': 'cityscapes'}
Traceback (most recent call last):
File "issegm/solve_ST.py", line 1007, in
Level 0
[(64L, 3L, 224L, 224L)]
Level 1
[(64L, 64L, 224L, 224L)]
Level 2
First block on level 2, stride: 2, dilate: 1
[(64L, 128L, 112L, 112L)]
Level 3
First block on level 3, stride: 2, dilate: 1
[(64L, 256L, 56L, 56L)]
Level 4
First block on level 4, stride: 2, dilate: 1
[(64L, 512L, 28L, 28L)]
Level 5
First block on level 5, stride: 2, dilate: 1
[(64L, 1024L, 28L, 28L)]
Level 6
First block on level 6, stride: 2, dilate: 2
Level 7
[(64L, 4096L, 28L, 28L)]
Scores
[(64L, 19L, 28L, 28L)]
_val_impl(args, model_specs, logger)
File "issegm/solve_ST.py", line 806, in _val_impl
mod.bind(dataiter_tgt.provide_data, dataiter_tgt.provide_label, for_training=False, force_rebind=True)
File "/home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/module/module.py", line 429, in bind
state_names=self._state_names)
File "/home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/module/executor_group.py", line 279, in init
self.bind_exec(data_shapes, label_shapes, shared_group)
File "/home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/module/executor_group.py", line 375, in bind_exec
shared_group))
File "/home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/module/executor_group.py", line 662, in _bind_ith_exec
shared_buffer=shared_data_arrays, **input_shapes)
File "/home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/symbol/symbol.py", line 1528, in simple_bind
raise RuntimeError(error_msg)
RuntimeError: simple_bind error. Arguments:
data: (1, 3L, 928L, 1856L)
softmax_label: (1, 26912L)
[09:56:45] src/storage/storage.cc:137: Compile with USE_CUDA=1 to enable GPU usage

Stack trace returned 10 entries:
[bt] (0) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x1e4d7a) [0x2b261db39d7a]
[bt] (1) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x1e53e1) [0x2b261db3a3e1]
[bt] (2) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2f70b05) [0x2b26208c5b05]
[bt] (3) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2f74adf) [0x2b26208c9adf]
[bt] (4) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2f769ff) [0x2b26208cb9ff]
[bt] (5) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2910c53) [0x2b2620265c53]
[bt] (6) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2910dcd) [0x2b2620265dcd]
[bt] (7) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x291e8df) [0x2b26202738df]
[bt] (8) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2925280) [0x2b262027a280]
[bt] (9) /home/guangrli/anaconda/envs/py2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x293385e) [0x2b262028885e]

Traceback (most recent call last):
File "issegm/solve_AO.py", line 168, in
main()
File "issegm/solve_AO.py", line 133, in main
for path in execute(cmd_val):
File "issegm/solve_AO.py", line 165, in execute
raise subprocess.CalledProcessError(return_code, cmd)
subprocess.CalledProcessError: Command '['python', 'issegm/solve_ST.py', '--dataset', 'gta', '--dataset-tgt', 'cityscapes', '--data-root', 'DATA_ROOT_GTA5', '--data-root-tgt', '/home/guangrli/Data/cityscapes/', '--split-tgt', 'train', '--output', 'gta2city/cbst-sp', '--model', 'cityscapes_rna-a1_cls19_s8', '--phase', 'val', '--weights', '../Data/gta_rna-a1_cls19_s8_ep-0000.params', '--mine-id-number', '3', '--test-scales', '1850', '--gpus', '0', '--init-tgt-port', '0.15', '--idx-round', '0', '--no-cudnn', '--test-flipping', '--with-prior', 'True', '--kc-policy', 'cb']' returned non-zero exit status 1`

As in solver_ST.py:

CBST/issegm/solve_ST.py

Line 802 in a28d842

input_label = 255 * np.ones((1, label_h * label_w), np.single)

The label is reshaped to (1, 26912).

[yzou2]

What's your hardware configurations?

[Solacex]

CUDA 10.0,
GPU: NVIDIA quadro P5000,
Linux: Red Hat 7.5

[yzou2]

The code runs well in my desktop. Did you compile mxnet with cuda?

[yzou2]

My configuration is:

Ubuntu 16.04,
Titan Xp/X/ GTX 1080
CUDA 8.0

[Solacex]

Thanks for your prompt reply, I'll try to check it.

[Solacex]

Could you show me print of [ dataiter_tgt.provide_data] and [dataiter_tgt.provide_label] in this line? I want to check if the shape is correct.

CBST/issegm/solve_ST.py

Line 806 in a28d842

mod.bind(dataiter_tgt.provide_data, dataiter_tgt.provide_label, for_training=False, force_rebind=True)

Plus, I have noticed the difference in version of CUDA, I will try to check it.

[yzou2]

Your shape is correct