Cannot convert caffe model with "Reshape" layer about mmdnn

[kitstar]

Hi @andrusza2, could you provide the model files for testing? Thanks.

[andrusza2]

Hi,
Here is an example of network definition with Reshape layer: https://1drv.ms/f/s!AnbrqVfJKXoMjWAjrqRwIfkBsj-o

If you need caffemodel file for testing, just generate it with random weights, e.g.:

import caffe
net = caffe.Net('/path/to/deploy.prototxt', caffe.TEST)
net.save('/path/to/test.caffemodel')

[kitstar]

Hi @andrusza2 . Implemented. Please try the newest code.

Note: when you try to convert IR->DST, please modify the axis parameter of Concat layer for 3 to 1.

For example, in tensorflow model code

shallow_q       = tf.concat([pool1_q2_flat, pool1_q3_flat], 3, name = 'shallow_q')
q_concat        = tf.concat([shallow_q_norm, fc8_q_norm], 3, name = 'q_concat')

to

shallow_q       = tf.concat([pool1_q2_flat, pool1_q3_flat], 1, name = 'shallow_q')
q_concat        = tf.concat([shallow_q_norm, fc8_q_norm], 1, name = 'q_concat')

No good method to figure out the real axis number. Any idea is welcome.

[andrusza2]

Thank you for the new version.
Now conversion is running without errors.

Unfortunately, I think there are more problems related to NCHW vs. NHWC format.

Assume we have seqeuence "Convolution -> Flatten -> Dense" in NCHW Caffe model.

After conversion to TensorFlow, the model is NHWC. Am I wrong or in that case the output of the "Flatten" layer will contain the same values, but in a different order? If so, the output of "Dense" layer is wrong, because the weights were trained on a different order of inputs. Could you check it?

Also LRN (local response normalization) layer in Caffe expects 4-D tensor with channels on the second dim, while TensorFlow expects channels on the last dimension.

[kitstar]

Assume we have seqeuence "Convolution -> Flatten -> Dense" in NCHW Caffe model.

After conversion to TensorFlow, the model is NHWC. Am I wrong or in that case the output of the "Flatten" layer will contain the same values, but in a different order? If so, the output of "Dense" layer is wrong, because the weights were trained on a different order of inputs. Could you check it?

Yes, we are aware of it and set up a simple rule to check if the dense layer is flattened by conv/pool layer. If it is, it will reshape to conv/pool layer shape, transpose, and flatten again.

Also LRN (local response normalization) layer in Caffe expects 4-D tensor with channels on the second dim, while TensorFlow expects channels on the last dimension.

Yes it is. We try to reset the axis of this kind of layers, like concat, LRN.

[andrusza2]

@kitstar - can we expect these patches in the near future?

[kitstar]

Hi @andrusza2, you have to modify some axes in your case manually and it should work.

[xieyufei1993]

Hi @andrusza2 . Implemented. Please try the newest code.

Note: when you try to convert IR->DST, please modify the axis parameter of Concat layer for 3 to 1.

For example, in tensorflow model code

I just want to know I should modify which file ?

[xieyufei1993]

what does axis parameter mean?

[kitstar]

modify

shallow_q       = tf.concat([pool1_q2_flat, pool1_q3_flat], 3, name = 'shallow_q')
q_concat        = tf.concat([shallow_q_norm, fc8_q_norm], 3, name = 'q_concat')

to

shallow_q       = tf.concat([pool1_q2_flat, pool1_q3_flat], 1, name = 'shallow_q')
q_concat        = tf.concat([shallow_q_norm, fc8_q_norm], 1, name = 'q_concat')

[bhadresh74]

@kitstar @andrusza2
Hi, I think this issue is still there.

Here is the prototext file I am trying to convert.

https://github.com/hszhao/PSPNet/blob/master/evaluation/prototxt/pspnet101_cityscapes_713.prototxt

Here is the command I used to convert Caffemodel to IR

mmtoir -f caffe -n path-to-prototxt -w path-to-caffemodel -o caffe_resnet_IR

Here is my stack trace.

Traceback (most recent call last):
File "/data/bdhanani/Experiments_04/Notebooks/cf_to_tf/mmdnn/bin/mmconvert", line 11, in
sys.exit(_main())
File "/data/bdhanani/Experiments_04/Notebooks/cf_to_tf/mmdnn/lib/python2.7/site-packages/mmdnn/conversion/_script/convert.py", line 95, in _main
ret = convertToIR._convert(ir_args)
File "/data/bdhanani/Experiments_04/Notebooks/cf_to_tf/mmdnn/lib/python2.7/site-packages/mmdnn/conversion/_script/convertToIR.py", line 9, in _convert
transformer = CaffeTransformer(args.network, args.weights, "tensorflow", args.inputShape, phase = args.caffePhase)
File "/data/bdhanani/Experiments_04/Notebooks/cf_to_tf/mmdnn/lib/python2.7/site-packages/mmdnn/conversion/caffe/transformer.py", line 318, in init
graph = GraphBuilder(def_path, self.input_shape, self.is_train_proto, phase).build()
File "/data/bdhanani/Experiments_04/Notebooks/cf_to_tf/mmdnn/lib/python2.7/site-packages/mmdnn/conversion/caffe/graph.py", line 450, in build
graph.compute_output_shapes(self.model)
File "/data/bdhanani/Experiments_04/Notebooks/cf_to_tf/mmdnn/lib/python2.7/site-packages/mmdnn/conversion/caffe/graph.py", line 269, in compute_output_shapes
node.output_shape = TensorShape(*NodeKind.compute_output_shape(node))
File "/data/bdhanani/Experiments_04/Notebooks/cf_to_tf/mmdnn/lib/python2.7/site-packages/mmdnn/conversion/caffe/graph.py", line 129, in compute_output_shape
return LAYER_DESCRIPTORSnode.kind
KeyError: None

Regarding the error, my guess is on these two things

1. The prototxt file has a layer name called "BN" which is nothing but a "BatchNorm" layer. If I parse the file with the layer name "BatchNorm", I get this error:

check failed: param_size <= num_param_blobs (4 vs. 3) Too many params specified for layer conv1_1_3x3_s2/bn

To tackle this error, I changed the layer definition and reduced batch norm parameter from 4 to 3 which afterwards lead me to this error:

Check failed: target_blobs.size() == source_layer.blobs_size() (3 vs. 4) Incompatible number of blobs for layer conv1_1_3x3_s2/bn

Which makes sense, since my weights has 4 blobs for the batch norm layer: scale, offset, mean and variance.

2. The prototxt file has a layer name called "Interp" which is custom layer implemented in the Caffe.

I am not sure which of these things from above causing this issue.

I have a small suggestions.

• Can the error be more informative than just "Keyerror":None ? Like, which layer is problematic or the root cause of the error.

Please let me know if anyone had the same issue or my approach is wrong.

Thanks,
Bee