Converted mediapipe palm detection coreml model cannot be deployed on IOS app about tflite2tensorflow

[TaoZappar]

Thanks for the reply. I also think it's a bug in coremltool. However, I try to find a way to work around it. My initial thinking is to use upsample2d to replace image.resize method in ResizeBilinear Op. Currently, got the model deployed on iOS, but the size of the final output is incorrect. I will keep tracking and let u know once I sort it out. Katsuya Hyodo ***@***.***> 于 2022年2月16日周三 下午2:34写道：

…

I don't think the number 240 in the error message is directly related to the error. The error occurs in the operation named *model_1/model/add_24/add*. [Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1 [image: image] <https://user-images.githubusercontent.com/33194443/154284738-9a42f361-3645-4248-983c-df8bad275b36.png> The shape of the tensor shown in the error message is obviously wrong. Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] This is because there are no operations in this model that process in 5 dimensions. I also noticed that the new height and new width in resizebilinear layer are all 0. Is that something related? No, not at all. It is correct that *new_height* and *new_width* are set to zero, as per the specification. Since the second input is set to *[24, 24]*, *ResizeBilinear (upsample)* will produce an output of size 24 in height and width. [image: image] <https://user-images.githubusercontent.com/33194443/154285717-5a510b64-e6f6-4bf0-8fcd-240956d6df76.png> The way I see it, it's a bug in CoreML. — Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV7EQ3SUWDMYQY3PDEUKVZLU3OYV7ANCNFSM5ORKIUNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[PINTO0309]

I see. I will add an option to enable CoreML optimization. Thank you for doing some meaningful research! 😸

[PINTO0309]

Commit: ab7d962
Release: https://github.com/PINTO0309/tflite2tensorflow/releases/tag/v1.19.0

[PINTO0309]

Sad to say, I don't own an IOS environment, so I can't test it. I'm not entirely sure if it will work, but could you please check if the CoreML model you downloaded from the following URL works?
model_coreml_float32.mlmodel.zip

docker run -it --rm \
-v `pwd`:/home/user/workdir \
ghcr.io/pinto0309/tflite2tensorflow:latest

wget https://github.com/google/mediapipe/raw/master/mediapipe/modules/palm_detection/palm_detection_full.tflite

tflite2tensorflow \
--model_path palm_detection_full.tflite \
--flatc_path ../flatc \
--schema_path ../schema.fbs \
--output_pb

tflite2tensorflow \
--model_path palm_detection_full.tflite \
--flatc_path ../flatc \
--schema_path ../schema.fbs \
--output_coreml

[TaoZappar]

Thanks for the reply. Just test the model on the IOS, still get the same error.

2022-02-16 12:04:35.315685+0000 MLModelCamera[12740:4737111] [espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Generic error": [Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1
2022-02-16 12:04:35.315728+0000 MLModelCamera[12740:4737111] [coreml] Error computing NN outputs -1
2022-02-16 12:04:35.315755+0000 MLModelCamera[12740:4737111] [coreml] Failure in -executePlan:error:.
failed to perform

I checked the layer 240 of the graph, pretty sure the ResizeBilinear caused this problem, as it has also been mentioned in many onnx->coreml conversion, e.g. :

(onnx/onnx-coreml#474)
(apple/coremltools#1105)

I also noticed that the new height and new width in resizebilinear layer are all 0. Is that something related?

Thanks

[PINTO0309]

I don't think the number 240 in the error message is directly related to the error. The error occurs in the operation named model_1/model/add_24/add.

[Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1

The shape of the tensor shown in the error message is obviously wrong.

Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1]

This is because there are no operations in this model that process in 5 dimensions.

I also noticed that the new height and new width in resizebilinear layer are all 0. Is that something related?

No, not at all. It is correct that new_height and new_width are set to zero, as per the specification. Since the second input is set to [24, 24], ResizeBilinear (upsample) will produce an output of size 24 in height and width.

The way I see it, it's a bug in CoreML.

[TaoZappar]

Hi, Katsuya

I am glad to tell you I find a way to work around. I was inspired by this article https://machinethink.net/blog/coreml-upsampling/ (go to the Cheatsheet section)
Instead of using tf.image.resize, I use tf.compat.v1.image.resize_bilinear. The hacked code I used for coreml specifically is as follows:

        elif op_type == 'RESIZE_BILINEAR':
            input_tensor = tensors[op['inputs'][0]]
            size_detail = interpreter._get_tensor_details(op['inputs'][1])
            size = interpreter.get_tensor(size_detail['index'])
            size_height = size[0]
            size_width = size[1]

            options = op['builtin_options']
            align_corners = options['align_corners']
            half_pixel_centers = options['half_pixel_centers']

            def upsampling2d_bilinear(x, size_height, size_width, align_corners, half_pixel_centers):
                if optimizing_for_edgetpu_flg:
                    return tf.image.resize_bilinear(x, (size_height, size_width))
                else:
                    if optimizing_for_openvino_and_myriad:
                        if half_pixel_centers:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=False,
                                half_pixel_centers=half_pixel_centers
                            )
                        else:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=True,
                                half_pixel_centers=half_pixel_centers
                            )
                    else:
                        y = tf.compat.v1.image.resize_bilinear(x, [size_height, size_width], align_corners=True) # slightly better way, which works on IOS
                        # y = tfv2.keras.layers.UpSampling2D()(x) #   no matching solution in COREML
                        # y = tfv2.image.resize(                  #   no matching solution in COREML
                        #     x,
                        #     [size_height, size_width],
                        #     method='bilinear'
                        # )
                        return y