Whisper tflite doesn't work about armnn

[federicoparra]

@Colm-in-Arm ? @catcor01 ? any ideas?

[Colm-in-Arm]

Hello Federico,

Sorry for the delay. We were busy getting 24.05 delivered.

I'm surprised you're not getting more detailed error messages. When I called ModifyGraphWithDelegate with that model I got a "kTfLiteApplicationError." result. This doesn't come from Arm NN but from the TfLite runtime itself. TfLite documentation says:

kTfLiteApplicationError : Delegation failed to be applied due to the incompatibility with the TfLite runtime, e.g., the model graph is already immutable when applying the delegate. However, the interpreter could still be invoked.

Examining the model in Netron shows a pretty monstrous WHILE operator at the end. Arm NN does not support conditionals and TfLite insists that an entire conditional subgraph must be delegated otherwise it will not allow delegation.

In summary this model cannot be run on Arm NN.

Colm.

[federicoparra]

With all due respect @Colm-in-Arm , that's not acceptable :(

We are talking about Whisper!
You feel comfortable saying that maybe one of the most important models around can't be supported by what you claim is supposed to be the fastest/best supported way to run neural networks on ARM?

It has trouble with LLMs, it can't do Whisper, so what can it do? Object détection? Is that what this library is meant to do?

In the meantime, MLC-LLM with it's compilation techniques achieves quite fast generative inference on the Mali GPU.

I'm sorry but like me, many will discard ARMNN when they realize it doesn't support any of the modern networks we need to run these days :(

[MatthewARM]

I did a bit of digging, it seems the kTfLiteApplicationError is being raised at subgraph.cc:2255 in Tensorflow Lite, which is checking if the delegate supports dynamically shaped tensors, so the error is the same as the warning that is also raised "WARNING: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors (tensor#1936 is a dynamic-sized tensor)."

For what it's worth, it looks like the built-in XNNPACK delegate also hits that path when Lite tries to apply it by default, and then at interpreter.cc:381 in Tensorflow Lite that error is ignored and inference proceeds using the default kernels without delegation.

I haven't checked what happens with the built-in Lite GPU delegate.

So this isn't really an Arm NN issue as such, this model is hitting the cutting-edge of what Tensorflow Lite can do in the default path and Tensorflow Lite doesn't seem ready to handle delegation of this model yet.