TF 1.13.2 inference Error : Didn't find op for builtin opcode about tinyneuralnetwork

[peterjc123]

@Ouskit With 55734c7, you can now use quantize_target_type='uint8', hybrid_quantization_from_float=True, hybrid_per_channel=False to get the desired model.

[peterjc123]

@Ouskit Below is the code that sets the op version for CONV_2D.
https://github.com/alibaba/TinyNeuralNetwork/blob/main/tinynn/converter/operators/op_version.py#L35-L49
It seems that you are using hybrid quantization, which is actually supported in TF 1.13.2 with op version = 1 as can be seen using the below links.
https://github.com/tensorflow/tensorflow/blob/v1.13.2/tensorflow/lite/kernels/conv.cc#L644
https://github.com/tensorflow/tensorflow/blob/v1.13.2/tensorflow/lite/kernels/register.cc#L170
So you may update the aforementioned logic to adapt the versioning to your usage.

[Ouskit]

@peterjc123 Thank your reply, I change [op_version.py#L35-L49](

TinyNeuralNetwork/tinynn/converter/operators/op_version.py

Lines 35 to 49 in 6bdac3b

	if op.op.code == ExtendedOperator.CONV_2D:
	if (
	str(op.inputs[0].dtype) == 'int8'
	and str(op.inputs[1].dtype) == 'int8'
	and str(op.outputs[0].dtype) == 'int8'
	):
	op.op.version = 3
	elif (
	str(op.inputs[0].dtype) == 'float32'
	and str(op.inputs[1].dtype) == 'int8'
	and str(op.outputs[0].dtype) == 'float32'
	):
	op.op.version = 2
	else:
	op.op.version = 1

) all op.version to 1 and inference's result successfully without performing dynamic quantization.

But when I want to do dynamic quantization, pass quantize_target_type='int8', hybrid_quantization_from_float=True, hybrid_per_channel=False to convertr, the tflite inference result is totally wrong in tf 1.13.2. But in tf 1.15.5 result is normal and ok.

Is there anything I don't notice? Thank you.

[peterjc123]

Please try quantize_target_type='uint8'.

[Ouskit]

@peterjc123 Thank your instant reply,
I inspected tf 1.13.2 inference result, it is got all 0.
After changing to quantize_target_type='uint8' will show AttributeError: Hybrid kernels supports int8 only

[Ouskit]

@Ouskit I change hybrid_quantization_from_float=False, and infernce result is normal, but the model size is remaining the same. Is it normal? Thank you.

[peterjc123]

@Ouskit I change hybrid_quantization_from_float=False, and infernce result is normal, but the model size is remaining the same. Is it normal? Thank you.

Yes, it means that you don't use hybrid quantization.

[Ouskit]

Please comment out https://github.com/alibaba/TinyNeuralNetwork/blob/main/tinynn/converter/base.py#L96 and change the types in the code in https://github.com/alibaba/TinyNeuralNetwork/blob/main/tinynn/converter/operators/hybrid_quantizer.py#L49 to torch.quint8.

Appreciate your reply,

After I did these changes, the result is from all Nan to very small number, but the model output is not correct

Road Semantic segmentation task output, model output, not correct

Correct result

[peterjc123]

@Ouskit
https://github.com/alibaba/TinyNeuralNetwork/blob/main/tinynn/converter/operators/hybrid_quantizer.py#L47-L53
Please update line 49 in hybrid_quantizer.py to the following code block.

                new_weight = quantize(name, weight, torch.qint8, torch.per_tensor_symmetric, q_type=np.int8)
                new_weight.dtype = np.dtype('uint8')

[Ouskit]

@Ouskit https://github.com/alibaba/TinyNeuralNetwork/blob/main/tinynn/converter/operators/hybrid_quantizer.py#L47-L53 Please update line 49 in hybrid_quantizer.py to the following code block.

                new_weight = quantize(name, weight, torch.qint8, torch.per_tensor_symmetric, q_type=np.int8)
                new_weight.dtype = np.dtype('uint8')

@peterjc123 This works! Now the model output is totally correct. Thank you for your help😊.