Freezing and optimizing the model about deepgaze

[santo4ul]

I think "ScalarSummary" is the output node. Despite using this, I'm not able to load the frozen/optimized graph using OpenCV DNN module. I get the below error,

terminate called after throwing an instance of 'cv::Exception'
  what():  OpenCV(4.3.0) dnn/src/tensorflow/tf_importer.cpp:664: error: (-215:Assertion failed) const_layers.insert(std::make_pair(name, li)).second in function 'addConstNodes'

Please let me know if I'm missing something.

Updated Freezing/Optimizing code, using "ScalarSummary" as output node and using "tf.float32.as_datatype_enum" for optimize_for_inference()

import tensorflow.compat.v1 as tf
import optimize_for_inference_lib

tf.disable_eager_execution()

#
# Generate .pbtxt file
#
saver = tf.train.import_meta_graph("./cnn_cccdd_30k.meta")
sess = tf.Session()
saver.restore(sess, "./cnn_cccdd_30k.tf")
graph = sess.graph
tf.train.write_graph(sess.graph.as_graph_def(), '.', 'pitch.pbtxt', as_text=True)
print("Saved pbtxt file..\n")

#
# Freeze graph
#

from tensorflow.python.tools import freeze_graph
freeze_graph.freeze_graph('pitch.pbtxt', "", False, 
                          './cnn_cccdd_30k.tf', "ScalarSummary",
                           "save/restore_all", "save/Const:0",
                           'pitch.pb', True, ""  
                         )

print("Generated pitch.pb file..\n")

#
# Optimize graph
#

inputGraph = tf.GraphDef()
with tf.gfile.Open('pitch.pb', "rb") as f:
  data2read = f.read()
  inputGraph.ParseFromString(data2read)
  
outputGraph = optimize_for_inference_lib.optimize_for_inference(
              inputGraph,
              ["Placeholder"], # an array of the input node(s)
              ["ScalarSummary"], # an array of output nodes
              tf.float32.as_datatype_enum)

# Save the optimized graph'test.pb'
f = tf.gfile.FastGFile('pitch_opt.pb', "w")
f.write(outputGraph.SerializeToString()) 

print("Generated Optimized pitch_opt.pb file..\n")

[mpatacchiola]

Hi @santo4ul

I do not know this particular tensorflow utility so I cannot directly suggest you how to use it.
However, if you need the name of the output unit then for sure this is not ScalarSummary, since this one is the summary used by tensorboard to record a scalar value.

The name of the output operation that you pointed out previously add_6 sounds correct to me, since this should be the add bias operation performed in the last layer just before the activation function (tanh).

[santo4ul]

Thankd @mpatacchiola for your inputs.

performed in the last layer just before the activation function (tanh).

add_6 happens before tanh or happens at last after tanh_4? The tensorflow scripts needs the last output node. My understanding is that in the output layer (below code), after tanh_4, add_6 is done. Plese correct me if I'm wrong.

            #Output layer 6
            out = tf.tanh(tf.matmul(dense1, self.hp_out_weights) + self.hp_out_biases)
            if(DEBUG == True): print("SHAPE out: " + str(out.get_shape()))
            return out

[mpatacchiola]

Hi @santo4ul

tanh is the very last operation, and it is placed after the add operation.

However, there are 6 tanh operations, so be sure that tanh_4 is what you need, most probably the very last tanh should be tanh_6.

You can give a look to the code here, and check the order of the variable in the method _allocate_pitch_variables() under def model(data): you will see the specific order of operations performed by the network.

In particular the output for the pitch is in this line of the code. As you can see this is the sixth layer, and there are three operations (in this order): matmul, add, tanh. With tanh being the very last operation before the output is returned.

[santo4ul]

Hi @mpatacchiola, Thanks for all your inputs.

After multiple failed attempts in freezing cnn_cccdd_30k.tf, I directly saved the model from the inference code (head_pose_estimate.py) and successfully froze the model.I'm able to load the model from OpenCV 4.3 successfully. Given below is the code that I use to freeze the model,

saver = tfv1.train.Saver()
saver.save(sess, './pitch_jupyter.ckpt')
tfv1.train.write_graph(sess.graph.as_graph_def(), '.', 'pitch.pbtxt', as_text=True)
from tensorflow.python.tools import freeze_graph
freeze_graph.freeze_graph('pitch.pbtxt', "", False, 
                          './pitch_jupyter.ckpt', "Tanh_4",
                           "save/restore_all", "save/Const:0",
                           'pitch_freeze.pb', True, "")

The final frozen pitch_freeze.pb can be successfully loaded from OpenCV 4.3. I'm still getting some wrong pitch values with OpenCV DNN module. Probably this has to do with the input format and/or dimensions. I'll debug this further.

The problem I see is that, the checkpoint cnn_cccdd_30k.tf, has so many unused nodes (such as dropouts, placeholders, etc) and the freeze_graph is not able to successfully freeze cnn_cccdd_30k.tf.

Hope this helps someone.

Thanks again.

[santo4ul]

Moreover, I see that Tanh_6 is not relevant during the inference. Tanh_4 is the last node and the output node for inference. And Placeholder is the input node.