Collection of scripts for inference of a SSD Detection Model trained with Tensorflow Object Detection API. Starting from a SSD-like object detection model, like MobileNetV3-SSD, we have different options to do inference. I collected here three of them, benchmarking on a general purpouse CPU and a Raspberry Pi4. All the scripts expect a folder with test images and will iterate over the images, pressing any key in the prediction window. The scripts for inference are named predict<..>, be sure to edit them before run with the required fields (take a look at the code and you'll notice). Part of this code comes from the official repository of Tensorflow.
- Model checkpoints and configuration file (pipeline.config) from the training. You can use a model from the Tensorflow Object Detection API Model zoo to test this code.
- Tensorflow 1.15
- Opencv > 4.3
- Tensorflow Object Detection API
- Numpy
- Pillow
- Pandas
Starting point is to have checkpoints from a training job, we'll have 3 files like this: model.ckpt.data-00000-of-00001, model.ckpt.index, model.ckpt.meta. First we need to freeze the model for inference:
python3 export_inference_graph.py --pipeline_config_path=pipeline.config --trained_checkpoint_prefix model.ckpt --output_directory=. --add_postprocessing_op=true
This will generate a frozen_inference_graph.pb which can be used for inference:
python3 predict_tensorflow.py
We can use the frozen_inference_graph already generated, together with a text-based representation of the model as described here. First, let's generate the text representation:
python3 tf_text_graph_ssd.py --input frozen_inference_graph.pb --config pipeline.config --output graph.pbtxt
We can now run:
python3 predict_opencv.py
We use now export_tflite_ssd_graph.py script to take this original TF graph, and convert it into an 'intermediate' form that removes all the control-flow logic, and leaves the graph with raw outputs that can be directly fed to a custom TFLite op (thats specifically designed for NMS). Then, you use tflite_convert to convert this intermediate representation into the final TFLite graph.
python3 export_tflite_ssd_graph.py --pipeline_config_path=pipeline.config --trained_checkpoint_prefix model.ckpt --output_directory=. --add_postprocessing_op=true
python3 tflite_converter.py
Now we have the final form of the Tensorflow Lite model tflite_quant_graph.tflite
, ready for inference on device.
python3 predict_tflite.py --model tflite_quant_graph.tflite --image <PATH WITH IMAGES>
The main goal of this repo was to have a rough idea of the performances of different methods for inference on device. As you can see from the table, OpenCV outperforms the other methods (at least with this kind of models) with the advantage of a very easy to use API for detection models. On the other hand, OpenCV uses an independent engine and not everything that comes with tensorflow is yet supported, you have to check with the model you want to use. Approx average inference time in milliseconds is reported. Model used is MobileNetv3-SSD from TFODAPI Model Zoo, retrained for the only class "person", on COCO 2017 dataset (selecting images with class person). For this test I took 5 random images from COCO 2017 containing class person. CPU used for this test is Intel i7-5500U CPU @ 2.40GHz.
Inference | Raspberry Pi4 | Desktop CPU |
---|---|---|
Tensorflow | N/A | 600 |
OpenCV DNN Module | 95 | 26 |
Tensorflow Lite | 100 | 30 |