yas-sim / openvino-ep-enabled-onnxruntime

Describing How to Enable OpenVINO Execution Provider for ONNX Runtime

OpenVINO Execution Provider Enabled onnxruntime

1. Description

ONNX runtime is a deep learning inferencing library developed and maintained by Microsoft.
'microsoft/onnxruntime' on GitHub

ONNX runtime can load the ONNX format DL models and run it on a wide variety of systems. It supports multiple processors, OSes, and programming languages.
Also, ONNX runtime supports multiple execution providers(EP) which is the backend inferencing engine library. ONNX runtime supports Intel OpenVINO toolkit, Intel DNNL, Intel nGraph, nVIDIA TensorRT, DirectML, ARM compute library, Android neural networks API, and many more EPs.

However, Intel OpenVINO EP is not enabled in the prebuilt binary distribution of ONNX runtime (v1.3.0).

In this project, I built the ONNX runtime from the source code and enabled the OpenVINO execution provider.
The project includes the steps to build and install ONNX runtime and a simple sample code to try ONNX runtime.

You can load the ONNX DL model directly in your program and run it as fast and efficient as OpenVINO with this OpenVINO EP.
Also, OpenVINO EP supports various Intel processors and accelerators such as CPU_FP32, GPU_FP32, GPU_FP16, MYRIAD_FP16, VAD-M_FP16, and VAD-F_FP16 (VAD = Intel Vision Accelerator Design = HDDL).
The performance is OpenVINO >> ONNX runtime (C++) > ONNX runtime (Python) in general. The performance difference between OpenVINO and ONNX runtime is around 2ms to 10ms based on my test result (HW: Core i7-6770HQ).

The sample code are simple CNN image classification programs and the DL model is ResNet-50.

---

ONNX runtimeはマイクロソフトが開発、保守しているディープラーニング推論ライブラリです。
'microsoft/onnxruntime' on GitHub

ONNX runtimeはサポートする数多くのシステム上でONNXフォーマットのDLモデルをロードし実行することが可能です。多くのプロセッサ、OS、プログラミング言語をサポートします。
またONNX runtimeは複数のExecution provider(EP)と呼ばれるバックエンド推論ライブラリをサポートします。Intel OpenVINO toolkit, Intel DNNL, Intel nGraph, nVIDIA TensorRT, DirectML, ARM compute library, Android neural networks APIなど数多くのEPをサポートしています。

しかしながら提供されているビルド済みバイナリパッケージではOpenVINO EPがイネーブルされていません (v1.3.0)。

そこで、このプロジェクトではONNX runtimeをソースコードからビルドし、OpenVINO EPをイネーブルしてみました。

このOpenVINO EPにより、(MOでIRに変換することなく)プログラム内で直接ONNX DLモデルを読み込み、OpenVINOのように高速に効率よく推論を行うことが可能になります。
また、OpenVINO EPを使用することにより各種インテルプロセッサやアクセラレータも利用可能となります。(CPU_FP32, GPU_FP32, GPU_FP16, MYRIAD_FP16, VAD-M_FP16, and VAD-F_FP16 (VAD = Intel Vision Accelerator Design = HDDL))
パフォーマンスはおおむねOpenVINO >> ONNX runtime (C++) > ONNX runtime (Python)の順になるようです。私のテスト結果によると、パフォーマンスの差は2ms~10ms程度でした。(HW:Core i7-6770HQ)

サンプルプログラムは簡単な画像分類(CNN)のプログラムでResnet-50を使用しています。

2. Prerequisites

• OpenVINO 2020.2
  • ONNX runtime v1.3.0 is compatible with OpenVINO 2020.2.
  • If you haven't installed it, go to the OpenVINO web page and follow the Get Started guide to do it.

3. Build and Install ONNX runtime

Note: You should have either one of onnxruntime or onnxruntime-openvino. If you have installed onnxruntime beforehand, please uninstall it before you install onnxruntime-openvino.

Linux

# Build onnxruntime
git clone https://github.com/microsoft/onnxruntime
git checkout v1.3.0
cd onnxruntime
./build.sh --config Release --build_shared_lib --build_wheel --enable_pybind --use_openvino CPU_FP32 --skip_tests

#Install Python module (optional)
pip3 install ./build/Linux/Release/dist/onnxruntime_openvino-1.3.0-cp36-cp36m-linux_x86_64.whl

Windows

# Build onnxruntime
git clone https://github.com/microsoft/onnxruntime
git checkout v1.3.0
cd onnxruntime
build.amd64.1411.bat --config Release --cmake_generator "Visual Studio 16 2019" --build_shared_lib --build_wheel --enable_pybind --use_openvino CPU_FP32 --skip_tests

#Install Python module (optional)
pip install .\build\Windows\Release\Release\dist\onnxruntime_openvino-1.3.0-cp36-cp36m-win_amd64.whl

4. Build and Run the sample applications (C++ and Python)

Linux (C++)

# Download Resnet-18 model and class label text
wget https://github.com/onnx/models/raw/master/vision/classification/resnet/model/resnet18-v2-7.onnx
wget https://raw.githubusercontent.com/HoldenCaulfieldRye/caffe/master/data/ilsvrc12/synset_words.txt

#Build C++ sample app
mkdir -p build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make
cd ..

#Run
cp $INTEL_OPENVINO_DIR/deployment_tools/demo/car.png .
cp build/onnxtest .
cp ../onnxruntime/build/Linux/Release/libonnxruntime.so .
cp ../onnxruntime/build/Linux/Release/libcustom_op_library.so .
./onnxtest

Windows (C++)

# Download Resnet-18 model and class label text
bitsadmin /transfer download https://github.com/onnx/models/raw/master/vision/classification/resnet/model/resnet18-v2-7.onnx %CD%\resnet18-v2-7.onnx
bitsadmin /transfer download https://raw.githubusercontent.com/HoldenCaulfieldRye/caffe/master/data/ilsvrc12/synset_words.txt %CD%\synset_words.txt

#Build C++ sample app
mkdir build
cd build
cmake -G "Visual Studio 16 2019" -DCMAKE_BUILD_TYPE=Release ..
msbuild onnxtest.sln /p:Configuration=Release
cd ..

#Run
copy "%INTEL_OPENVINO_DIR%\deployment_tools\demo\car.png" .
copy build\Release\onnxtest.exe .
copy ..\onnxruntime\build\Windows\Release\Release\onnxruntime.dll .
copy ..\onnxruntime\build\Windows\Release\Release\custom_op_library.dll .
onnxtest.exe

Linux (Python)

Command options:
-d: OpenVINO device name. Must be one of CPU_FP32, GPU_FP16, GPU_FP32, MYRIAD_FP16, VAD-M_FP16, VAD-F_FP16. (default: CPU_FP32)
-m: ONNX model file name (default: resnet18-v2-7.onnx)

cp $INTEL_OPENVINO_DIR/deployment_tools/demo/car.png .
python3 onnxruntime-test.py [-d device_name] [-m model_name]

Windows (Python)

Command options:
-d: OpenVINO device name. Must be one of CPU_FP32, GPU_FP16, GPU_FP32, MYRIAD_FP16, VAD-M_FP16, VAD-F_FP16. (default: CPU_FP32)
-m: ONNX model file name (default: resnet18-v2-7.onnx)

cp $INTEL_OPENVINO_DIR/deployment_tools/demo/car.png .
python3 onnxruntime-test.py [-d device_name] [-m model_name]

5. Test Environment

• Ubuntu 18.04 / Windows 10 1909
• OpenVINO 2020.2
• ONNX runtime 1.3.0

See Also

• Using Open Model Zoo demos
• Model Optimizer
• Model Downloader
• ONNX runtime

openvino-ep-enabled-onnxruntime's Issues

ImportError: cannot import name 'set_seed'

Environment

• Ubuntu 18.04 x86_64
• onnxruntime 1.3.0
• OpenVINO 2020.2
• Python 3.6
• cmake 3.17.3

Problem that arose

Use a pre-built .whl installer or a self-built and generated .whl installer Then, the following error message is displayed and onnxruntime cannot be imported It was not. This may be an issue that is directly unrelated to the purpose of this repository, which is to enable OpenVINO EP, but if you can give me some information, I would appreciate it.

$ python3 onnxruntime-test.py
Traceback (most recent call last):
  File "onnxruntime-test.py", line 5, in <module>
    import onnxruntime
  File "/usr/local/lib/python3.6/dist-packages/onnxruntime/__init__.py", line 12, in <module>
    from onnxruntime.capi._pybind_state import get_all_providers, get_available_providers, get_device, set_seed, RunOptions, SessionOptions, set_default_logger_severity, NodeArg, ModelMetadata, GraphOptimizationLevel, ExecutionMode, OrtDevice, SessionIOBinding
ImportError: cannot import name 'set_seed'

If anyone needs a prebuilt version 1.7.1 for python 3.7 and 3.8 (linux_x86_64)

Each build took me about 3 hours, and I wasted a few more hours dealing with glibc/cmake version compatibility issues and rebuilding again, so to save you some time, you can get the files here.
3.7_ubuntu18 version is compatible with python:3.7.10-slim docker image which is based on debian buster.
3.7_ubuntu18 was built in openvino/ubuntu18_runtime:2021.2 container and was successfully tested with a test from this repo.
3.8_ubuntu20 and 3.7_ubuntu20 were built in openvino/ubuntu20_runtime:2021.2 container and weren't tested.
Special thanks to the creator of this repo for build instructions/flags. Each version was built with the flags mentioned in readme file.

Edit: for some reason I get worse results, in comparison to onnxruntime from pip: 2s vs 0.4s. Maybe that's because I'm testing on a VM that might not even be running on an Intel CPU (even though, /proc/cpuinfo gives me vendor_id: GenuineIntel). Anyways, I'd be glad if anyone with a real hardware tested the build to make sure that's not an issue.
Also I tried to run prebuilt version 3.6 from this repo, in 3.6.13-slim container and I got an error Cannot load onnxruntime.capi. Error: 'libinference_engine.so: cannot open shared object file: No such file or directory', and this file is not present in the tgz archive.

build onnxruntime with openvino do not generate *.whl file ...

i build the onnxruntime with openvino ,the command is "sudo ./build.sh --config Release --build_shared_lib --build_wheel --enable_pybind --use_openvino CPU_FP32 --skip_tests" .after buiding ,in the end it does not generate any *.whl file in any location.
who can tell me why???