elxy / shadernn

ShaderNN is a lightweight deep learning inference framework optimized for Convolutional Neural Networks on mobile platforms.

What is ShaderNN?

ShaderNN is a lightweight deep learning inference framework optimized for Convolutional Neural Networks. It provides high-performance inference for deep learning applications in image and graphics process on mobile devices.

Why use ShaderNN?

• Targeted for real time graphic and image post-processing
  • Directly operate the texture data of graphics graphics and image applications, save big I/O time which is critical for real time application on mobile platforms;
  • Native OpenGL ES based, easily integrate with the graphics rendering pipeline to maximize the use of computing resources, suits for rendering, image/video and game AI applications;
• High Performance
  • Makes full use of the parallel computing advantages of GPU Shader to implement core operators;
  • Pre-building the static computation graph for inference first and then running it, compared to the dynamic graph, the graph structure can be optimized before running, such as constant folding, operator fusion, etc., which can obtain faster forward operation speed;
  • When the model is running, the running backend will be selected statically or dynamically according to the platform resources, and the running parameters of the kernel will be dynamically adjusted to achieve the best energy consumption utilization at runtime
  • Optimizes for Convolutional Neural Networks to improve real-time performance;
  • Supports heterogeneous device hybrid computing, and currently supports CPU and GPU;
  • Provides a demo app pipeline optimized for throughput over latency, minimized data transfer and optimized for video processing
• Lightweight & Portability & Extensibility
  • OpenGL-based does not require reliance on other third-party technology libraries and optimized for mobile platforms, making it easy to port, deploy and upgrade;
  • Simple input/output interface, compatible with GPU processing;
• Versatility
  • Supports popular framework formats such as TensorFlow/PyTorch/ONNX;
  • Supports popular Convolutional Neural Networks (CNN), such image classification, object detection, image segmentation, image enhancement;
  • Supports user-defined operators, convenient to implement new operators and models;

Typical Application Scenerios:

• ShaderNN is good at graphics and image processing pipelines, and here listed some typical scenerios, such as ray tracing denoise, deep learning super sampling, high dynamic range, super resolution and style transfer etc.
  

Architecture:

Getting Started:

• Pleaes refer to Getting-Started.md for details.

Model Conversion:

• Support conversion from TensorFlow , PyTorch and ONNX based models. Please refer to ModelConversion.md for details.

Model Zoo/Examples:

• Provide image classification, object detection, image segmentation and image enchancement models for reference. Please refer to ModelZoo.md for details.

Operators:

• Implement basic CNN operators by using fragment shader, computer shader or CPU. For a complete list of operators being supported, please refer to Operators.md for details.

Benchmark:

• Benchmark models based in Model Zoo against TFLite framework. Please refer to Benchmark.md for details.

Style Transfer Demo:

• Style Transfer example running on Android demo app using ShaderNN framework is shown below. The pretrained models are inferenced to showcase styles like Candy, Mosaic, Rain Princess and Udnie.

  

Branching Policy:

• For dev branches for your own use, please prefix it with "your_name/". For example, "bruce.lee/training_session_1"

License:

• Apache License 2.0

Acknowledgement:

ShaderNN makes use of the following third party libraries:

• ncnn
• MNN
• opencv
• libyuv
• libjpeg
• cli11
• cmrc
• eigen
• glad
• glfw
• glm
• picojson
• stb_image
• readerwriterqueue

ShaderNN makes use of models trained and converted from Tensorflow, PyTorch and ONNX, and uses Netron visualizer:

• Tensorflow
• PyTorch
• ONNX
• Netron