redisai / aibench

AIBench, a tool for comparing and evaluating AI serving solutions. forked from [tsbs](https://github.com/timescale/tsbs) and adapted to AI serving use case

aibench

This repo contains code for benchmarking deep learning solutions, including RedisAI. This code is based on a fork of work initially made public by TSBS at https://github.com/timescale/tsbs.

Current DL solutions supported:

• RedisAI: an AI serving engine for real-time applications built by Redis Labs and Tensorwerk, seamlessly plugged into ​Redis.
• Nvidia Triton Inference Server: An open source inference serving software that lets teams deploy trained AI models from any framework (TensorFlow, TensorRT, PyTorch, ONNX Runtime, or a custom framework), from local storage or Google Cloud Platform or AWS S3 on any GPU- or CPU-based infrastructure.
• TorchServe: built and maintained by Amazon Web Services (AWS) in collaboration with Facebook, TorchServe is available as part of the PyTorch open-source project.
• Tensorflow Serving: a high-performance serving system, wrapping TensorFlow and maintained by Google.
• Common REST API serving: a common DL production grade setup with Gunicorn (a Python WSGI HTTP server) communicating with Flask through a WSGI protocol, and using TensorFlow as the backend.

Current use cases

Currently, aibench supports two use cases:

• creditcard-fraud [details here]: from Kaggle with the extension of reference data. This use-case aims to detect a fraudulent transaction based on anonymized credit card transactions and reference data.

• vision-image-classification[details here]: an image-focused use-case that uses one network “backbone”: MobileNet V1, which can be considered as one of the standards by the AI community. To assess inference performance we’re recurring to COCO 2017 validation dataset (a large-scale object detection, segmentation, and captioning dataset).

Current DL solutions supported per use case:

Use case/Inference Server	model	RedisAI	TensorFlow Serving	Torch Serve	Nvidia Triton	Rest API
Vision Benchmark (CPU/GPU) (details)	mobilenet-v1 (224_224)	✔️	Not supported	Not supported	✔️	Not supported
Fraud Benchmark (CPU) (details)	Non standard Kaggle Model with the extension of reference data	✔️ docs	✔️ docs	✔️ docs	Not supported	✔️ docs

Installation

The easiest way to get and install the go benchmark programs is to use go get and then issuing make:

# Fetch aibench and its dependencies
go get github.com/RedisAI/aibench
cd $GOPATH/src/github.com/RedisAI/aibench

make

Blogs/White-papers that reference this tool

• [1] Announcing RedisAI 1.0: AI Serving Engine for Real-Time Applications, May 19, 2020

aibench's Issues

add rules for quickly testing all variants on CI

• Build
• Vision benchmark
• Fraud benchmark

Add support for nvidia benchmarks

Modify/fork https://github.com/NVIDIA-AI-IOT/jetson_benchmarks and execute on RedisAI for supported framework

add option to use mysql as backend for reference data

Allow to pass different dimension variable order on vision benchmark

Given the different backends dimension order, we should be able to prepare the tensors on HxWxC formatting or CxHxW formatting.

PyTorch benchmarks

Models to be assessed:

• DLRM: https://github.com/facebookresearch/dlrm

ONNX benchmarks

Further references:

• microsoft/onnxruntime#133
• https://github.com/Microsoft/onnxruntime/tree/master/onnxruntime/test/perftest
• https://cloudblogs.microsoft.com/opensource/2020/01/21/microsoft-onnx-open-source-optimizations-transformer-inference-gpu-cpu/

include vision benchmark details on Readme.md

enable running a model solely on transactional data ( no reference data )

Support for tensor batching on vision-benchmark

• RedisAI
• Triton

Add multi-model benchmark

• 10K models
• reference data can be interesting to have

Enable opt-in runtime metrics collections per benchmark runner

Each runner struct that is of inference.Processor type should implement a common CollectRunTimeMetrics() method that will asks the specific runner to fetch (opt-in) runtime stats that will then be stored on the results file.
Return data should include the collection timestamp and an interface with all fetched data.

torchserve support