This project demonstrates how to process a video stream in real-time using WebAssembly and apply a pre-trained food classification model to each frame of the video in order to determine if food is present in that frame, all by integrating WasmEdge into YoMo serverless.
Open-source projects that we used:
- Serverless stream processing framework YoMo
- Integrate with WasmEdge to introduce WebAssembly, interop TensorFlow Lite model
- A deep learning model found on TensorFlow Hub; make sure to download
TFLite (aiy/vision/classifier/food_V1)
, which was created by Google
Advantages:
- โก๏ธ Low-latency: Streaming data processing applications can now be done in far edge data centers thanks to YoMo's highly efficient network services
- ๐ Security: WasmEdge runs the data processing function in a WebAssembly sandbox for isolation, safety, and hot deployment
- ๐ High Performance: Compared with popular containers, such as Docker, WasmEdge can be up to 100x faster at startup and have a much smaller footprint
- ๐ฏ Edge Devices: As WasmEdge consumes much less resources than Docker, it is now possible to run data processing applications on edge devices
$ git clone https://github.com/yomorun/yomo-wasmedge-tensorflow.git
$ go install github.com/yomorun/cli/yomo@latest
$ yomo version
YoMo CLI version: v0.1.3
Or, you can download the pre-built binary tarball yomo-v0.1.3-x86_64-linux.tar.gz.
Details about YoMo CLI
installation can be found here.
Install WasmEdge with its Tensorflow and image processing extensions
wget -qO- https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash -s -- -e all -p /usr/local
If you have any questions about installation, please refer to the official documentation. Currently, this project works on Linux machines only.
$ sudo apt-get update
$ sudo apt-get install -y ffmpeg libjpeg-dev libpng-dev
Write app.go to integrate WasmEdge-tensorflow
:
Get WasmEdge-go
:
$ cd flow
$ go get -u github.com/second-state/WasmEdge-go/wasmedge
Download pre-trained TensorflowLitee model: rust_mobilenet_food_lib_bg.so, store to flow
directory:
$ wget -P flow 'https://github.com/yomorun/yomo-wasmedge-tensorflow/releases/download/v0.2.0/rust_mobilenet_food_lib_bg.so'
$ yomo serve -c ./zipper/workflow.yaml
$ cd flow
$ go run --tags "tensorflow image" app.go
Download this demo video: hot-dog.mp4, store to source
directory, then run๏ผ
$ wget -P source 'https://github.com/yomorun/yomo-wasmedge-tensorflow/releases/download/v0.2.0/hot-dog.mp4'
$ go run ./source/main.go ./source/hot-dog.mp4