nnstreamer / nnstreamer-ros

NNStreamer extension plugins for ROS support

NNStreamer

Neural Network Support as Gstreamer Plugins.

NNStreamer is a set of Gstreamer plugins that allow Gstreamer developers to adopt neural network models easily and efficiently and neural network developers to manage neural network pipelines and their filters easily and efficiently.

Architectural Description (WIP)

Toward Among-Device AI from On-Device AI with Stream Pipelines, IEEE/ACM ICSE 2022 SEIP
NNStreamer: Efficient and Agile Development of On-Device AI Systems, IEEE/ACM ICSE 2021 SEIP [media]
NNStreamer: Stream Processing Paradigm for Neural Networks ... [pdf/tech report]
GStreamer Conference 2018, NNStreamer [media] [pdf/slides]
Naver Tech Talk (Korean), 2018 [media] [pdf/slides]
Samsung Developer Conference 2019, NNStreamer (media)
ResearchGate Page of NNStreamer

Official Releases

	Tizen	Ubuntu	Android	Yocto	macOS
	5.5M2 and later	16.04/18.04/20.04/22.04	9/P	Kirkstone
arm		Available	Available	Ready	N/A
arm64		Available			N/A
x64			Ready	Ready	Available
x86		N/A	N/A	Ready	N/A
Publish	Tizen Repo	PPA	Daily build	Layer	Brew Tap
API	C/C# (Official)	C	Java	C	C

• Ready: CI system ensures build-ability and unit-testing. Users may easily build and execute. However, we do not have automated release & deployment system for this instance.
• Available: binary packages are released and deployed automatically and periodically along with CI tests.
• Daily Release
• SDK Support: Tizen Studio (5.5 M2+) / Android Studio (JCenter, "nnstreamer")
• Enabled features of official releases

Objectives

• Provide neural network framework connectivities (e.g., tensorflow, caffe) for gstreamer streams.

  • Efficient Streaming for AI Projects: Apply efficient and flexible stream pipeline to neural networks.
  • Intelligent Media Filters!: Use a neural network model as a media filter / converter.
  • Composite Models!: Multiple neural network models in a single stream pipeline instance.
  • Multi Modal Intelligence!: Multiple sources and stream paths for neural network models.

• Provide easy methods to construct media streams with neural network models using the de-facto-standard media stream framework, GStreamer.

  • Gstreamer users: use neural network models as if they are yet another media filters.
  • Neural network developers: manage media streams easily and efficiently.

Maintainers

• MyungJoo Ham

Committers

• Jijoong Moon
• Geunsik Lim
• Sangjung Woo
• Wook Song
• Jaeyun Jung
• Hyoungjoo Ahn
• Parichay Kapoor
• Dongju Chae
• Gichan Jang
• Yongjoo Ahn
• Jihoon Lee

Components

Note that this project has just started and many of the components are in design phase. In Component Description page, we describe nnstreamer components of the following three categories: data type definitions, gstreamer elements (plugins), and other misc components.

Getting Started

For more details, please access the following manuals.

• For Linux-like systems such as Tizen, Debian, and Ubuntu, press here.
• For macOS systems, press here.
• To build an API library for Android, press here.

Applications

• Edge-AI Examples
• Products with NNStreamer
• NNStreamer example applications: Github / Screenshots

CI Server

• CI service status
• TAOS-CI config files for nnstreamer.

AI Acceleration Hardware Support

Although a framework may accelerate transparently as Tensorflow-GPU does, nnstreamer provides various hardware acceleration subplugins.

• Movidius-X via ncsdk2 subplugin: Released
• Movidius-X via openVINO subplugin: Released
• Edge-TPU via edgetpu subplugin: Released
• ONE runtime via nnfw(an old name of ONE) subplugin: Released
• ARMNN via armnn subplugin: Released
• Verisilicon-Vivante via vivante subplugin: Released
• Qualcomm SNPE via snpe subplugin: Released
• NVidia via TensorRT subplugin: Released
• TRI-x NPUs: Released
• NXP i.MX series: via the vendor
• Others: TVM, TensorFlow, TensorFlow-lite, PyTorch, Caffe2, SNAP, ...

Contributing

Contributions are welcome! Please see our Contributing Guide for more details.

nnstreamer-ros's Issues

[CI Test] 0001

This issue is created for the purpose of CI system test.

Build failure when building unittest_tensor_ros_sink

Issue Description

When building nnstreamer-ros using cmake, link error related with gstreamer-check occurs as below.

How to Reproduce

$ mkdir build
$ cd build
$ cmake ..
$ VERBOSE=1 make
...
[100%] Linking CXX executable unittest_tensor_ros_sink
cd /home/again4you/nnstreamer_work/ref/nnstreamer-ros/build/tests/tensor_ros_sink && /usr/bin/cmake -E cmake_link_script CMakeFiles/unittest_tensor_ros_sink.dir/link.txt --verbose=1
/usr/bin/c++      CMakeFiles/unittest_tensor_ros_sink.dir/unittest_tensor_ros_sink.cpp.o CMakeFiles/unittest_tensor_ros_sink.dir/usr/src/gtest/src/gtest-all.cc.o  -o unittest_tensor_ros_sink -rdynamic -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -lnnstreamer -lgstbase-1.0 -lgstreamer-1.0 -lgobject-2.0 -lglib-2.0 -lpthread -lnnstreamer -lgstbase-1.0 -lpthread 
CMakeFiles/unittest_tensor_ros_sink.dir/unittest_tensor_ros_sink.cpp.o: In function `test_tensor_ros_sink_properties_Test::TestBody()':
/home/again4you/nnstreamer_work/ref/nnstreamer-ros/tests/tensor_ros_sink/unittest_tensor_ros_sink.cpp:35: undefined reference to `gst_harness_new'
/home/again4you/nnstreamer_work/ref/nnstreamer-ros/tests/tensor_ros_sink/unittest_tensor_ros_sink.cpp:92: undefined reference to `gst_harness_teardown'
..

Reason

• The main reason is that gstreamer-check is not linked properly.
  IMOH, one of the possible reason is that PKGS in PKG_CHECK_MODULES are defined twice.

ROS2 Support

We have potential clients who are using ROS2 right now.
(I will have meeting tomorrow and fetch their requirements.)

Discussion

How are we going to provide "standard build infrastructure" for Tizen?

Option 1: Add basic ROS packages to devel:AIC:Tizen:5.0:nnsuite in build.tizen.org.
- Need to consult the admin, ask them if we can add a few ROS packages there?
- Need to create /contrib/ros/*.git in review.tizen.org for such ROS packages.
- Need to minimize the number of ROS packages.

Option 2: Do it in the private build server only 😠
- We can endure this for a while (several months). So if option 1 appears too far away, option 2 is fine.

GBS build failure

Because of unpackaged files, below error occurs when building gbs.

• Link: https://build.tizen.org/package/live_build_log/devel:AIC:Tizen:5.0:nnsuite/nnstreamer-ros/standard/i586

[  119s] Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/abuild/rpmbuild/BUILDROOT/nnstreamer-ros-0.0.1-3.1.i386
[  119s] error: Installed (but unpackaged) file(s) found:
[  119s]    /usr/lib/gstreamer-1.0/libtensor_ros_src.so

[Build] pdebuild fails

pdebuild failes on the master branch.

Build logs are as follows:

make[3]: Entering directory '/build/nnstreamer-ros-2019.2.29/build'
cd /build/nnstreamer-ros-2019.2.29/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /build/nnstreamer-ros-2019.2.29 /build/nnstreamer-ros-2019.2.29/gst/tensor_ros_src /build/nnstreamer-ros-2019.2.29/build /build/nnstreamer-ros-2019.2.29/build/gst/tensor_ros_src /build/nnstreamer-ros-2019.2.29/build/gst/tensor_ros_src/CMakeFiles/tensor_ros_src.dir/DependInfo.cmake --color=
Scanning dependencies of target tensor_ros_src
make[3]: Leaving directory '/build/nnstreamer-ros-2019.2.29/build'
make -f gst/tensor_ros_src/CMakeFiles/tensor_ros_src.dir/build.make gst/tensor_ros_src/CMakeFiles/tensor_ros_src.dir/build
make[3]: Entering directory '/build/nnstreamer-ros-2019.2.29/build'
[ 77%] Building CXX object gst/tensor_ros_src/CMakeFiles/tensor_ros_src.dir/tensor_ros_src.cc.o
cd /build/nnstreamer-ros-2019.2.29/build/gst/tensor_ros_src && /usr/bin/x86_64-linux-gnu-g++   -DNNS_VERSION=\"0.0.3\" -Dtensor_ros_src_EXPORTS -I/build/nnstreamer-ros-2019.2.29/./common/include -I/usr/include/gstreamer-1.0 -I/usr/lib/x86_64-linux-gnu/gstreamer-1.0/include -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/nnstreamer  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2  -fPIC   -Wall -Werror -fPIC -g -std=c++11 -o CMakeFiles/tensor_ros_src.dir/tensor_ros_src.cc.o -c /build/nnstreamer-ros-2019.2.29/gst/tensor_ros_src/tensor_ros_src.cc
In file included from /build/nnstreamer-ros-2019.2.29/gst/tensor_ros_src/tensor_ros_src.h:34:0,
                 from /build/nnstreamer-ros-2019.2.29/gst/tensor_ros_src/tensor_ros_src.cc:38:
/build/nnstreamer-ros-2019.2.29/./common/include/nns_ros_subscriber.h:27:21: fatal error: ros/ros.h: No such file or directory
compilation terminated.
gst/tensor_ros_src/CMakeFiles/tensor_ros_src.dir/build.make:65: recipe for target 'gst/tensor_ros_src/CMakeFiles/tensor_ros_src.dir/tensor_ros_src.cc.o' failed
make[3]: *** [gst/tensor_ros_src/CMakeFiles/tensor_ros_src.dir/tensor_ros_src.cc.o] Error 1
make[3]: Leaving directory '/build/nnstreamer-ros-2019.2.29/build'
CMakeFiles/Makefile2:1857: recipe for target 'gst/tensor_ros_src/CMakeFiles/tensor_ros_src.dir/all' failed
make[2]: *** [gst/tensor_ros_src/CMakeFiles/tensor_ros_src.dir/all] Error 2
make[2]: Leaving directory '/build/nnstreamer-ros-2019.2.29/build'
Makefile:141: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/build/nnstreamer-ros-2019.2.29/build'
dh_auto_build: make -j1 returned exit code 2
	cd /build/nnstreamer-ros-2019.2.29
debian/rules:21: recipe for target 'build' failed
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
I: copying local configuration
E: Failed autobuilding of package
I: unmounting dev/pts filesystem
I: unmounting run/shm filesystem
I: unmounting proc filesystem
I: cleaning the build env 
I: removing directory /var/cache/pbuilder/build/6355 and its subdirectories