chili-epfl / qml-cvcamera

QML plugin wrapper for fast OpenCV camera access on desktop & Android

qml-cvcamera

CVCamera is a QML wrapper for fast camera access using OpenCV. It works on desktop and Android.

The following is required for CVCamera to work:

• Ubuntu 14.04
• Qt 5.3.2 (also tested on 5.3.1)
• OpenCV 3.0.0-alpha(master build should also just work)

QML API

The QML API consists of the following:

• device : int - ID of the camera that will be opened, usually goes like 0, 1, ...
• size : QSize - Size of the camera image
• deviceList : QStringList - Names of the detected cameras, indices corresponding to IDs
• cvImage : QVariant - Contains the camera image as a cv::Mat

The full API (API above + non-user parts) can be seen in cvcamera.types.

CVCamera can be used in one or both of two ways:

• Set a CVCamera object as the source property of a VideoOutput in order to display in on the screen
• Set a CVCamera object's cvImage property as some property of a QML object to export the camera image as a cv::Mat. The format of this image is usually RGB in desktop and YUV420sp (a.k.a YUV NV21) on Android.

Desktop Build

1. Install libv4l-dev.

2. Build and install OpenCV:

cd platforms
mkdir build-desktop
cd build-desktop
cmake ../.. -DCMAKE_INSTALL_PREFIX=/usr

At this point, OpenCV components other than calib3d, core, features2d, flann, highgui, imgcodecs, imgproc, video and videoio are not needed and can be disabled. Note: I had to disable WITH_GSTREAMER manually for it to compile without errors. It is not needed and can be disabled. These options can be accessed with ccmake ../...

make -j 5
make install

3. Build and install CVCamera:

mkdir build-desktop
cd build-desktop
/qt/install/root/5.3/gcc_64/bin/qmake ..
make -j 5
make install

Now the CVCamera QML plugin is installed alongside Qt's QML plugins and can be used similar to any other plugin.

Android Build

In addition to the OS, Qt and OpenCV requirements, you need:

• Android SDK and Android API 14 (Android version 4.0)

• Android NDK r9d

These instructions assume armv7-a target architecture. For other architectures, adapt the instructions to your liking.

1. Export a standalone NDK toolchain:

cd /path-to-android-ndk-r9d
./build/tools/make-standalone-toolchain.sh \
    --platform=android-14 \
    --install-dir=/desired/path/to/android/standalone/toolchain \
    --toolchain=arm-linux-androideabi-4.8

Be aware that if you don't have write access to /desired/path/to/android/standalone/toolchain, the script fails silently.

2. Set up the following environment variables:

export ANDROID_HOME=/path/to/android/sdk/linux
export ANDROID_SDK_ROOT=$ANDROID_HOME
export ANDROID_SDK=$ANDROID_SDK_ROOT
export ANDROID_NDK_ROOT=/path-to-android-ndk-r9d
export ANDROID_NDK=$ANDROID_NDK_ROOT
export ANDROID_NDK_STANDALONE_TOOLCHAIN=/path/to/android/standalone/toolchain
export ANDROID_STANDALONE_TOOLCHAIN=$ANDROID_NDK_STANDALONE_TOOLCHAIN
export ANDROID_ABI=armeabi-v7a
export ANDROID_NATIVE_API_LEVEL=14
export ANDROID_TOOLCHAIN_NAME=arm-linux-androideabi-4.8

3. Build and install OpenCV:

cd platforms
mkdir build-android
cd build-android
cmake ../.. \
    -DCMAKE_TOOLCHAIN_FILE=../android/android.toolchain.cmake \
    -DCMAKE_INSTALL_PREFIX=$ANDROID_STANDALONE_TOOLCHAIN/sysroot/usr/share/opencv/

At this point, enable BUILD_SHARED_LIBS.

OpenCV components other than calib3d, core, features2d, flann, highgui, imgcodecs, imgproc, video and videoio are not needed and can be disabled. Leave these options untouched if you are unsure what to do.

The options can be accessed with ccmake ../.. after running cmake.

make -j 5
make install

4. Build and install CVCamera:

mkdir build-android
cd build-android
/qt/install/root/5.3/android_armv7/bin/qmake ..
make -j 5
make install

Now the CVCamera QML plugin is installed alongside Qt's QML plugins and can be used similar to any other plugin.

Note that you still need to bundle the required OpenCV libraries with ANDROID_EXTRA_LIBS and give camera access to your app with ANDROID_PERMISSIONS and ANDROID_FEATURES in the project that uses CVCamera. See samples/preview/preview.pro for an example.

Running Samples

See samples/README.md.

qml-cvcamera's Issues

OpenCV prebuilt `libnative_camera.so`s don't have brightness API

Add the API and build from source so that we have prebuilts that do have this feature. This will enable us to disable auto-exposure.

CameraActivity lacks functions implementation

There's no implementation for the member functions of class CameraActivity and there's no CameraActivity.cpp ?

IOS support

_Add support for IOS_

App window sometimes doesn't launch

On Android, app window sometimes launches into a black screen, long-pressing the menu button to bring up the app list seems to solve the issue.

On desktop, it seems to happen randomly and the window doesn't launch at all. It seems to happen less frequently in debug mode. Debug trace shows it to be waiting on a condition that is outside the Chilitags source.

undefined reference to 'cv::String::allocate(unsigned long)'

cvcamera.o: In function cv::String::String(char const*)': /usr/local/include/opencv2/core/cvstd.hpp:602: undefined reference to cv::String::allocate(unsigned long)'
Makefile:268: recipe for target 'VideoTol' failed
cvcamera.o: In function cv::String::~String()': /usr/local/include/opencv2/core/cvstd.hpp:648: undefined reference to cv::String::deallocate()'
cvcamera.o: In function cv::Mat::Mat(int, int, int, void*, unsigned long)': /usr/local/include/opencv2/core/mat.inl.hpp:500: undefined reference to cv::error(int, cv::String const&, char const*, char const*, int)'
/usr/local/include/opencv2/core/mat.inl.hpp:516: undefined reference to `cv::error(int, cv::String const&, char const*, char const*, int)'
collect2: error: ld returned 1 exit status

Preview sample segfaults on Android 4.4.4

preview sample segfaults on Android 4.4.4 inside libc. Debug trace shows three ??? calls.

Almost certainly the fault of libnative_camera_r4.4.0.so, and there isn't any more recent version yet.

Almost certainly will help to test with more recent NDKs.

Android native camera library is not chosen automatically

OpenCV relies on OpenCV Manager to get the correct libnative_camera_r*.so in place so that it can be loaded with the libnative_camera_r prefix without paying attention to the version. This doesn't allow a scenario where OpenCV Manager is not used and all of libnative_camera_r*.so are bundled with the APK for cross-version compatibility. A rewrite of https://github.com/Itseez/opencv/blob/master/modules/androidcamera/src/camera_activity.cpp#L167 is needed. This is best submitted as a PR to OpenCV itself. If this PR is rejected, https://github.com/Itseez/opencv/blob/master/modules/androidcamera/src/camera_activity.cpp will be forked to CVCamera.

For now, bundling the correct version for your device is the only solution.

Segmentation Fault

Hey,
I just compiled your plugin and opencv 3.0.0 alpha. Sadly when i try to start a program using your plugin - e.g. your samples - it immediatelly crashes with "Segmentationfault".
Now i wondere wether the problem occurs in your plugin or the opencv part.
Do you know how i can check that out?
Kind Regards

System: Raspberry Pi using jessie up to date freshly installed

€: I just checked the make output file of opencv and it has several warnings about array-using out of it boundaries. in:
modules/imgproc/src/contours.cpp
modules/imgproc/test/test_convull.cpp