Goal: SDK size <= 1 MB and combined resources (object detection + regression models) <= 4 MB.
- Hunter package management and CMake build system by Ruslan Baratov, as well as much of the cross platform Qt work: "Organized Freedom!" :)
- A C++ and OpenGL ES 2.0 implementation of Fast Feature Pyramids for Object Detection (see Piotr's Matlab Toolbox) for face and eye detection; UPDATE: The acf code has been moved to a separate repository with a hunter package HERE
- Iris ellipse fitting via Cascaded Pose Regression (Piotr Dollar, et al) + XGBoost regression (Tianqi Chen, et al)
- Face landmarks and eye contours provided by "One Millisecond Face Alignment with an Ensemble of Regression Trees" (Kazemi, et al) using a modified implementation from Dlib (Davis King) (normalized pixel differences, line indexed features, PCA size reductions)
- OpenGL ES friendly GPGPU shader processing and efficient iOS + Android texture handling using a modified version of ogles_gpgpu (Markus Kondrad) with a number of shader implementations taken directly from GPUImage (Brad Larson)
iPhone @ 30 FPS (VIDEO) |
---|
FEATURE | SPECIFICATION |
---|---|
eyelids | 2D points 0-15 |
crease | 2D points 16-24 |
iris center | 2D point 25 |
outer limbus | limbus intersection with ray from outer corner to iris center |
inner limbus | limbus intersection with ray from inner corner to iris center |
iris ellipse | 2D center, minor axis, major axis, angle (radians) |
pupil ellipse | 2D center, minor axis, major axis, angle (radians) |
- the left eye is obtained by Y axis mirroring
- total (27*2)+(2*5) = 64 parameters
- the eye crease is useful for pose indexing, but better guidelines are needed
- the 2D limbus points are slightly redundant (given the ellipse iris model) but the intersection points are stable with respect to squinting and provide an efficient anchor for posed indexed features (accurate point-to-ellipse distances are non-trivial and are fairly computationally intensive)
- currently 2D only (gaze angle ground truth would be beneficial)
Drishti is a CMake based project
that uses the Hunter package
manager to download and build project dependencies from source as
needed. Hunter contains detailed
documentation, but a few high
level notes and documentation links are provided here to help orient
first time users. In practice, some working knowledge of CMake may also
be required. Hunter itself is written in CMake, and is installed as part
of the build process from a single HunterGate()
macro at the top of
the root CMakeLists.txt
file (typically
cmake/Hunter/HunterGate.cmake
) (you don't have to build or install
it). Each CMake dependency's find_package(FOO)
call that is paired
with a hunter_add_package(FOO CONFIG REQUIRED)
will be managed by
Hunter. In most cases, the only system requirement for building a Hunter
project is a recent CMake with
CURL
support and a working compiler correpsonding to the operative toolchain.
Hunter will maintain all dependencies in a
versioned
local
cache by
default (typically ${HOME}/.hunter
) where they can be reused in
subsequent builds and shared between different projects. They can also
be stored in a server side binary
cache --
select toolchains will be backed by a server side
binary cache (https://github.com/elucideye/hunter-cache) and will
produce faster first time builds (use them if you can!).
The
Travis
(Linux/OSX/iOS/Android) and
Appveyor
(Windows) CI scripts in the project's root directory can serve as a
reference for basic setup when building from source. To support cross
platform builds and testing, the CI scripts make use of
Polly: a set of common CMake
toolchains paired with a simple polly.py
CMake build script. Polly
is used here for convenience to generate CMake
command line
invocations -- it is not required for building Hunter projects.
To reproduce the CI builds on a local host, the following setup is recommended:
- Install compiler: http://cgold.readthedocs.io/en/latest/first-step.html
- Install CMake (and add to
PATH
) - Install Python (for Polly)
- Clone Polly and add
<polly>/bin
toPATH
Note: Polly is not a build requirement, CMake can always be used directly, but it is used here for convenience.
The bin/hunter_env.{sh,cmd}
scripts (used in the CI builds) can be
used as a fast shortcut to install these tools for you. You may want to
add the PATH
variables permanently to your .bashrc
file (or
equivalent) for future sessions.
Note: If you are building with an Android toolchain, you should set the TOOLCHAIN
variable appropriately before "sourcing" the hunter_env.{sh,cmd}
setup script. That will call install-ci-depepdencies.py
, and if an Android toolchain is detected, it will install it in your temporary _ci
folder. This helps standardize the setup for first time builds, although a different configuration or layout can be used after you are are able to build properly. If you have a pre-installed NDK and are familiar with Android builds, you can set your ANDROID_NDK_<version>
variable approriately (e.g., export ANDROID_NDK_r10e=${YOUR_PATH_TO_NDK_FOLDER}/android-ndk-r10e
). The latest Android Studio releases also provides native support for standard CMake builds (previously only a custom Android specific CMake could be used), so you can use Hunter based projects directly through Android Studio. For a sample project please see _android-studio-with-hunter
# For Android, set be sure to set the TOOLCHAIN variable before sourcing bin/hunter_env.{sh,cmd}
export TOOLCHAIN=android-ndk-r10e-api-19-armeabi-v7a-neon-hid-sections
export CONFIG=Release
Linux/OSX/Android/iOS | Windows |
---|---|
source bin/hunter_env.sh |
bin\hunter_env.cmd |
After the environment is configured, you can build for any supported
Polly
toolchain (see polly.py --help
) with a command like this:
polly.py --toolchain ${TOOLCHAIN} --config-all ${CONFIG} --fwd --install --verbose --reconfig
Note: The --reconfig
flag is included in the example above, which will re-run the CMake configure step (to incorporate CMake changes) for you. It is a reasonable step to add in cases where you aren't sure if it is needed.
Please see the README for the drishti-hci console application to see an example of a full eye tracking pipeline with the GPGPU optimizations.
Drishti is also available as a hunter package. If you would like to integrate drishti in your project, please see the hunter drishti package documentation.
Steps:
Add cmake/HunterGate.cmake
and a minimal cmake/Hunter/config.cmake
to your project:
mkdir -p cmake/Hunter
wget https://raw.githubusercontent.com/hunter-packages/gate/master/cmake/HunterGate.cmake -O cmake/HunterGate.cmake
wget https://raw.githubusercontent.com/ruslo/hunter/master/examples/drishti/config.cmake -O cmake/Hunter/config.cmake
Add HunterGate(URL <url> SHA1 <sha1>)
to the top of your CMakeLists.txt
(You can find updated release information here).
include("cmake/HunterGate.cmake")
HunterGate(
URL "https://github.com/ruslo/hunter/archive/v0.19.140.tar.gz"
SHA1 "f2c30348c05d0d424976648ce3560044e007496c"
LOCAL # use cmake/Hunter/config.cmake
)
Finally, add the drishti package to your CMakeLists.txt and link it to your target:
hunter_add_package(drishti)
find_package(drishti CONFIG REQUIRED)
target_link_libraries(your_app_or_lib drishti::drishti)
You can customize the drishti package (and dependencies) by specifying a VERSION and/or CMAKE_ARGS (options) list for each package in cmake/Hunter/config.cmake
.
Please see https://github.com/elucideye/drishti_hunter_test for a minimal working example using the drishti hunter package.
The configurations listed below have all been tested. In general, most
C++11 toolchains should work with minimal effort. A CI
comment
indicates that the configuration is part of the Travis or Appveyor CI
tests, so all Hunter packages will be available in the server side
binary cache.
Linux (Ubunty Trusty 14.04):
TOOLCHAIN=gcc-5-pic-hid-sections-lto
CONFIG=Release
# CITOOLCHAIN=libcxx
CONFIG=Release
# w/ clang 3.8
OSX:
TOOLCHAIN=osx-10-11-hid-sections-lto
CONFIG=Release
# CITOOLCHAIN=osx-10-12-sanitize-address-hid-sections
CONFIG=Release
# CITOOLCHAIN=xcode-hid-sections
CONFIG=Release
# generic
iOS:
TOOLCHAIN=ios-nocodesign-10-1-arm64-dep-9-0-device-libcxx-hid-sections-lto
CONFIG=MinSizeRel
# CITOOLCHAIN=ios-10-1-arm64-dep-8-0-hid-sections
CONFIG=Release
Android (from OSX):
TOOLCHAIN=android-ndk-r10e-api-19-armeabi-v7a-neon-hid-sections
CONFIG=MinSizeRel
# CITOOLCHAIN=android-ndk-r10e-api-19-armeabi-v7a-neon-hid-sections-lto
CONFIG=MinSizeRel
Windows:
TOOLCHAIN=vs-14-2015-sdk-8-1
CONFIG=Release
# CITOOLCHAIN=vs-14-2015-sdk-8-1
CONFIG=Debug
# CITOOLCHAIN=vs-14-2015-win64-sdk-8-1
CONFIG=Release
# CITOOLCHAIN=vs-14-2015-win64-sdk-8-1
CONFIG=Debug
# CI
The polly out of source build trees are located in
_builds/${TOOLCHAIN}
, the final build products (the stuff you want)
are installed in _install/${TOOLCHAIN}
, and the build logs are
dumped in _logs/${TOOLCHAIN}
. The iOS frameworks are installed in
_frameworks/${TOOLCHAIN}
.