tcsantini / eyerectoo

Mirror to https://es-git.cs.uni-tuebingen.de/santini/EyeRecToo

EyeRecToo

EyeRecToo is a second-generation hardware-agnostic open-source software for head-mounted eye trackers. Its main raison d'être is to provide an open platform to replace the data acquisition functionality from eye-tracker vendors' software, which typically are expensive, closed-source, and geared toward their own devices. It deprecates EyeRec.

For data analysis functionality replacement, we recommend Eyetrace.

Why the name?

Originally, EyeRec was a phonetical play on words and the eye-related functionality as in I rec[ord]. EyeRecToo adds on the word play: It is the version two and, similar to EyeRec, records too. All bad puns are intended :-)

Supported Platforms

• Windows 64 bits

This is the platform that gets the most testing and recommended.

• Initiall support for Ubuntu 16.04 64 bits has been added; no binaries available at the moment though.

Only supports camera access using the uvcengine, including the Pupil Labs eye tracker. It has been tested for pupil detection, gaze estimation, and data recording. This is alpha.

• Notes

It's also possible to get EyeRecToo running on 32 bits platforms, but no Pupil eye tracker support is available.

Supported Devices

Theoretically, cameras using DirectShow (on Windows) and v4l2 (on Linux) should work out of the box.

UVC-compliant cameras are suuported through the UVC Engine. On Windows, it is typically necessary to change camera drivers to be able to use the uvcengine; see the Running section for details.

Tested Eye Trackers:

• Dikablis Essential / Pro (see Ergoneers)
• Pupil DIY (see Pupil Labs)
• Eivazi's microscope add-on (see Eivazi, S et al. 2016)
• Pupil Eye Tracker, both Cam1 and Cam2 models (see Pupil Labs)
• The inconspicuous modular eye trackers from Eivazi, S et al 2018

Tested webcams:

• Playstation Eye
• Microsoft LifeCam HD-3000
• Microsoft LifeCam HD-5000
• Microsoft LifeCam HD-6000
• Microsoft LifeCam VX-1000
• Logitech C920
• Logitech C510
• Logitech C525
• Logitech QuickCam PRO 9000
• Logitech QuickCam Express

Running

Binaries can be downloaded here.

NOTE: Additionally, you may need to install drivers to access your eye tracker cameras:

1. Dikablis eye trackers require the VRMagic drivers (make sure you have the 64 bits version)
2. For Pupil eye trackers, check the Pupil Labs' guide.
3. (Windows only) for better interfacing with UVC-compliant cameras, we recommend to change the camera drivers to enable the uvcengine. This can be easily achieved through Zadig: a) Check Options->"list all devices", b) uncheck Options->"Ignore Hubs or Composite Parents" , c) change your camera's interface driver to libusbK, d) change your camera's composite parent to libusbK

Developing

The official EyeRecToo repository is located at: https://atreus.informatik.uni-tuebingen.de/santini/EyeRecToo

Windows

The particulars of the test build system we use are:

• QtCreator 4.0.3
• Qt >= 5.7.0 (MSVC 2015, 64 bit)
• Visual Studio 2015
• Microsoft Windows 8.1

All libraries/includes (for Windows 64 bits) should be in the deps directory already; to build:

1. Install Visual Studio 2015
2. Install Qt 5.7.0 (and Qt Creator)
3. Open the project
4. Run qmake and build
5. Before running, add deps/runtime/Release or deps/runtime/Debug to your PATH accordingly to your build type.

Both Release and Debug versions are functional although the latter may run at much lower sampling rates.

Linux

The particulars of the test build system we use are:

• QtCreator 4.4.1
• Qt 5.9.2
• GCC 5.4.0
• Ubuntu 16.04.3

Requires the installation of some packages (see README.LINUX); to build:

1. Install Qt 5.9.2 (and Qt Creator)
2. Run ./linux-setup.sh (or do the steps manually)
3. Open the project
4. Run qmake and build
5. Should be good to go :-)

Data Format

Text data files (.tsv) use the tsv format -- i.e., tab delimited files.

Video data files (.mp4) are uncompressed MJPEG files using MPEG-4 Part 14 containers. Note: timing information from these files is not reliable. You can find the timestamp for each frame in its respective *Data.tsv counterpart.

---

• meta.csv

Contains meta data about the program, recording, and host machine.

• <Camera Widget>Data.tsv and <Camera Widget>.mp4

Frames and respective processing tuples, e.g., detected pupil, markers, etc; these are FieldData and EyeData in the code. The number of entries in the .tsv must match the amount of frames in the video file. For each camera widget (e.g., eyes, field), one of these pairs is generated with the appropriate name -- e.g., RightEye.tsv, RightEye.mp4.

• JournalData.tsv

Synchronized data file. This file includes entries containing only synchronized tuples (DataTuple in the code). Note: despite gaze estimation being part of FieldData (since it's the correct reference frame), it is available only after synchronization and, thus, in this file, not on FieldData.tsv).

• [0-9]*-calibration.tup

Data tuples used for calibration. The number prefix indicates the timestamp of the calibration.

---

For convenience, we also provide the ERTViewer.exe application, which allows one to visualize, do some basic annotation, and record gaze-overlayed videos from an EyeRecToo recording.

Note: This is just a temporary application until we start integrating a data viewer/analyzer for video data in Eyetrace. There are some bugs (e.g., the overlayed pupil can be slightly unsynchronized), and there is no focus on usability nor documentation.

Markers

Default ArUco markers can be found under the utils/all-markers directory.

Default collection marker for CalibMe calibrations is provided at utils/CalibMeCollectionMarker.png.

Camera calibration patterns are utils/asymmetric_circle_grid_4x11_20mm.pdf (default) and utils/chessboard_9x6_25mm.pdf.

These markers can also be found only here.

Video Tutorials

An introduction to EyeRecToo is available on YouTube. It is a bit outdated since its from 2017, but the principles remain the same.

<iframe width="640" height="480" src="https://www.youtube.com/embed/vbrfjpfYkvg" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

If you want to learn more about CalibMe and unassisted calibration, here is a talk about it at CHI2017.

<iframe width="640" height="480" src="https://youtube.com/embed/gYtF3j2Hdl4" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe>

Configuration Files

All configuration files are found in the cfg directory. These are *.ini files that can be easily edited if needed.

Keyboard Shortcuts

These shortcuts can be changed through cfg/CommandManager.ini. Key codes can be found here.

Exposed functionality at the moment includes:

• remoteCalibrationToggleKey = Qt::Key_PageDown:

Toggles calibration (including CalibMe marker collection).

• calibrationToggleKey = Qt::Key_S:

Toggles calibration.

• collectionToggleKey = Qt::Key_C:

Toggles CalibMe marker collection.

• recordingToggleKey = Qt::Key_R:

Toggles recording.

• remoteRecordingToggleKey = Qt::Key_PageUp:

If subject is not set, sets subject to "remote". Toggles recording.

• previewToggleKey = Qt::Key_B:

Freezes previews; useful, e.g., during development to check gaze estimation / pupil detection, or for didactic purposes.

eyerectoo's Issues

VRmagic drivers website

The link to the VRmagic drivers in the readme is outdated.
I contacted VRmagic and they moved the website to this location: https://www.vrmagic-imaging.com/service/downloads/usb-platform-downloads

Request for the source code of "Grip"

Hi, Santini.

Recently, I try to find the implementation of Grip in your EyeRec and EyeRecToo. I also found the project web of eyerectoo was moved to: https://www.hci.uni-tuebingen.de/research/Engagement%20&%20Interaction/eyerectoo.html.
However, I can't find the source code of "Get a Grip". I read the code of gaze estimation in EyeRecToo but I think it may not be the implementation. Could you share the source code?

Thanks for your contributions to the community. Looking forward to your reply : )

Problem Installing Qt 5.9.2

Now-a-days, the linux version of Qt 5.9.2 is not available, will it run with another version of Qt? as I tried but the config files are very specific to 5.9.2 version.

Incorrect calls to cv::Sobel() in PuRe?

It is hard to be sure with C++ being as unsafe as it is, but it looks like the two calls to cv::Sobel() take one too few arguments. They look like:

Sobel(blurred, dx, dx.type(), 1, 0, 7, 1, BORDER_REPLICATE);
Sobel(blurred, dy, dy.type(), 0, 1, 7, 1, BORDER_REPLICATE);

but the interface to cv::Sobel() is:

Sobel( InputArray src, OutputArray dst, int ddepth,
                         int dx, int dy, int ksize = 3,
                         double scale = 1, double delta = 0,
                         int borderType = BORDER_DEFAULT );

Note that the borderType parameter is in the 8th position, but the calls in PuRe.cpp put it at the 7th position. I'm guessing that there is a missing delta parameter. In OpenCV (3.4.2 at least), BORDER_DEFAULT is 1 (which means the value of delta being used here is 1), and the borderType being used is BORDER_DEFAULT, not BORDER_REPLICATE.

Running Grip on Recorded PupilCore Data

Hi Thiago,

Can grip be run on data that was already collected using Pupil Core? I am looking for a script(preferably python) that can run on existing recordings. I set up EyeRecToo but it only seems to work with new recordings.

Thanks in advance.