brownhci / webgazer

WebGazer.js: Scalable Webcam EyeTracking Using User Interactions

WebGazer.js

WebGazer.js is an eye tracking library that uses common webcams to infer the eye-gaze locations of web visitors on a page in real time. The eye tracking model it contains self-calibrates by watching web visitors interact with the web page and trains a mapping between the features of the eye and positions on the screen. WebGazer.js is written entirely in JavaScript and with only a few lines of code can be integrated in any website that wishes to better understand their visitors and transform their user experience. WebGazer.js runs entirely in the client browser, so no video data needs to be sent to a server. WebGazer.js can run only if the user consents in giving access to their webcam.

• Official website
• API Docs

Features

• Real time gaze prediction on most major browsers
• No special hardware; WebGazer.js uses your webcam
• Self-calibration from clicks and cursor movements
• Easy to integrate with a few lines of JavaScript
• Swappable components for eye detection
• Multiple gaze prediction models
• Useful video feedback to user

Build the repository

If you want to build the repository from source follow these instructions:

# Ensure Node is downloaded: https://nodejs.org/en/download/ (tested on v16 and v18)
git clone https://github.com/brownhci/WebGazer.git
cd WebGazer
#install the dependencies
npm install
#build the project
npm run build

Examples

Examples of how WebGazer.js works can be found here.

to use on any website

<script src="webgazer.js" type="text/javascript" >

to use in any modern framework

const webgazer = require('webgazer'); // npm package 'webgazer' is sync with this repository

or you can you do

import webgazer from 'webgazer'

How to run the Example HTML files

Within the /www directory there are two example HTML files:

• calibration.html: This example includes additional user feedback, such as a 9-point calibration sequence, accuracy measurements and an informative help module.
• collision.html: This example contains a game where the user can move an orange ball with their eyes, which in turn collides with blue balls.

To run the example files as a server:

# Clone the repository and download NodeJS using the steps listed above
# Move into the www directory and download the additional dependencies
cd www
npm install
# Run the webpage index.html as a server
npm run serve

Browser Support

The following browsers support WebGazer.js:

• Google Chrome
• Microsoft Edge
• Mozilla Firefox
• Opera
• Safari

Publications

Note: The current iteration of WebGazer no longer corresponds with the WebGazer described in the following publications and which can be found here.

@inproceedings{papoutsaki2016webgazer,
author     = {Alexandra Papoutsaki and Patsorn Sangkloy and James Laskey and Nediyana Daskalova and Jeff Huang and James Hays},
title      = {{WebGazer}: Scalable Webcam Eye Tracking Using User Interactions},
booktitle  = {Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI-16)},
pages      = {3839--3845},
year       = {2016},
organization={AAAI}
}

@inproceedings{papoutsaki2017searchgazer,
author     = {Alexandra Papoutsaki and James Laskey and Jeff Huang},
title      = {SearchGazer: Webcam Eye Tracking for Remote Studies of Web Search},
booktitle  = {Proceedings of the ACM SIGIR Conference on Human Information Interaction \& Retrieval (CHIIR)},
year       = {2017},
organization={ACM}
}

@inproceedings{papoutsaki2018eye,
author={Papoutsaki, Alexandra and Gokaslan, Aaron and Tompkin, James and He, Yuze and Huang, Jeff},
title={The eye of the typer: a benchmark and analysis of gaze behavior during typing.},
booktitle={Proceedings of the 2018 ACM Symposium on Eye Tracking Research \& Applications (ETRA)},
pages={16--1},
year={2018},
organization={ACM}
}

Who We Are

• Alexandra Papoutsaki
• Aaron Gokaslan
• Ida De Smet
• Xander Koo
• James Tompkin
• Jeff Huang

Other Collaborators

• Nediyana Daskalova
• James Hays
• Yuze He
• James Laskey
• Patsorn Sangkloy
• Elizabeth Stevenson
• Preston Tunnell Wilson
• Jack Wong

Acknowledgements

Webgazer is developed based on the research that is done by Brown University, with recent work at Pomona College as well. The current maintainer is Jeff Huang. The calibration example file was developed in the context of a course project with the aim to improve the feedback of WebGazer, proposed by Dr. Gerald Weber and his team Dr. Clemens Zeidler and Kai-Cheung Leung.

This research is supported by NSF grants IIS-1464061, IIS-1552663, and the Brown University Salomon Award.

License

Copyright (C) 2016 Brown WebGazer Team

Licensed under GPLv3. Companies have the option to license WebGazer.js under LGPLv3 while their valuation is under $1,000,000. For other licensing options, please contact [email protected]

webgazer's Issues

Fallback to slower javascript version

This official websites demo works perfectly, but when I serve the webgazer.js script locally, it tells me that some webGL programs couldn't be setup and falls back to a slower javascript version. My browser (latest chrome) tells me webgl and canvas are fully hardware accelerated.

Can I help it somehow ?

(Looking at the demo site source, it seems like there a some undocumented requirements for embedding webgazer ( like the drawloop, some overlay stuff ))

Profiling

Run profiling and diagnostic tools to determine focus areas for optimization. Use this Issue to track areas where performance suffers the most.

Incorporate jsobjectdetect.js

Follow clmtrackr implementation.

Incorporate Head Movement

Currently the position of the head is not being accounted for. Any motion of the head can severely impact the accuracy of the model. Webgazer should include information about the head position in its regression models to improve accuracy.

License

Choose what license we will release the code under.

window.URL.createObjectURL will stop working soon

[Deprecation] URL.createObjectURL with media streams is deprecated and will be removed in M68, around July 2018. Please use HTMLMediaElement.srcObject instead. See https://www.chromestatus.com/features/5618491470118912 for more details.

Error Reporting

The pipeline of different extraction steps does not report errors informatively. The pipeline does not also track any accumulation of errors. This means that any issues are opaque to end users and much more difficult to debug. Will require API changes.

Transfer API

From Google Doc to HTML. Connect to website development.

Offline Training

Consider the effectiveness and viability of performing offline training. This could be individual user data, global user data, or data from a study. This may allow a baseline performance which jump starts the accuracy.

WebGazer script needs to be added on body not head

Need to investigate why clmtrackr collapses to a slower version when script added in head.

asm.js

Examine feasibility of building webgazer with asm.js for performance gains.

http://asmjs.org/

Use Temporal Smoothing for Improved Accuracy.

CLMTrakr works very well, but produces relatively jittery output. This is severely hampering the accuracy of the detector. We should implement some form of temporal filtering similar to a Kalman filter to increase accuracy and reduce noise. Other suggestions on how to combat this issue are welcome.

Updating regression on mouse move should be optional

The tracker regression data is updated on mouse click and mouse move under the assumption that the user is looking at the mouse pointer while performing those action. I think that for mouse movement this assumption is wrong. IMO in most cases the uses looks at the target of movement, instead of the mouse pointer.

What do you think?

No Git tags in GitHub repo

Hey @Skylion007,

I'd like to host your library, WebGazer.js on cdnjs — It is one of the most famous free and public web front-end CDN services which is used by ~1,143,000 websites worldwide.

The git auto-updater for cdnjs relies on git tags to recognize a release version and automatically update the library in the cdnjs repo.

If it's not too much to ask, can you please add git tags for this repo, say 1.0.0?

Webcam eyes tracking and performing functions

Hi Team,

I want to create a website where a link should open if I look at it for more than 2-3 seconds which will require webcam eyes tracking.
Is this possible with WebGazer?

It would be great if we get to see some demos of it.
Thanks.

Consider different Multidimensional Arrays Implementations

So after looking through the optimizations of this library, we are more or less at a local minimum when it comes to our naive representation of multidimensional arrays. They are currently untyped and simply use the classic Array of Array implementation. We should consider ways of storing our arrays in alternative ways and possibly typing them to get better performance. There is a case to be made that we might want to implement something similar to NDArray, but doing so it contingent on issue #35 . LU decomposition still takes up a rather large portion of the run time.

Currently here are three areas of optimization to focus on:

1. track() inside of CLMR tracker
2. getPupilFeatures()
3. LUDecomposition() and the entire ridge methods

Optimizing these areas will be important if we want to make it practical to run on mobile.

Some reading on the topic:
https://0fps.net/2013/05/22/implementing-multidimensional-arrays-in-javascript/
https://github.com/mikolalysenko/ndarray-experiments

Cross Domain Training Data Storage

Currently, we use HTML5 storage to store the user's training data. We may in the future want to add the ability to store local data cross domain using various tricks such as the this library uses.

Web Workers

Investigate the feasibility and performance gains from using web workers (javascript threads). Training the model could be moved to a separate thread for potentially large performance gains.

https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers

Pupil dilation?

Is it possible to detect pupil diameter, so we can also measure changes on its size?

As far as I can see, the library detects the eye shape, and puts a pixel on its center, as a reference to the pupil, but I'm not sure if is possible to detect where the iris finishes.

Thanks!

Create npm module for webgazer

Making it an NPM installable module will sure drive the package's adoption by the community.

Can webgazer be utilized to type with your eyes?

Great product!

Was wondering if webgazer can be used to type using an onscreen keyboard?

Use TypedArrays for DataWindows in util.js

Currently, datawindows utilize generic arrays. In the future, they should use TypedArrays to speed up computation. Also, this construct seems common enough where I would also recommend seeking out an optimized library to carry out this computation. Regardless, using generic arrays as is being utilize now will significantly slow down the library.

Convert repository so it's require JS compatible.

As part of #35, we should probably make this module follow the requirejs framework for easy integration with other projects. Resolving this issue would also help solve #26 and make it easier for us to include dependencies into the project. Any help in this regard would be greatly appreciated and we look forward to pull requests to help satisfy it.

Time to reach Accurate

Currently we assume that webgazer is accurate immediately. However, this is a false assumption for all the models. We should expose some ability for developers to determine when the model has begun to output accurate predictions

GPL licence

Hello,

Thanks for your excellent software. I was wondering whether your team would consider licensing this software under the LGPL so it can be used in commercial products—we're a small startup doing some work in the voice recognition area, and it would be fantastic to use your product to detect whether the user had finished talking, or whether they were just thinking, by the position of their eyes. I imagine we'd do a bit of work on the library to support this use case, and would happily give back anything we did to the project.

I know the GPL debate is a deeply personal one, but I actively write and maintain open source projects myself (as you'll see from my github) and licence under MIT. My feeling is that if I've gone to the trouble of writing library code, I'm happy for people to use it for whatever purpose they need it to.

Thank you for your consideration!

Roger

Which files need to be hosted on CDNJS?

Hey @Skylion007,

I'd like to host your library, WebGazer.js on cdnjs — It is one of the most famous free and public web front-end CDN services which is used by ~1,143,000 websites worldwide.

Before hosting the WebGazer.js library on cdnjs, I wanted to verify the files which need to be hosted online so users can use your library directly.

As far as I can understand, these are the files which need to be hosted:

build/webgazer.js
build/webgazer.min.js

Please clarify if any additions are needed. Thanks!

Webgazer works only over https. If you are doing development locally you need to setup a local server

Even after running it as http-server, the index.html file keeps giving alert "Webgazer works only over https. If you are doing development locally you need to setup a local server" on google chrome version 64.

Please help.

Example Pages

Creating simple example pages with demonstrating basic functionality of WebGazer. e.g. showing the video from clmtrackr and a circle for the gaze prediction. Once we have those pages move them to the website.

getPupil and related functions only in Linear Regression

For now the ridge regression is not utilizing the pupil position. Nevertheless the getPupil and its associated functions are used. We should move them to the linear regression pipeline, they waste unnecessarily computations.

Adjust video resolution via api

Currently video is set to be default video size * scaling factor where the scaling factor is .5. This should be changed to be a configurable parameter.

Determine Performance vs Feature and Training Example Size

Currently we don't have any indication as to how performance characteristics change when feature vectors are changed or when the total number of training examples grows large. Determining these growth relationships means we can set appropriate maximums and set optimal trade offs.

Blink Detector

Current skeleton code needs to be fleshed out. Missing a distance metric with which to determine the outliers corresponding to a blink.

Navigator.getUserMedia is off standard

The current draft of W3 standard no longer the encourages the use of navigator.getUserMedia but instead uses navigator.mediadevices.getUserMedia. Furthermore, the second standard uses a promise based system instead of a callback based one which holds several advantages.

https://w3c.github.io/mediacapture-main/#dom-mediadevices-getusermedia

Files not included in build script

src/ridgeRegThreaded.js and src/ridgeWorker.js are not included on the build/toCombine file list. Is this intentional? Right now the threadedRidge regression use a webgazer.reg.RidgeRegThreaded function which is not defined.

license.md does not contain license

The file only contains the licensing header information, but I didn't find the actual license file included in the project. You should include a copy of the license (found here http://www.gnu.org/licenses/gpl.txt).

Further speed up mat.js with OpenGL + LinAlg libraries

Currently, mat.js relies on naive, un-optimized code ported from Java to for use throughout the libraries. We should look into porting some of this code to rely on optimized, high performance code. If anyone has any Javascript libraries they would like to recommend, please list them here.

Documentation - Break into components

Tentative List of components

• Trackers for facial feature detectors
• Regression Models For now ridge regression
• Utilities

Incorporate tracking.js

Follow clmtrackr implementation.

WebGazer eye detection not accurate for people wearing spectacles

This example (https://webgazer.cs.brown.edu/demo.html?) gives less accurate result on wearing spectacles. How to solve?

x and y Canvas position for eyes, mouth...

Hi,

I am having a difficult time to get the x,y position for th eyes polygon. Is there any function that return this data?

node_modules folder missing

The demo.html relies on node_modules/ code which is missing from the repository

Clean Up Repo Commit Process + Make a Crossplatform Build Script

Currently, when you submit a change to the repository, you must first modify the individual source files, then run the build script, then copy the newly built script to the www folder. This is terribly inefficient and each change can become 23MB or more! We are also in desperate need of defining a style guide and documentation for committing to the repository.

Browser Cross Compatibility

Currently we only have tested webgazer on Chrome and Firefox. Ideally it should work across all major browsers, but this is currently unknown.

Different Regression Models

Implement and test accuracy of regression models.

How to stop video stream once .end() is called?

I'm curious how one would stop the video stream once webgazer.end() is called? I don't see any documentation on this and don't see anything in code where .stop() is being call on the stream.

As it is right now when I call webgazer.end() my webcam is still active.

Potential Issue: Warn user of potential downscaling

So apparently, it appears as the positions returned by the CLMGaze are the coordinates of the canvas. This means if the camera is fed in a higher resolution than the canvas, then the video gets down sampled significantly and the eye features only get reduced to bounding boxes that approximately 20 by 10 pixels instead of the usual sized bounding boxes.

Furthermore, since the pixels are taken from the canvas instead of directly from the webcam, this means that we are always feeding it very low resolution video even if we enforce the maximum resolution via the webcam API. Unfortunately, the current solution requires the video element to take up the entire screen on most devices that have HD webcams. However, it still seems to work okay when I add the hidden CSS selector to the video ID. Perhaps we will need to have a hidden video source to serve as the real canvas to pull the pixels from and a separate video to show the user.

The issue has to do with how we attempt to overlay the canvas of the video. While the raw video output from the webcam remains unchanged, Chrome actively resizes of the video and CLMTrakr appears to only have access to the none webcam resolution.

Upgrading the video canvas to HD dimensions greatly improved the accuracy of the eye tracker to within an inch or so of my screen after only a few training samples. This should be a high priority to improve the accuracy of the tracker significantly and should make this repository much more viable.

TLDR: If we fix how we are accessing the video stream we can increase the accuracy by almost a significant amount.

Fixation and Saccade algorithms

We have no algorithms in place to detect fixations or saccades. This would be a useful feature for scientific domains.

Add a Unit Testing Framework and Write Unit Tests

While we are cleaning up the repository, it would certainly help to add a unit tests to this repository, especially now that we are optimizing the code base. Any help and all help would greatly be appreciated in this matter.

threadedRidge and linear regression models return (0, 0)

On the master branch, fresh copy of the aggregate webgazer.js, neither threadedRidge nor linear regression updates the predicted x and y values. They both return {x: 0, y: 0}. For contrast, both regular ridge and weighted Ridge work fine.

User specific properties

At the moment there is a skeleton set up for saving and loading configurations and training data on the same domain. This should be extended to incorporate other user specific components such as: computer performance maximums, generalized features, model weights, etc.

We should also see about possible ways of sharing data across different domains. This may require a minimal webserver to share localstorage data or turning the library into a plugin.