brownhci / webgazer Goto Github PK

Run profiling and diagnostic tools to determine focus areas for optimization. Use this Issue to track areas where performance suffers the most.

I'm curious how one would stop the video stream once webgazer.end() is called? I don't see any documentation on this and don't see anything in code where .stop() is being call on the stream.

As it is right now when I call webgazer.end() my webcam is still active.

As part of #35, we should probably make this module follow the requirejs framework for easy integration with other projects. Resolving this issue would also help solve #26 and make it easier for us to include dependencies into the project. Any help in this regard would be greatly appreciated and we look forward to pull requests to help satisfy it.

The pipeline of different extraction steps does not report errors informatively. The pipeline does not also track any accumulation of errors. This means that any issues are opaque to end users and much more difficult to debug. Will require API changes.

Making it an NPM installable module will sure drive the package's adoption by the community.

While we are cleaning up the repository, it would certainly help to add a unit tests to this repository, especially now that we are optimizing the code base. Any help and all help would greatly be appreciated in this matter.

The tracker regression data is updated on mouse click and mouse move under the assumption that the user is looking at the mouse pointer while performing those action. I think that for mouse movement this assumption is wrong. IMO in most cases the uses looks at the target of movement, instead of the mouse pointer.

What do you think?

Follow clmtrackr implementation.

The demo.html relies on node_modules/ code which is missing from the repository

Currently we assume that webgazer is accurate immediately. However, this is a false assumption for all the models. We should expose some ability for developers to determine when the model has begun to output accurate predictions

Hi Team,

I want to create a website where a link should open if I look at it for more than 2-3 seconds which will require webcam eyes tracking.
Is this possible with WebGazer?

It would be great if we get to see some demos of it.
Thanks.

Tentative List of components

• Trackers for facial feature detectors
• Regression Models For now ridge regression
• Utilities

So after looking through the optimizations of this library, we are more or less at a local minimum when it comes to our naive representation of multidimensional arrays. They are currently untyped and simply use the classic Array of Array implementation. We should consider ways of storing our arrays in alternative ways and possibly typing them to get better performance. There is a case to be made that we might want to implement something similar to NDArray, but doing so it contingent on issue #35 . LU decomposition still takes up a rather large portion of the run time.

Currently here are three areas of optimization to focus on:

1. track() inside of CLMR tracker
2. getPupilFeatures()
3. LUDecomposition() and the entire ridge methods

Optimizing these areas will be important if we want to make it practical to run on mobile.

Some reading on the topic:
https://0fps.net/2013/05/22/implementing-multidimensional-arrays-in-javascript/
https://github.com/mikolalysenko/ndarray-experiments

Currently we only have tested webgazer on Chrome and Firefox. Ideally it should work across all major browsers, but this is currently unknown.

Currently video is set to be default video size * scaling factor where the scaling factor is .5. This should be changed to be a configurable parameter.

So apparently, it appears as the positions returned by the CLMGaze are the coordinates of the canvas. This means if the camera is fed in a higher resolution than the canvas, then the video gets down sampled significantly and the eye features only get reduced to bounding boxes that approximately 20 by 10 pixels instead of the usual sized bounding boxes.

Furthermore, since the pixels are taken from the canvas instead of directly from the webcam, this means that we are always feeding it very low resolution video even if we enforce the maximum resolution via the webcam API. Unfortunately, the current solution requires the video element to take up the entire screen on most devices that have HD webcams. However, it still seems to work okay when I add the hidden CSS selector to the video ID. Perhaps we will need to have a hidden video source to serve as the real canvas to pull the pixels from and a separate video to show the user.

The issue has to do with how we attempt to overlay the canvas of the video. While the raw video output from the webcam remains unchanged, Chrome actively resizes of the video and CLMTrakr appears to only have access to the none webcam resolution.

Upgrading the video canvas to HD dimensions greatly improved the accuracy of the eye tracker to within an inch or so of my screen after only a few training samples. This should be a high priority to improve the accuracy of the tracker significantly and should make this repository much more viable.

TLDR: If we fix how we are accessing the video stream we can increase the accuracy by almost a significant amount.

Consider the effectiveness and viability of performing offline training. This could be individual user data, global user data, or data from a study. This may allow a baseline performance which jump starts the accuracy.

The current draft of W3 standard no longer the encourages the use of navigator.getUserMedia but instead uses navigator.mediadevices.getUserMedia. Furthermore, the second standard uses a promise based system instead of a callback based one which holds several advantages.

https://w3c.github.io/mediacapture-main/#dom-mediadevices-getusermedia

Great product!

Was wondering if webgazer can be used to type using an onscreen keyboard?

Currently, mat.js relies on naive, un-optimized code ported from Java to for use throughout the libraries. We should look into porting some of this code to rely on optimized, high performance code. If anyone has any Javascript libraries they would like to recommend, please list them here.

Hey @Skylion007,

I'd like to host your library, WebGazer.js on cdnjs — It is one of the most famous free and public web front-end CDN services which is used by ~1,143,000 websites worldwide.

The git auto-updater for cdnjs relies on git tags to recognize a release version and automatically update the library in the cdnjs repo.

If it's not too much to ask, can you please add git tags for this repo, say 1.0.0?

For now the ridge regression is not utilizing the pupil position. Nevertheless the getPupil and its associated functions are used. We should move them to the linear regression pipeline, they waste unnecessarily computations.

[Deprecation] URL.createObjectURL with media streams is deprecated and will be removed in M68, around July 2018. Please use HTMLMediaElement.srcObject instead. See https://www.chromestatus.com/features/5618491470118912 for more details.

Follow clmtrackr implementation.

We have no algorithms in place to detect fixations or saccades. This would be a useful feature for scientific domains.

Need to investigate why clmtrackr collapses to a slower version when script added in head.

src/ridgeRegThreaded.js and src/ridgeWorker.js are not included on the build/toCombine file list. Is this intentional? Right now the threadedRidge regression use a webgazer.reg.RidgeRegThreaded function which is not defined.

From Google Doc to HTML. Connect to website development.

Currently, when you submit a change to the repository, you must first modify the individual source files, then run the build script, then copy the newly built script to the www folder. This is terribly inefficient and each change can become 23MB or more! We are also in desperate need of defining a style guide and documentation for committing to the repository.

At the moment there is a skeleton set up for saving and loading configurations and training data on the same domain. This should be extended to incorporate other user specific components such as: computer performance maximums, generalized features, model weights, etc.

We should also see about possible ways of sharing data across different domains. This may require a minimal webserver to share localstorage data or turning the library into a plugin.

Examine feasibility of building webgazer with asm.js for performance gains.

http://asmjs.org/

Currently, we use HTML5 storage to store the user's training data. We may in the future want to add the ability to store local data cross domain using various tricks such as the this library uses.

Investigate the feasibility and performance gains from using web workers (javascript threads). Training the model could be moved to a separate thread for potentially large performance gains.

https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers

Creating simple example pages with demonstrating basic functionality of WebGazer. e.g. showing the video from clmtrackr and a circle for the gaze prediction. Once we have those pages move them to the website.

Current skeleton code needs to be fleshed out. Missing a distance metric with which to determine the outliers corresponding to a blink.

Hi,

I am having a difficult time to get the x,y position for th eyes polygon. Is there any function that return this data?

Is it possible to detect pupil diameter, so we can also measure changes on its size?

As far as I can see, the library detects the eye shape, and puts a pixel on its center, as a reference to the pupil, but I'm not sure if is possible to detect where the iris finishes.

Thanks!

Hey @Skylion007,

I'd like to host your library, WebGazer.js on cdnjs — It is one of the most famous free and public web front-end CDN services which is used by ~1,143,000 websites worldwide.

Before hosting the WebGazer.js library on cdnjs, I wanted to verify the files which need to be hosted online so users can use your library directly.

As far as I can understand, these are the files which need to be hosted:

build/webgazer.js
build/webgazer.min.js

Please clarify if any additions are needed. Thanks!

Even after running it as http-server, the index.html file keeps giving alert "Webgazer works only over https. If you are doing development locally you need to setup a local server" on google chrome version 64.

Please help.

Choose what license we will release the code under.

This official websites demo works perfectly, but when I serve the webgazer.js script locally, it tells me that some webGL programs couldn't be setup and falls back to a slower javascript version. My browser (latest chrome) tells me webgl and canvas are fully hardware accelerated.

Can I help it somehow ?

(Looking at the demo site source, it seems like there a some undocumented requirements for embedding webgazer ( like the drawloop, some overlay stuff ))

Hello,

Thanks for your excellent software. I was wondering whether your team would consider licensing this software under the LGPL so it can be used in commercial products—we're a small startup doing some work in the voice recognition area, and it would be fantastic to use your product to detect whether the user had finished talking, or whether they were just thinking, by the position of their eyes. I imagine we'd do a bit of work on the library to support this use case, and would happily give back anything we did to the project.

I know the GPL debate is a deeply personal one, but I actively write and maintain open source projects myself (as you'll see from my github) and licence under MIT. My feeling is that if I've gone to the trouble of writing library code, I'm happy for people to use it for whatever purpose they need it to.

Thank you for your consideration!

Roger

This example (https://webgazer.cs.brown.edu/demo.html?) gives less accurate result on wearing spectacles. How to solve?

The file only contains the licensing header information, but I didn't find the actual license file included in the project. You should include a copy of the license (found here http://www.gnu.org/licenses/gpl.txt).

Currently we don't have any indication as to how performance characteristics change when feature vectors are changed or when the total number of training examples grows large. Determining these growth relationships means we can set appropriate maximums and set optimal trade offs.

Currently the position of the head is not being accounted for. Any motion of the head can severely impact the accuracy of the model. Webgazer should include information about the head position in its regression models to improve accuracy.

Implement and test accuracy of regression models.

CLMTrakr works very well, but produces relatively jittery output. This is severely hampering the accuracy of the detector. We should implement some form of temporal filtering similar to a Kalman filter to increase accuracy and reduce noise. Other suggestions on how to combat this issue are welcome.

Currently, datawindows utilize generic arrays. In the future, they should use TypedArrays to speed up computation. Also, this construct seems common enough where I would also recommend seeking out an optimized library to carry out this computation. Regardless, using generic arrays as is being utilize now will significantly slow down the library.

On the master branch, fresh copy of the aggregate webgazer.js, neither threadedRidge nor linear regression updates the predicted x and y values. They both return {x: 0, y: 0}. For contrast, both regular ridge and weighted Ridge work fine.