👀 VRCFaceTracking

Provides real eye tracking and lip tracking in VRChat via the HTC Vive Pro Eye's SRanipal SDK.

🎥 Demo

https://www.youtube.com/watch?v=KbbfYW-hnMk

🛠 Avatar Setup

For this app to work, you'll need to be using an avatar with the correct parameters or an avatar config file with the correct mappings. The system is designed to control your avatar's eyes and lips via simple blend states but what the parameters control is completely up to you.

List of Parameters

👀 Eye Parameters

Eye Tracking Setup Guide

It's not required to use all of these parameters. Similar to the setup of parameters with Unity Animation Controllers, these are all case-sensitive and must be copied EXACTLY as shown into your Avatar's base parameters. A typical setup might look something like this:

Please make sure you disable the built in simulated eye tracking in your avatar descriptor. This will almost certainly mess with things if left on. Personally, I've also had some issues with blink blendshapes being overrided by my gesture layer so if you can see your eyes fine but others see them half closed, I would reccomend removing your Additive layer so the default is not applied. It should say "None (Runtime Animator Controller)" if it's removed correctly.

Feel free to consult the wiki for a setup guide and more info as to what each parameter does

👄 Lip Parameters

There are a large number of parameters you can use for lip tracking.

Lip Tracking Setup Guide - Basic setup guide

Combined Lip Parameters - Combined parameters to group mutually exclusive face shapes.

Blend Shape Setup - Reference of standard blend shapes for to be used with facetracking

You can also refer to this visual guide from NeosVR on what the following API parameters do: https://casuallydotcat.wordpress.com/2020/02/10/the-ultimate-neos-blend-shape-guide-february-2020/

⛓ External Modules

Use the following modules to add support for other hardware:

VRCFTVarjoModule - Adds support for Varjo eye tracking (Varjo Aero)
LiveLink - Adds support for LiveLink face tracking (iPhone)
PimaxEyeTracking - Adds support for Pimax eye tracking
VRCFTOmniceptModule - Adds support for HP Omnicept eye tracking
NoVRCFT - Adds support for webcam based eye and face tracking using NeosWCFaceTrack and OpenSeeFace

👋 Credits

VIVE for the SRanipal SDK and their awesome hardware! ❤

hai-vr / vrcfacetracking Goto Github PK