- It requires Oculus Casting and a local backend to process video feed from the deployed Meta Mixed Reality app.
- You need a PC running the local backend with ngrok, with the main display showing the Oculus Casting tab in full screen.
- Pre-requisites
- Unity
- Meta Quest 3
- Gemini API Key
- ngrok
- Python
Run commmands:
cd server
choco install ngrok
ngrok config add-authtoken <token>
pip install -r requirements.txt
- Create server/.env file.
If you have Gemini API Key:
GOOGLE_API_KEY=AIz...PA8
- Start local python web server
Google Gemini Backend (Gemini Vision + Gemini Pro with Function Calling)
cd server && python gemini.py
- Make public with ngrok
In separate terminal in server folder, run:
cd server && ngrok http http://127.0.0.1:5000
I run this command so it's set to "trusting-ostrich-measured.ngrok-free.app/data" by default.
cd server && ngrok http --domain=trusting-ostrich-measured.ngrok-free.app http://127.0.0.1:5000
- Update the endpoint URL in the Meta Quest C# script.
- Build and run to Quest 3.
Build "mainScene" to Quest 3.
- Start Oculus Casting to computer, enter full screen.
- Talk to GARVIS by facing your palm up, stop recording your speech by facing your palm down.