torinmb / mediapipe-touchdesigner

GPU Accelerated MediaPipe Plugin for TouchDesigner

MediaPipe TouchDesigner Plugin

A GPU Accelerated, self-contained, MediaPipe Plugin for TouchDesigner that runs on Mac and PC with no installation. This project currently supports all MediaPipe vision models except Interactive Segmentation and Image Embedding.

Overview

To get an idea of what's possibe with the plugin, and a quick tutorial on how to get up and running, check out our introduction video on YouTube

Setup

Download the latest release.zip from the Release Section. Open up the MediaPipe TouchDesigner.toe file. All of the components are stored inside the /toxes folder. The main component is MediaPipe.tox. All of the other components are examples of how to load and display the associated model data in TouchDesigner.

On the MediaPipe component once it's loaded you can select your webcam from the drop-down. You can turn on and off the different MediaPipe models as well as preview overlays. There're also sub-menus available for each model to customize them further.

A note on resolution

Currently the model is limited to 720p input resolution - as long as that's a resolution your webcam supports, you're good to go.

Components

The plugin consists of a number of components:

Media Pipe tox

This component launches a Chromium web browser to host and run all of the MediaPipe vision tasks. It has a DAT output for each vision task, as well as a TOP output showing the video feed and any overlays.

Face detector tox

Use this to process the face detection results

Face detection guide

Face tracking tox

Use this to process the facial landmark detection results

Face landmark detection guide

Hand tracking tox

Use this to process the hand landmark and gesture detection results

Gesture recognition task guide

Object tracking tox

Use this to process the object detection results

Object detection task guide

Pose tracking tox

Use this to process the pose landmark detection results

Pose landmark detection guide

Image segmentation tox

Use this to key out segmentation results Image segmentation guide

Note: If you're hoping to get the most accurate web-cam cutout use the MultiCam model in the MediaPipe.tox. There's also a toggle to display only the background cutout which you can enable while on the multiclass model.

Image classification tox

Use this to identify what the image might contain Image classification guide

Sending TOPs from TouchDesigner

SpoutCam on Windows

If you are on Windows, you can use Spout to send any TOP from TouchDesigner to MediaPipe by using a Syphon Spout Out TOP and SpoutCam. Take a look at our introduction video on YouTube for how to set it up.

Download SpoutCam

SpoutCam

Unzip the folder

Run SpoutCam Settings.exe

Note there is no installer, it will run from wherever the folder is

• Enter the frame rate and resolution to match your source material in TouchDesigner
• Enter the default TouchDesigner Spout output name of TDSyphonSpoutOut into the Starting Sender box
• Click register to create the virtual webcam

Drop a Syphon Spout Out TOP into your project

Feed your desired TOP into it

In MediaPipe, select SpoutCam as your webcam source

The media should appear with no frames of delay!

Diagnosing SpoutCam showing only noise, and other Spout problems

For diagnosing this issue, I recommend downloading the Spout2 files from the same GitHub profile as the SpoutCam Plugin. This adds some settings and diagnostics options when working with Spout. In my case, my laptop graphics proved to be the problem! When running diagnostics, checking for compatibility and looking at the running spout cam process (all possible through the SpoutPanel and SpoutSettings) it told me that the texture sharing was failing, thus resulting in the noise. After reading through the SpoutSettings and their explanations (not the SpoutCamSettings!), I made sure to put all processes involving spout (so sender & receiver) on the same graphics pipeline (by changing the .exe graphic settings in Windows, this is explained in SpoutSettings), since my laptop has both integrated CPU graphics, as well as a GPU.

This actually fixed my issue!

So for anyone that has a laptop, multiple GPUs or any other Spout issues in combination with this TouchDesigner plugin, I recommend downloading the rest of the Spout software and tinkering a bit.

Thanks to @vVattghern for finding this answer

Syphon on Mac

There is no SpoutCam equivalent on Mac, but you can use Syphon to send video to OBS, and then use the OBS Virtual Webcam output to the MediaPipe task. It's not the most elegant solution, but it works.

Performance tips

Realtime CHOP

There are lots of interesting bits of data in the CHOP output of the MediaPipe tox:

detectTime

How long the MP detector took to run in ms.

drawTime

How long the overlays and segmentation colors took to draw in ms.

sourceFrameRate

The frame rate of the webcam video source MediaPipe is using.

realTimeRatio

What ratio of a frame it took to process the video. So 0.1 means 0.1 of a frame was needed for MediaPipe to process. This uses the project's cookRate as a reference

totalInToOutDelay

The number of project frames delay MediaPipe introduces. In TouchDesigner 2022.33910 this is at least 3 frames for the web browser component. If you are using Spout or Syphon to send video to a virtual webcam, you can use this as a parameter in a cache TOP to sync your input video source back up with the MediaPipe output TOP.

isRealTime

Tells you if the whole process is able to keep up with the input frame rate.

Turn off what you're not using

The MediaPipe detection tasks are very CPU and GPU intensive, so turn off any that you aren't using

Hyperthreading

If you are on a PC, you can greatly improve all CPU-based render times within TouchDesigner, including the MediaPipe tasks by disabling HyperThreading (Intel CPUs) or Simultaneous Multi-Threading (SMT - AMD CPUs). On my laptop it was a 60-80% improvement.

Enabling/disabling HyperThreading is done in your system BIOS, so look up the instructions on how to do that for your computer. You can re-enable HyperThreading at any point if you want to go back and it is not a risky change to make to your system.

How do the plugins work?

This project loads the different MediaPipe models through a web browser. All of the ML models are downloaded locally and stored inside TouchDesigner's virtual file system including the website so the component can run without an internet connection. The models run using web assembly and the data coming back from the models are piped into TouchDesigner through a local WebSocket server running inside TD. This allows the components to run as standalone .tox files with GPU acceleration on any device with no setup.

Architecture

As MediaPipe currently only supports GPU acceleration via the Javascript implementation, and this is the only version that does not require installing local libraries, we have implemented this version, by using the following three main components:

1. Web (and websocket) server
2. Web browser
3. JSON decoders

Web server

The web server component has an embedded set of web pages that are served to the web browser just like any website would do. It also acts as a websocket server that allows two-way-communication between the web browser and TouchDesigner.

Web browser

The embedded Chromium support in TouchDesigner allows us to run a full web browser within TouchDesigner. This web browser opens the web pages served by the web server, which allow it to run all of the MediaPipe detection components and render the final video stream. The web browser also sends coordinate data and other detection data back to TouchDesigner via a websocket connection.

JSON decoders

We send the data from the MediaPipe instance back to TouchDesigner in JSON format. We then use the additional detection tox files to process this JSON data into useful things that can be used elsewhere in TouchDesigner.

Building from source

This package uses yarn with vite inside node, which gives you a few options. Firstly, you need to download and install node.js

Installation

Install Node.js

Install yarn package manager globally npm install --global yarn

yarn install

This installs vite and all the other dependencies required. Should only be needed the first time.

Debugging of the web page

There are 2 options for debugging the web page

1. Debug the page as-is within TouchDesigner
2. Live debug a development page

Debug the existing web page

Once the MediaPipe project has loaded withing TouchDesigner, you can open a regular Chrome browser on your desktop and go to http://localhost:9222 which will open the developer tools console for the Chromium instance embedded within the MediaPipe component. Note that this will have a performance impact as the page is being rendered twice (once in TouchDesigner, once in Chrome), but it gives you access to the console log to see what's going on and maybe debug some issues.

Live debug a development page

yarn dev

This launches a tiny web server on port 5173 that fires a reload to the browser every time you save a file change. Super useful for debugging the web page.

Suggested use for this:

1. Run yarn dev via the command line, or the console window of VSCode
2. Load the TouchDesigner project and toggle all the settings you'll want to test with.
3. Enter the MediaPipe COMP
4. Disable the webbrowser component by clicking the X
5. Copy the current_url DAT string
6. Paste the url into a new Chrome tab
7. Edit the url and replace the port number with 5173, so for example replace localhost:3001 with localhost:5173

Chrome will now load the page from yarn dev, while still communicating with TouchDesigner via websockets. Note that parameter changes you make in TouchDesigner will not be reflected in the web page unless you repeat the steps to copy the url.

If you press F12 to open the Chrome Developer Tools you can start digging into the console to see error messages.

Build

yarn build

This builds all the web page files and puts them into the _mpdist folder. If this folder exists, the web pages used within the MediaPipe COMP will be served from here.

Version Update / Release

There is a build_release COMP in the top level at the top of the toe file that handles an automated build process.

To create a release zip file:

1. Open the MediaPipe TouchDesigner.toe file
2. Navigate to the layout you want to be loaded when someone opens the project for the first time.
3. Press Ctrl Alt B to trigger a build. The project will lock up for a while during the build process.
4. The build script will:

• Remove and recreate a release folder.
• Perform a yarn install
• Perform a yarn build
• Load all files from _mpdist into the MediaPipe COMP Virtual File System
• Remove all external script references on DATs
• Export all tox files to release/toxes
• Save the toe file to release
• Zip the contents of the release folder to release.zip
• Give you a popup showing the number of errors.
• Clicking ok (or anywhere!) causes the toe file to reload from the main folder so you're back where you started.
• If there were any errors in the build, you can turn on the TouchDesigner text console and run the build again to see what the errors are.

Package Version Update

yarn version --patch for backward-compatible bug fixes.

yarn version --minor for backward-compatible new features.

yarn version --major for changes that break backward compatibility.

mediapipe-touchdesigner's Issues

Open the plugin normally, but there is a black edge on the right side of the camera, help!

Hello, everyone.
My plug-in can open normally, but there will be a black area on the right side, how to solve it?
System: MAC M1 PRO

Latest Release issues

Noticed that older versions of mediapipe still work but after each save and file rename TD will fail to load all the media pipe assets and whole networks are gone inside the mideapipe window , including operator IN and OUT nodes.
could this be due to the new release update ?
it was saving a and reloading fine 2 weeks ago last release update was 7 days ago?
images below of the standard network vs the missing one ,

im teaching a class MAY 11TH using touchdesinger and mediapipe was a big part of the workshop.

please get back to me

SpoutCam source not available

Hello there,

I am new to TouchDesigner. I'm trying to setup this great mediapipe TD plugin with an NDI camera as my source.
As suggested, I have replaced the VideoDeviceIn TOP with the NDI in TOP and I have followed the instructions in the youtube video for setting up SpoutCam in my windows environment.
My issue is that SpoutCam is not showing as a webcam source in the mediapipe container.

Any ideas?

Thanks in advance!
(I have the latest releases of the plugin and SpoutCam)

Num Poses

Can the number of detected Poses be modified?
Thanks

Fixed: Sometimes ML not running.

Hi there. Again, so in love with this implementation and how smoothly you can install.
No worries about running servers ourselves or anything. Really beautiful.

One Issue I keep having for a couple of days now is that sometimes the ML won't run.

Video-feed (through ndi in my case) runs fine.
And I am checking the port in localhost in the browser. And indeed I see the models start running in the console outputs.

Not that I have a lot of slack in my wits, So I am at their end.
hehe.

I have also checked the resolution. it is 1280x720.
So I don't think this is the problem too?

Thank you very much!

VideoDeviceIn not running

Hi Torin and Dom,

First I want to let you know my appreciation for the work you put into making and sharing this.
For the image segmentation, my videodevicein1 is not working unfortunately.
Only one device can run at a time on my version it seems.

version 2022.32050
windows 10

Webcam initalization assumption

hey dom and torin,

i see an issue where on some machines the webpage would run and show the camera but the prediction-loop is not initialzed. i tracked this down to what i believe is a wrong assumption in your setup() routine. i am not really familiar with js so maybe i'm reading things wrong, but here goes:

• first thing in setup() is opening the websocket
• i assume this also directly attempts to connect to the websocketserver
• if the connection is opened the client sends a list of available cameras
• meanwhile setup() continues with a bunch of stuff and at the end tries to start the webcam and then initializes the prediction-loop but only if the webcam is already running! if the webcam is not running yet at this point, we currently have no chance of initializing the prediction-loop

i wonder if instead, at the end of setup() we don't start the camera but always start the prediction-loop. in the loop we check if the camera is running. so whenever the camera is then started or changed, prediction would start?!

Loading... message into MediaPipe

Hi guys,

Do you know why I get a Loading... message into my MediaPipe?

I reseted also (clicked on pulse) with no effect.
Windows OP:

Version TD: 2023.11220 Experimental

Thanks for your help!
Ana

Endless Loading State

Hi there! Love your content, brightly put and well made!
I am running windows 10, x64 bit system, with the latest version of TD (2022.33910)
and connecting a proper USB Logitech webcam - downloaded the latest version of the plugin Release v0.2.1.
Once launching the MediaPipre TouchDesigner.toe file, all the components shows as expected, but the loading container of MediaPipe stays loading endlessly.
I can turn on and off the parameters, but nothing affects the input of the MediaPipe container, keeping it in loading state.
My computer has no problem recognizing the camera as a Video device Input, and it even shows in the MediaPipe Parameters as a Webcam option (so no webcam detection issues),
and tried it on the MediaPipre website to make sure it worked (even tho i would rather use it through your cool Nodes! haha).
Also, absolutely no red cross appears anywhere, so all good but still loading. Only suspicious thing i found was about the webrowser module in the MediaPipe Container - it appears pitch black. I reckon might be because the toe file has problems connecting the web page that has the tools? Camera privacy issues? Still searching haha
Thanks for your awesome contribution! Looking forward to try it!
Best of luck and your guidance is much appreciated!

Facetracking translate error xForm MATRIX

The tool is incredible.
With my team we are implementing it right now and we are trying to take it to a festival in France for an installation!
///
I'm having problems, despite having followed the whole walkthrough and tutorial, to use facetracking_ to track the full movement of the head. It seems to only take rotation and not translation and position. Attached is a short video of the error I got stuck in.

Thank you in advance for your support!!!
Andrea

facetrack_translate_error1__.mov

OFFLINE mediapipe.tox everytime I re-open the project.

It only happens to me that every time I close a project then when I reopen it MediaPipe.tox is offline with a red cross and also the related tool (facetracking for example)?. Is there a way, maybe saving everything in the same folder, MediaPipe.tox etc. included (?) but I have already tried and every time I restart I have to delete the old nodes and replace them .

Thank u in advantage <3
Love u guys
Andrea

Crop/modify webrender's input

HI Guys,
I would really love to create an interactive app, but I have only one camera as an input device. Therefore I would like to make it split screen, by compositing two mediapipe instances, one for the left half and one for the right half of the screen. It is easy to do this with the output, but I would love to do this with the input too. This way the left half of the screen would detect gestures for only the mentioned half of the input. I can transform webrender's output, but it won't change field of view for gesture detection - which I'd love to crop/mask out. Do you have any idea how could i do this?
thanks a lot,
R

Mediapipe tox is empty

Hello, Im trying your project for the first time (laptop windows 11 rtx 2070 with td version 2022.33910) and I cant get it to work. Followed all the steps (spout,...) and still it seems that the mediapipe tox is empty and dont know why... tried re-init, reimporting the tox, among other things without luck. Here some screenshots.

Thanks in advance,

Joan

Use it with Intel Realsense

Very nice work. Thanks a lot. It brings me a lot forward in my project in which I need the object detection. One Question: Is it possible to connect a Realsense Camera to it and get the distances of the obejcts?

Not Working

Hi guys! First of all, thank you for your work on TD: I have followed you for 2 years on YT.

I have an issue with this plugin. I'm using Windows 10 with the new release built of TD: 2023.10130.
In the dropdown menu for the webcam, it isn't visible the HD Camera of the PC (MSI GP75 Leopard) and also the MediaPipe component presents a red cross:

Inside the component the issue is this:

How can I fix this?

Thanks for your time ;)

Spoutcam not showing up on the Mediapipe container

Hi, thank you for your hard work on this one. I have one issue to report, I do not seem able to find the spoutcam on the drop-down menu of the Mediapipe container (on windows 11). I have registered the Spoutcam as detailed on the instructions page.

Also when I load the spoutcam on the VideoDeviceIn TOP it shows up as just black.

Any help would be appreciated!

EDIT: I figured the error I am getting on the Mediapipe container might be enlightening:

file load version conflict

Hey there! I just wanted to express my gratitude for the amazing work you all have done. I really appreciate the fact that you share it with others. However, I encountered an error while trying to open the project file. Although I accepted the prompt to open the project, the MediaPipe seems to be stuck in the loading state and is not working properly.

OBS Virtual Camera Not Working

I've followed correctly step by step the guide from the video but the virtual camera in TD doesn't show up, here some screenshoots. I've also tried to reset the camera multiple times or change the Websocket Port or the Mediapipe Port by 1 (ex. 3001 to 3002) but still not working. What I can do to fix this problem?
My device: MacBook Pro M1 2020, TD 2022.35320

No info from face_tracking blend_shapes

Hi,

I have this strange issue. Everything is working fine except the face_tracking blend_shape info. It can detect my face, but it doesn't register the blend_shape information, it's not showing the data at least. However, it does show the info from all the other tracking options.

I have attached some screenshots. I'm curious to hear if there is something I'm missing, or something I might not be aware of.

Im using a MacBook Pro M1, 16 GB memory with macOS Sonoma Version 14.0
Touchdesigner 099 Non-Commercial 64-bit build 2023.11170
Mediapipe release v0.3.2

Thanks :)))

Multiple poses not outputting

I'm sure that this isn't a real issue but I can't figure out how multiple poses are meant to work. I have dialed up the pose count from 1 and while media pipe seems to track multiple poses just fine, the output only gives me coordinates for 1 pose at a time, usually flipping back and forth between my pose and the person's next to me.

How is this function meant to work?

Also love these toxs, thank you!

Mediapipe keeps showing connecting

I am trying to use Mediapipe via touchdesigner, but the camera screen keeps showing "loading"

SpoutCam not showing

Hi Torin, Hi Dom, such a great work!!! I have a problem: SpoutCam is not showing in the webcam menu. I have downloaded the newest versions etc. I am on PC windows 10. What can I have done wrong? thank you soooo much!!!!!

Media Pipe Loading Issue

Similar to other issues, I'm getting the endless loading screen of media pipe. I believe I have set everything up correctly, but I am not getting a "SpoutCam" Option in the Media Pipe Webcam dropdown. Some screenshots:

Specs:

TD Version: 2022.33910
MediaPipe TD Release: v0.2.1
OS: Windows 10

Please let me know if I'm leaving any information out. Really excited to get into this, everything looks great!

How to close the green line and red points of the track?

Hi, it is an amazing plung-in for TD, thank for sharing this！
I am a beginner of TD. I created a point following the movement of one finger point，and I wanted to use an over TOP to add the circle TOP above the video of realtime me. I connected the mediapipe directly to the over TOP，but I don't want the grean line and red dot still appearing there. Is there any method to do that excepting add one more camera but to use the realtime video in the mediapipe?

I connect the mediapipe to the over

I want to delet the line and point

Just curious: why use the Javascript version of mediapipe instead of Python?

Hi, I just found this library and first of all, it looks awesome in terms of ease of use!

But since I have been dabbling in MediaPipe a bit and I'm currently trying to decide on the best approach for my project, I'd just be interested in why you chose to go with the JS npm version of MediaPipe for this project. Wouldn't the Python version of mediapipe be somewhat more intuitive or the more obvious option, given TouchDesigner's Python integration? Curious about the thoughts behind this decision and potential downsides you are already aware of. Thank you!

Object detection Touchdesigner bounding boxes not aligned

I noticed while using the object detection portion of the TOX that the bounding boxes that Touchdesginer is drawing do not align with bounding boxes that Mediapipe is drawing. They are offset and don't accurately map back onto the video.

Thanks for putting this resource together, it is a really great tool and addition to touchdesigner!

Is it possible to use a movie file in instead of the web render ?

I wanna try to use your facial detection on a video not the webcam so I wanna know if there is a way to do this ?
Have a nice day

Media Pipe won't find webcams

Hey. I've set up Media Pipe on 2 PC's now. I am trying to set it up on one more, but the MediaPipe tox webcams are updating. I have Spoutcam setup and everything, but the webcams scroll down doesn't change. I've tried a bunch of fixes, but none have worked.

Poor Performance

First of all, thanks for all the wonderful open source work!

Hand tracking works through spoutCam however the performance is quite poor. Perhaps five to seven frames per second. I'm running a high end machine. New threadrippers, 256 gigs of ram, A6000 graphics card. Windows 10 pro Version 10.0.19045 Build 19045, TD 2022.33910.

One thing I am confused about is whether there's a way to track directly from a video device or webcam. I can't get that method to work at all. Only SpoutCam input seems to work.

.tox import issue

I´ve testing mediapipe directly from the MediaPipe TouchDesigner.toe file, and it work just nice.
but when I open a new TD project and try to import any of the .tox files from mediapipe it just doesnt import, I´ve tried to import and drag n´drop them, but they doesnt show at my workspace, it doesnt show a popup or warning neither.
Maybe someone have an alternative way of import them?

2024-01-22.11-05-31.mp4

Incorrect MimeType preventing javascript from loading on some machines

Hey there!

I was testing on a few different computers and noticed that on some PCs Mediapipe wouldn't load.

From taking a look at the webpage I saw that this error was occurring,

Failed to load module script: Expected a JavaScript module script but the server responded with a MIME type of "text/plain". Strict MIME type checking is enforced for module scripts per HTML spec.

I went into the Mediapipe tox and added these lines of code in the mp_webserver_callbacks dat.

if fileName.endswith('.js'): mimeType = ['application/javascript'] else: mimeType = mimetypes.guess_type(fileName, strict=False)

This solved the issue for me and allowed the TOX to function correctly.

mediapip do not working no image

windos10 rtx3060 laptop

SpoutCam not working in Mediapipe tox

Hi Torin and Dom, Thanks for all amazing tool. Making it easy for beginners to get started.
I have some problems while using webcam with SpoutCam. I selected 'webcam' in videofilein TOP, and selected 'SpoutCam' in the Mediapipe tox. Both of the resolutions are set to 1280 x 720. However, the Mediapipe window shows RGB noise.

When I use laptop cam and obs in Mediapipe tox, it is successful. Using the webcam, Mediapipe window have image but not working.

I also tried changing the port to 3002, but not working.

I found an interesting part: when I select 'SpoutCam' in Mediapipe tox, window shows RGB noise. However, when I open videofilein TOP and also select 'SpoutCam', videofilein TOP shows the image.

These are the images that I tried loading http://localhost:9222/.
Select 'webcam' in Mediapipe tox

Select 'SpoutCam' in Mediapipe tox

TD Version：2022.33910, 2022.35280
MediaPipe TD Release：v0.2.2
System：Windows 11

If there's anything I haven't noticed or done wrong, please let me know! Thank you for your guidance!

detecting 2 hands only

Hi, I'm trying to detect more than 2 hands, but it's always detecting maximum 2 hands. If I set the hand count to 1 it still detects 2, tried setting it to detect 4 or 6 but still detecting 2.

Using Windows 11 Enterprise version 22H2 OS build 22621.2428.

Any suggestions?

Thanks

Mac 2020 m1 webcam

T.T

RTSP Feed instead of using Webcam?

Hello Torin,

Awesome work on this repo. I'm trying to figure out how it would be possible to modify the code to enable RTSP feed input instead of a webcam.

The reason, I'm trying to figure this out is because, I realize that when needing multiple types of data, faces, landmarks, and so on, we get better results by running multiple MediaPipe bases for marginal performance costs. I'm using NDI webcams instead of Spout right now, because it allows me to share the same feed over 4 MediaPipe bases on the same machine, but I'm limited by the 4 NDI Cam sources.

I believe that with a multicast RTSP it would be possible to use the same source across multiple instances of MediaPipe bases.

Use Chrome Desktop Capture as video source?

This looks awesome! I'm excited to try your project. I've been experimenting with mediapipe js, but have never used TouchDesigner.

I want to recommend the Chrome Desktop Capture as another option for bringing video into the mediapipe-touchdesigner.

In this repo I used Chrome's Desktop Capture as the video input source for landmark detection.
OBS-face-tracking-with-P5js

In this setup a webcam source is created in OBS, then a projector window is opened on that source. Chrome is used to capture the OBS projector window. Landmark data is sent from Chrome to OBS via websockets.

Using NDI, Spout/Syphon or OBS virtual camera accomplish the same purpose, but this options may be useful too.

Thanks for sharing this project!

Not saving anything I've added

When I save the file, in the same folder of release with toxes, it does not save anything I've worked or I've added when I open the file. If I open the file in other folders(files from the backup folder), it still contains all the work I've done but it's having a connecting issue loading the necessary files (I'm guessing toxes files need to be in the same folder?) I tried moving the same file(from one of the backups) which had all the data in it to the file where the toxes folder is(to the folder in the screenshot) and when I open the file, all the work disappears... How can I fix this?

I can't find my SpoutCam

Really like your content, it is very useful to me, but I encountered some problems when I use.
I am running windows 10, x64 bit system, with the version of TD (2022.33600)
I can load it in TouchDesigner TDSyphonSpoutOutTOP,but MediaPipe Container report an error.

Infinite Loading

I have an infinite loading screen -- not sure where to start troubleshooting. My camera is live in the input. I tried incrementing the mediapipe port and pulsing, no joy. Thoughts?

Web console:

Webcam picture visible, but models not working

Hello,
thank you for this fantastic project first of all.

I've had the MediaPipe TouchDesigner.toe working for a short moment, but then changed the number of faces to detect, as suggested in the tutorial video. Ever since it does not work anymore. It shows the raw webcam picture in the MediaPipe component but it does not seem to be processed further. Tried restarting, loading again, etc.

Windows 11, TouchDesigner 2023.11290, Release 0.3.3, Logitech BRIO webcam

Thanks for any hints!

Loads up but doesnt work

I tried all the troubleshooting methods I could find, but none of them seemed to help. My TD version is 2022.33910. I registered & unregistered on Spoutcam as directed in the tut as well. I also tried changing the port to 3002, but no good. My webcam and internet connection are fine. I tried checking Spoutcam on the Mediapipe website and it worked fine. My starting res is 1280 x 720 as mentioned.

I hope the screenshots help you find out the issue. Really excited to use the plugin.

thanks!

"Webcam failed to start. Is it being used somewhere else? Input must also be capable of 720p, 8 bit."

Encountering error "Webcam failed to start. Is it being used somewhere else? Input must also be capable of 720p, 8 bit." with SpoutCam The footage I am using is 720p, and I cant see any other app using SpoutCam.

Windows 11
TouchDesigner 202311340
Release v0.4.2

Camera Feed

Hi, I can't wait to use this, except my Mac camera feed isn't showing on the media pipe when I select FaceTime HD camera. I've tried the pulse button and made sure that my Mac allows Touch Designer in the security settings. Please advise as to what I should do next. I have a feeling TD is trying to choose the webcam OBS virtual and T's phone. Neither of which Im familiar with.

Thanks

facetracking - faceMask vs transformmatrix

Hi,
in you're example i'm trying to match the white faceMask to the canonical_face_model connected to the transformmatrix you put on your example.
rotation, ... seems to be ok. But the Z (zoom in / out with head distance from camera. When i'm closed to the camera the canonical_face_model is smaller and bigger when i'm far for the camera.

Any idea ?

My idea is to make a "kind of snapchat filter" with a 3D head oculter that follow the head size.

Thanks

endless loading

Hi, thanks for all this work. Unfortunately I can't get the Spoutcam option in the Mediapipe container as webcam. When using the moviefile in TOP, and no webcam, also nothing happens.
I found this warning:

face tracking sample issues

Hi,
First, thank you for your great works! It helps me a lot.

I have an issue about the face tracking sample.
It can detect my face, the face part and the texture part work well. However, the halo part does not work. It doesn’t move with my face and show nothing. I'm curious to hear if there is something I'm missing.

(Also, the same problem with others, it does show the info from all the other tracking options.)

(By the way, I want to know how to change the resolution of the camera? I’m using the FaceTime HD camera of Mac, it shows error message but seems working to show my face. Do I need to change?)

I have attached the screenshot.

——————————————
MacBook Pro (Intel, 2020) with macOS BigSur 11.7.10

Touchdesigner 099 build 2022.35320

Mediapipe release v0.3.2
——————————————

Thank u sooo much.

Hands "Max Number of Hands" selection not working

Despite setting Max Number of Hands param to 1, both hands still detected and sending data to CHOPs.

Disregard

Disregard, as it was a configuration issue.

Top input

Could it support other TOP input, not just limited to using a webcam?
Thanks