cansik / mediapipe-osc

MediaPipe examples which stream their detections over OSC.

MediaPipe OSC

MediaPipe examples which stream their detections over OSC to be used in other applications.

Install & Run

Currently this is only tested on Windows and MacOS. It's recommended to use Python3 (>3.7) and a virtual environment.

python install -r requirements.txt

To run an example use the basic python command to start up the script.

# start pose detection with webcam 0
python pose.py --input 0

# start pose detection with video
python pose.py --input yoga.mp4

Other parameters are documented in the following list or algorithm specific.

• input - The video input path or video camera id (default 0)
• min-detection-confidence - Minimum confidence value ([0.0, 1.0]) for the detection to be considered successful. (default 0.5)
• min-tracking-confidence - Minimum confidence value ([0.0, 1.0]) to be considered tracked successfully. (default 0.5)
• ip - OSC ip address to send to (default 127.0.0.1)
• port - OSC port to send to (default 7500)

Full-Body Pose Landmark Model (BlazePose Tracker)

The landmark model currently included in MediaPipe Pose predicts the location of 33 full-body landmarks (see figure below), each with (x, y, z, visibility). Note that the z value should be discarded as the model is currently not fully trained to predict depth, but this is something we have on the roadmap.

Reference: mediapipe/solutions/pose

Additional Parameters

--model-complexity MODEL_COMPLEXITY
                      Set model complexity (0=Light, 1=Full, 2=Heavy).
--no-smooth-landmarks
                      Disable landmark smoothing.
--static-image-mode   Enables static image mode.

Format

• count - Indicates how many poses are detected (currently only 0 or 1)
• list of landmarks (33 per pose) (if pose has been detected)
  • x - X-Position of the landmark
  • y - Y-Position of the landmark
  • z - Z-Position of the landmark
  • visibility - Visibility of the landmark

/mediapipe/pose [count, x, y, z, visibility, x, y, z, visibility ...]

Hand Detection

The hand detection model is able to detect and track 21 3D landmarks.

Format

• count - Indicates how many hands are detected
• list of landmarks (21 per hand) (if hands has been detected)
  • x - X-Position of the landmark
  • y - Y-Position of the landmark
  • z - Z-Position of the landmark
  • visibility - Visibility of the landmark

/mediapipe/hands [count, x, y, z, visibility, x, y, z, visibility ...]

Face Detection

The face detection model is able to detect multiple faces and 5 keypoints. At the moment only the bounding box is sent over OSC.

Format

All values are normalized to the image width and height.

• count - Indicates how many faces are detected
• list of one bounding box per face (if faces has been detected)
  • xmin - X-Position of the top-left bounding box anchor
  • ymin - Y-Position of the top-left bounding box anchor
  • width - Width of the bounding box
  • height - Height of the bounding box
  • score - Confidence score of the bounding box

/mediapipe/faces [count, xmin, ymin, width, height, score, xmin, ymin, width, height, score ...]

Face Mesh

tbd

Examples

Currently, there are very basic receiver examples for processing. Check out the examples folder.

About

• Example code and documentation adapted from google/mediapipe
• OSC sending and examples implemented by cansik

mediapipe-osc's Issues

FaceMesh

Adjusted the hands.py file to fit the facemesh. Bit of overwellming data, but is works.
This repository helps with navigating.
https://github.com/ManuelTS/augmentedFaceMeshIndices

facemeshOSC.zip

... how do i know which data mean what ... ?

Hello Sir,
well, this really seems to work, thumbs up !!!
I can get the OSC-data into my musical environment.
Alone, the data are pretty anonymous, so i don't know how to route the arguments to the right address.
Is there any "path" you can recommend how to know which data mean what ?
Sry for me being so clumsy ...,, best regards !
P.S.: Your program works fine with dataclasses0.6; version 0.8 doesn't seem to be supported ...

No matching distribution found for dataclasses==0.8

pip install of requirements.txt failed with this error above on Windows 10 64-Bit with Python 3.9.4

formatting OSC message hands.py

Hi, this seems to work pretty well! I'm sending the OSC message to Wekinator, but it seems that the incoming OSC messages misses some values...

1. it seems it doesnt count the hands ((i always get a 0 value, also with both hands)
2. it doenst give a value for the visibility

Example of the message [0, 0.37387, 0.66106, 0, 0, 0.35562, 043452, 0.27642, 0, etc]

Does anyone know what's going on? I've seen there have been some recent updates in the landmark framework from Mediapipe. Could this be a problem?

Thanks in advance

LICENSE?

Hi,

I was wondering what license this is?

Trying to make something just as remixable as MIT license.