Coder Social home page Coder Social logo

mediapipe-osc's Introduction

MediaPipe OSC

MediaPipe examples which stream their detections over OSC to be used in other applications.

Install & Run

Currently this is only tested on Windows and MacOS. It's recommended to use Python3 (>3.7) and a virtual environment.

python install -r requirements.txt

To run an example use the basic python command to start up the script.

# start pose detection with webcam 0
python pose.py --input 0

# start pose detection with video
python pose.py --input yoga.mp4

Other parameters are documented in the following list or algorithm specific.

  • input - The video input path or video camera id (default 0)
  • min-detection-confidence - Minimum confidence value ([0.0, 1.0]) for the detection to be considered successful. (default 0.5)
  • min-tracking-confidence - Minimum confidence value ([0.0, 1.0]) to be considered tracked successfully. (default 0.5)
  • ip - OSC ip address to send to (default 127.0.0.1)
  • port - OSC port to send to (default 7500)

Full-Body Pose Landmark Model (BlazePose Tracker)

The landmark model currently included in MediaPipe Pose predicts the location of 33 full-body landmarks (see figure below), each with (x, y, z, visibility). Note that the z value should be discarded as the model is currently not fully trained to predict depth, but this is something we have on the roadmap.

Pose Description

Reference: mediapipe/solutions/pose

Additional Parameters

--model-complexity MODEL_COMPLEXITY
                      Set model complexity (0=Light, 1=Full, 2=Heavy).
--no-smooth-landmarks
                      Disable landmark smoothing.
--static-image-mode   Enables static image mode.

Format

  • count - Indicates how many poses are detected (currently only 0 or 1)
  • list of landmarks (33 per pose) (if pose has been detected)
    • x - X-Position of the landmark
    • y - Y-Position of the landmark
    • z - Z-Position of the landmark
    • visibility - Visibility of the landmark
/mediapipe/pose [count, x, y, z, visibility, x, y, z, visibility ...]

Hand Detection

The hand detection model is able to detect and track 21 3D landmarks.

Format

  • count - Indicates how many hands are detected
  • list of landmarks (21 per hand) (if hands has been detected)
    • x - X-Position of the landmark
    • y - Y-Position of the landmark
    • z - Z-Position of the landmark
    • visibility - Visibility of the landmark
/mediapipe/hands [count, x, y, z, visibility, x, y, z, visibility ...]

Face Detection

The face detection model is able to detect multiple faces and 5 keypoints. At the moment only the bounding box is sent over OSC.

Format

All values are normalized to the image width and height.

  • count - Indicates how many faces are detected
  • list of one bounding box per face (if faces has been detected)
    • xmin - X-Position of the top-left bounding box anchor
    • ymin - Y-Position of the top-left bounding box anchor
    • width - Width of the bounding box
    • height - Height of the bounding box
    • score - Confidence score of the bounding box
/mediapipe/faces [count, xmin, ymin, width, height, score, xmin, ymin, width, height, score ...]

Face Mesh

tbd

Examples

Currently, there are very basic receiver examples for processing. Check out the examples folder.

About

mediapipe-osc's People

Contributors

cansik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

mediapipe-osc's Issues

LICENSE?

Hi,

I was wondering what license this is?

Trying to make something just as remixable as MIT license.

... how do i know which data mean what ... ?

Hello Sir,
well, this really seems to work, thumbs up !!!
I can get the OSC-data into my musical environment.
Alone, the data are pretty anonymous, so i don't know how to route the arguments to the right address.
Is there any "path" you can recommend how to know which data mean what ?
Sry for me being so clumsy ...,, best regards !
P.S.: Your program works fine with dataclasses0.6; version 0.8 doesn't seem to be supported ...

formatting OSC message hands.py

Hi, this seems to work pretty well! I'm sending the OSC message to Wekinator, but it seems that the incoming OSC messages misses some values...

  1. it seems it doesnt count the hands ((i always get a 0 value, also with both hands)
  2. it doenst give a value for the visibility

Example of the message [0, 0.37387, 0.66106, 0, 0, 0.35562, 043452, 0.27642, 0, etc]

Does anyone know what's going on? I've seen there have been some recent updates in the landmark framework from Mediapipe. Could this be a problem?

Thanks in advance

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.