This is a Unity (2021.3.18f1) Native Plugin to use MediaPipe (0.10.1).
The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.
With this plugin, you can
- Write MediaPipe code in C#.
- Run MediaPipe's official solution on Unity.
- Run your custom
Calculator
andCalculatorGraph
on Unity.โ ๏ธ Depending on the type of input/output, you may need to write C++ code.
Here is a Hello World! example.
Compare it with the official code!
using Mediapipe;
using UnityEngine;
public sealed class HelloWorld : MonoBehaviour
{
private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
calculator: ""PassThroughCalculator""
input_stream: ""in""
output_stream: ""out1""
}
node {
calculator: ""PassThroughCalculator""
input_stream: ""out1""
output_stream: ""out""
}
";
private void Start()
{
var graph = new CalculatorGraph(_ConfigText);
var poller = graph.AddOutputStreamPoller<string>("out").Value();
graph.StartRun().AssertOk();
for (var i = 0; i < 10; i++)
{
graph.AddPacketToInputStream("in", new StringPacket("Hello World!", new Timestamp(i))).AssertOk();
}
graph.CloseInputStream("in").AssertOk();
var packet = new StringPacket();
while (poller.Next(packet))
{
Debug.Log(packet.Get());
}
graph.WaitUntilDone().AssertOk();
}
}
For more detailed usage, see the API Overview page or the tutorial on the Getting Started page.
This repository does not contain required libraries (e.g. libmediapipe_c.so
, Google.Protobuf.dll
, etc).
You can download them from the release page instead.
file | contents |
---|---|
MediaPipeUnityPlugin-all.zip |
All the source code with required libraries. If you need to run sample scenes on your mobile devices, prefer this. |
com.github.homuler.mediapipe-*.tgz |
A tarball package |
MediaPipeUnityPlugin.*.unitypackage |
A .unitypackage file |
If you want to customize the package or minify the package size, you need to build them by yourself.
For a step-by-step guide, please refer to the Installation Guide on Wiki.
You can also make use of the Package Workflow on Github Actions after forking this repository.
โ ๏ธ libraries that can be built differ depending on your environment.
โ ๏ธ GPU mode is not supported on macOS and Windows.
Editor | Linux (x86_64) | macOS (x86_64) | macOS (ARM64) | Windows (x86_64) | Android | iOS | WebGL | |
---|---|---|---|---|---|---|---|---|
Linux (AMD64) 1 | โ๏ธ | โ๏ธ | โ๏ธ | |||||
Intel Mac | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | ||||
M1 Mac | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | ||||
Windows 10/11 (AMD64) 2 | โ๏ธ | โ๏ธ | โ๏ธ |
Here is a list of solutions that you can try in the sample app.
๐ The graphs you can run are not limited to the ones in this list.
Android | iOS | Linux (GPU) | Linux (CPU) | macOS (CPU) | Windows (CPU) | WebGL | |
---|---|---|---|---|---|---|---|
Face Detection | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Face Mesh | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Iris | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Hands | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Pose | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Holistic | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Selfie Segmentation | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Hair Segmentation | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | |
Object Detection | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ | โ๏ธ |
Select Mediapipe/Samples/Scenes/Start Scene
and play.
If you've built native libraries for CPU (i.e. --desktop cpu
), select CPU
for inference mode from the Inspector Window.
Make sure that you select GPU
for inference mode before building the app, because CPU
inference mode is not supported currently.
https://github.com/homuler/MediaPipeUnityPlugin/wiki
Note that some files are distributed under other licenses.
- MediaPipe (Apache Licence 2.0)
- emscripten (MIT)
third_party/mediapipe_emscripten_patch.diff
contains code copied from emscripten
- FontAwesome (LICENSE)
- Sample scenes use Font Awesome fonts
See also Third Party Notices.md.
Footnotes
-
Tested on Arch Linux. โฉ
-
Running MediaPipe on Windows is experimental. โฉ