keijiro / bodypixbarracuda

BodyPix for Unity Barracuda

BodyPixBarracuda

BodyPixBarracuda is an implementation of the BodyPix person segmentation and pose estimation model that runs on the Unity Barracuda neural network inference library.

System requirements

• Unity 2020.3 LTS or later

About the ONNX file

I converted the original BodyPix model (provided as tfjs) into ONNX using tfjs-to-tf and tf2onnx. See the Colab notebook for further details.

ResNet support

This package supports the ResNet architecture (more accurate but slower and bigger models) but doesn't contain those ONNX files due to the file size limit of GitHub and npm.js. You can download them from here instead.

To use those models, create a new BodyPix ResourceSet file and set the model, architecture, and stride fields accordingly.

How to install

This package uses the scoped registry feature to resolve package dependencies. Open the Package Manager page in the Project Settings window and add the following entry to the Scoped Registries list:

• Name: Keijiro
• URL: https://registry.npmjs.com
• Scope: jp.keijiro

Now you can install the package from My Registries page in the Package Manager window.

bodypixbarracuda's Issues

Max Person Detection

Excuse me,
Do you know how to limit the max person segmentation,
I just want single person segmentation.
Thank you.

Extremely low framerate

Running the project as is, or with my own image or video, runs at about 5 frames/sec and video lags at about 0.5 f/s.

Using unity 2021.1.11f1

Multi-person support

• Segmentation filter: It should segment individuals separately.
• Keypoint detect: It should provide a list of poses for multiple people. Currently, it only returns a single pose.

Getting started with Colored Masks

Apologizes for the ignorant question. I'm very new to Unity.

I'm trying to implement the Colored Mask body segmentation in Unity. After installing the BodyPixBarracuda as per your README, what are the next steps?

I've tried your sample project and everything works perfectly. I'm just not sure how to change things to fit my needs.

Any help would be appreciated.

Thanks

how to access keypoints data

sorry for the noob question, but I couldn't figure it out by myself.
How can i read x and y properties for the nose, ears, etc?

mobile device support

I tried to run it on a mobile device and it is very slow, is there a way to decrease computation requirement?

IOS Release build Example

Is this possible, I can not override barracuda and burst to not use the preview packages

Request Pytorch source code

The existing model is too poor, and there is no Pytorch source code, which is no different from having no examples at all. Please request the Pytorch source code so that I can train it myself.

(Request) Sentis support and updated models

Hey, firstly thanks for the effort you've put into this project - it works great!

With Unity set to phase out Barracuda in favour of Sentis, are there currently any plans to update this project with Sentis support? As I said, it's a great asset, so it'd be a shame if it was rendered unusable by Unity's shift to Sentis.

Also, I was wondering if there are any plans to include updated or new models in the project? If not, do you have any advice for how I could adapt the project to support more models? I notice that some OBS plugins for example use things like RVM MobileNetV3 model and MediaPipe, among others, so I'm somewhat interested in how those models would perform in Unity.

Thanks again!

Does RIght Face include a part of RIght Arm in Body Part Segmentation?

I'm thinking of making a mask for the head only, but I think it includes the side of the left arm. Is this the specification of the bodypix model?

how much miliseconds takes execution ?

hi. how much miliseconds takes the pose estimation?. could it be run on mobile ? if so how much fps?

Thanks.