tucan9389 / poseestimation-tfliteswift

The baseline project for inferencing various Pose Estimation tflite models with TFLiteSwift on iOS

This project is Pose Estimation on iOS with TensorFlow Lite.
If you are interested in iOS + Machine Learning, visit here you can see various DEMOs.

2D pose estimation in real-time	3D pose estimation

Features

• Support 2D pose estimaiton TFLite models
  • Real-time demo with Metal framwork
  • Photo album demo
• Support 3D pose estimation TFLite model
  • Real-time demo with Metal framwork (but realtime model is not ready yet)
  • Real-time pose matching demo
  • Photo album demo
• Render the result keypoints of 2D pose estimation in 2D demo page
• Render the result keypoints of 3D pose estimation with SceneKit
• Render the heatmaps of 2D pose estimation output
  • Part Confidence Maps for typical heatmap based models
  • Part Affinity Fields for OpenPose (2D multi-person)
• Implemented pose-matching with cosine similiarity in 3D pose demo
• Implemented to fix the shoulders' keypoints in 3D pose estimation to pre-process for pose-matching

Models

Source Images

Name	gif	img-0	img-1	img-2
Source	-

Joint Samples

Model Names	gif	img-0	img-1	img-2
PoseNet	-
PEFM CPM	-
PEFM Hourglass	-
OpenPose (multi-person)	-

Heatmap-ConfidenceMap Samples

Model Names	gif	img-0	img-1	img-2
PoseNet	-	-	-	-
PEFM CPM	-
PEFM Hourglass	-
OpenPose (multi-person)	-

Heatmap-PAF Samples

Model Names	gif	img-0	img-1	img-2
OpenPose (PAF x)	-
OpenPose (PAF y)	-

Meta Data

2D

✅ vs ☑️	Name	Size	Inference Time on iPhone11Pro	Post-process Time on iPhone11Pro	PCKh-0.5	multi person vs single person	Model Source	Paper	tflite download
✅	PoseNet	13.3 MB	-	-	-	single	tensorflow/tensorflow	PersonLab
✅	PEFM CPM	2.4 MB	-	-	-	single	edvardHua/PoseEstimationForMobile	Convolutional Pose Machines
✅	PEFM Hourglass v1	1.8 MB	-	-	-	single	edvardHua/PoseEstimationForMobile	Stacked Hourglass Networks
✅	PEFM Hourglass v2	1.7 MB	-	-	-	single	edvardHua/PoseEstimationForMobile	Stacked Hourglass Networks
✅	OpenPose	7.8 MB	-	-	-	multi	ildoonet/tf-pose-estimation	OpenPose
☑️	AlphaPose	-	-	-	-	single	osmr/imgclsmob	RMPE
☑️	SelecSLS	-	-	-	-	single	osmr/imgclsmob	-
☑️	IBPPose	-	-	-	-	single	osmr/imgclsmob	-
☑️	Lightweight OpenPose	-	-	-	-	single	osmr/imgclsmob	OpenPose

3D

✅ vs ☑️	Name	Size	Inference Time on iPhone11Pro	Post-process Time on iPhone11Pro	(metric)	Model Source	Paper	tflite download
✅	Baseline3DPose	137.1 MB	347 ms	79 ms	-	mks0601/3DMPPE_POSENET_RELEASE	Baseline3D	download
✅	LiteBaseline3DPose	16.6 MB	116 ms (cpu only)	19 ms (cpu only)		SangbumChoi/MobileHumanPose	MHP	download

Requirements

• Xcode 11.3+
• iOS 11.0+
• Swift 5
• CocoaPods

gem install cocoapods

Build & Run

1. Install dependencies with cocoapods

cd ~/{PROJECT_PATH}
pod install

2. Open the PoseEstimation-TFLiteSwift.xcworkspace and run the project

See also

• motlabs/awesome-ml-demos-with-ios
• TensorFlow Lite or Tensorflow models provided by:
  • CPM and Hourglass model provided by edvardHua/PoseEstimationForMobile
  • PoseNet model provided by tensorflow/examples
  • OpenPose model provided by ildoonet/tf-pose-estimation
  • Various model provided by osmr/imgclsmob
  • 3DMPPE PoseNet model provided by mks0601/3DMPPE_POSENET_RELEASE
• Pose estimation with Core ML - tucan9389/PoseEstimation-CoreML

License

This repository is licensed under Apache 2.0. Full license text is available in LICENSE.

poseestimation-tfliteswift's Issues

Highlight the line when the line's similarity is upper than threshold in 3d pose matching demo page

Baseline3DPoseEstimator returns only 14 points, not 18 points.

I have cloned this repo and tried 3D pose estimation using Baseline3DPoseEstimator, not LightBaseline3DPoseEstimator.
I have noticed that only 14 points are displayed on the screen. I mean from Pelvis to L_Wrist. As a result, right half of the human body is not drawn on the screen, while they are drawn on gifs on the REAME, and they are defined in types on the code.
I just thought this is dues to vagueness of bitmaps I input to the model, but whatever images I give to the model, it returns 14 points, which I have verified using print function.
I have read through softArgmax3D function, but I don't understand mathematical intricacies of the function, so it might be the problem.
Does anyone know how to solve this problem?

Support OpenPose tflite model (multi-pose)

Post-process Logic

• 1. Get points from confidence map (make nodes) #38
• 2. Make a bipartite graph with each part of points (make edges) #42
• 3. Calculate line integral with PAF (compute costs of each edge) #42
• 4. Connect each point and make human (solve assignment) #42
• 5. Merge if a point is used duplicated in multiple human #42

Checklist

• Add parameter control UIs for PAF #47
• Limit the number of human on post-process #48
• Test other model

Reference

• https://arvrjourney.com/human-pose-estimation-using-openpose-with-tensorflow-part-2-e78ab9104fc8
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• https://github.com/ildoonet/tf-pose-estimation

Add gif demo on README for PoseNet

Still Image

screenshot1	screenshot2	screenshot3	screenshot4

Live Image

Content Source

• http://www.franmenez.com/fotografo-en-granada-fotografia-creativa-en-granada-fotografo-diferente-en-granada-fotografo-break-dance-en-granada/
• https://www.instagram.com/p/B9cIyjsnihN/
• https://youtu.be/OrOZxOTIv-g

Support framework for each model

• Cocoapods
• Carthage
• Swift Package Manager
• Manual Framework

Calculate the inference time of each model

Dear author,
May I ask you how to calculate the inference (the inference time and pose process time) time of each 3D human pose model as you showed?
Thank you so much!

Control threshold with slider

EXC_BAD_ACCESS (code=1, address=0x14fdff900) during scale operation on cropped image

The app sometimes crashes during the scale operation in vImageScale_ARGB8888(&croppedImage, &resizedImage, nil, vImage_Flags(0)) == kvImageNoError

Drawing heatmap on overlay view for debug

• Drawing heatmap for confidence map
• Drawing heatmap for PAF x-axis
• Drawing heatmap for PAF y-axis

can not find the file lightweight_baseline_choi.tflite

Hi, I was trying to run your application, but I cant seem to find lightweight_baseline_choi.tflite model file, can you please add the link for downloading.

Implement benchmark on real device (CPU, GPU, NPU)

Dataset

• with coco 2017 test set (http://cocodataset.org/#download)

Metric

• PCKH-0.5 (the most popular metric in paperswithcode/pose-estimation-on-mpii-human-pose)
  • implement reference: edvardHua/PoseEstimationForMobile's benchmark.py

Define

ref: https://nanonets.com/blog/human-pose-estimation-2d-guide/

Percentage of Correct Key-points - PCK: A detected joint is considered correct if the distance between the predicted and the true joint is within a certain threshold. The threshold can either be:

• [email protected] is when the threshold = 50% of the head bone link
  ...

Grouping issue on multi pose estimation (OpenPose)

Summary

• Nose-REye and Nose-LEye matching is incorrect in the general situation

Samples

case 1	case 2	case 3	case 4

case 5	case 6	case 7	case 8

Connection to assetsd was interrupted - assetsd exited, died, or closed the photo library Message from debugger: Terminated due to memory issue

hello,

I am getting this error on iPhone8. No apps running on background. I think the msg is clear but any tips to make is work ? maybe load less models ?

Connection to assetsd was interrupted - assetsd exited, died, or closed the photo library
Message from debugger: Terminated due to memory issue

any idea ?

Support SimpleBaselinePoseEsitmation tflite model (3D-pose)

Reference

2D

• https://github.com/tucan9389/tf2-mobile-pose-estimation
• https://github.com/mks0601/TF-SimpleHumanPose

3D

• https://github.com/una-dinosauria/3d-pose-baseline
• https://github.com/SangbumChoi/MobileHumanPose

.tflite files are missing

I am trying to run the project in Xcode 11.4.1 but all the .tflite files are missing under imgclsmob-Pose folder in Xcode. Anyone, please help me out with this. I want to compile the project.

Thanks

Inference with a still image

Purpose

Implement to import an image and inference

Tasks

• Create tabbar for separating live captured image inference - #11
• Get an image from device's photo album - #14
• Inference the image - #25
• Draw the result of inference - #25

Implement pose matching

Already implement in PoseEstimation-CoreML repo. I'll copy and paste the logic to this repo.

Reference

https://github.com/tucan9389/PoseEstimation-CoreML

Cannot inference with GPU on simulator (Xcode Version 11.4.1 (11E503a))

Summary

Couldn't run the model with .metal option on iPhone simulator after updating my Xcode to 11.4.1

Environment

• macOS: Version 10.15.4 (19E287)
• Xcode: Version 11.4.1 (11E503a)
• TensorFlowLiteSwift: 2.1.0

How to solve

• If you want to run on the simulator, change the accelerator default param to .cpu (163 line on TFLiteImageInterpreter.swift)

Support OpenPose tflite model (single-pose)

Reference

https://github.com/ildoonet/tf-pose-estimation/issues/355

Separate each post-process step for multi pose estimation

Related Comment

• #42 (comment)

Summary

There are 5 steps on multi-pose estimation post-process. NMS step is separated as a class, but other steps aren't.

Draw the keypoint result on view

Responsibility of each layer

There are three layers for using tflite pose estimation model:

• TFLiteImageInterpretor: the wrapper class for image preprocessing before inference
• PoseEstimator: a simple protocol for abstraction
• PoseNetPoseEstimator: concrete implement for PoseNet model

TFLiteImageInterpretor

TFLiteImageInterpretor is similar to VNCoreMLRequest. It has some pre-process logic for image and a tflite interpretor. Only TFLiteImageInterpretor class has a dependency with TensorFlowLiteSwift.

PoseEstimator

A simple protocol for abstraction. Users can create the concrete type conforming to PoseEstimator protocol and then use the estimator instance with the protocol, not concrete type. So except for the creation point, there is no use of the concrete type.

PoseNetPoseEstimator

PoseNetPoseEstimator is concrete type of PoseEstimator. It has PoseNet model specific information like input/output shape and normalization type.
For example, tensorflow/examples's PoseNet model needs following information:

• input: [1, 257, 257, 3]
• output: [1, 9, 9, 17]
• input range: 0.0~1.0
• input type: float32
• quantized: false

TODO

• preprocess
• inference
• postprocess

Measure process, inference time and show in live

Inference with live image captured by device's camera

Support PoseNet tflite model

Model Infos

• source: https://www.tensorflow.org/lite/models/pose_estimation/overview
• file name: posenet_mobilenet_v1_100_257x257_multi_kpt_stripped.tflite
• file size: 13.3 MB
• input: [1, 257, 257, 3]
  • rgb color image
  • need normalized input 0.0~1.0
• outputs: 4 output - postprocess in iOS
  • outputs[0]: [1, 9, 9, 17]
    • 17 heatmaps (because 17 keypoints output)
  • outputs[1]: [1, 9, 9, 34]
    • 17 * 2 offsets (because col and row offsets output) → as input size
  • outputs[2]: [1, 9, 9, 32] - ??
  • outputs[3]: [1, 9, 9, 32] - ??

Benchmark

mAP

Preparing

Processing Time

Preparing

How it performs

https://www.tensorflow.org/lite/models/pose_estimation/overview

Pose capturing and matching

• capture 3d pose by tapping a button or listening to "capture"
  • speech recognition: https://github.com/jen2/Speech-Recognition-Demo
• match the inferred pose and captured pose
• show the matching score and highlight which captured pose is matched

lightweight_baseline_choi model shape issues

Hi ,

I tried my own trained model, but the output shape of my model is 1 x 21 x 3, but the model that you gave has an output shape of 1 x 672 x 32 x 32. I trained the model using https://github.com/SangbumChoi/MobileHumanPose repo. Could you please tell me why is this issue happening?

Thanks and Regards,
Ashish

any models trained on Leeds Sports Pose Dataset ?

thanks

Update README for v1.1.0

• Add gif demos for each model (1 image per model)
  • Video: <#TODO#>
• Add png demos for each model (3~4 images per model)
  • Image1: <#TODO#>
  • Image2: <#TODO#>
  • Image3: <#TODO#>
  • Image4: <#TODO#>

Package.swift has no Package.swift manifest for version 0.2.8

I try to build the project locally but have issues with the TFLiteSwift-Vision dependeny.
I get the error:

No such module 'TFLiteSwift_Vision'

and

/Package.swift has no Package.swift manifest for version 0.2.8 in https://github.com/tucan9389/TFLiteSwift-Vision

Using Xcode 13.4.1.

Any ideas what I am doing wrong?

Convert imgclsmob model to tflite

Building project errors

Hi trying to build this example and found this list of build-time errors

Support shoulder auto-adjustment in pose matching

Goal

• For robust pose matching algorithm, implement pre-processing before cosine similarity

How to make it

1. Adjust predicted pose keypoints by captured right and left shoulder's keypoint
   • transform the predicted keypoints as same shoulder keypoints coordinate between predicted and captured

logo

Use Accelerate framework on post-processing

Editing Point

https://github.com/tucan9389/PoseEstimation-https://github.com/tucan9389/PoseEstimation-TFLiteSwift/blob/da2931e39eea3065d72d4c83383387bbef482bc0/PoseEstimation-TFLiteSwift/Argmax.swift#L25-L41

Reference

https://github.com/hollance/CoreMLHelpers/blob/0bd7f983c3e73abcffb692b4428e889ed4cf01b6/CoreMLHelpers/Math.swift#L23-L82

Support multi-person pose estimator

baseline_moon_noS.tflite

Hello. Could u plz share baseline_moon_noS.tflite?

LiteBaseline3DPose No such file

Hello, I am having trouble finding these files.

PoseEstimation-TFLiteSwift/PoseEstimation-TFLiteSwift/LiteBaseline3DPose/litebaseline_3dpose_real_S.tflite: No such file or directory

PoseEstimation-TFLiteSwift/PoseEstimation-TFLiteSwift/LiteBaseline3DPose/litebaseline_3dpose_real_L.tflite: No such file or directory

I found the 'imgclsmob-Pose' and the 'Baseline3DPose' tflite files but am still having trouble finding the 'LiteBaseline3DPose' files

Thank you in advance.

which 3D model is being used?

Dear author!
Thank you for your work.
I am new to swift code and ios devices.
May I ask you when I build your code, which 3D pose model being used? the basedline3D or the lite one?
And how can I check it?
Thank you so much!

Please specify license.

Hi.
Thanks for your awesome work, we refered your postprocess implementation about openpose+TFlite.
When we'd like to use in a production code, we wondering if we can use it.

Support PoseEstimationForMobile tflite model

Reference

• repo: https://github.com/edvardHua/PoseEstimationForMobile
• tflite model:
  • cpm model
  • hourglass model

CPM Model

Model Infos

Benchmark

Hourglass Model

Model Infos

Benchmark

Where is the code for the 2D demo?

Hi, in the readme you also have the 2D Demo .gif:
https://user-images.githubusercontent.com/37643248/77227994-99ba2a80-6bc7-11ea-9b08-9bb57723bc42.gif

But where is the matching code to get this result? I only see the 3D views?

Support front camera on LiveImageViewController

Add post-processing param control UIs

New UIs

• Separate with sections
  • Part, Pair, Human
• Control
  • Threshold of pair (Pair section)
  • Filter size using on NMS (Pair section)
  • Maximum number of people (Human section)

Concept UI

Limit maximum number of human in multi-pose post process step

Support imgclsmob's pose estimation model

There are so many pose estimation models on imgclsmob repo. I'll collect the model's link and mobile models converted to .tflite format.

Model Name	Model Extractor	TFLite Model Link	TFLite Model Size	TFLite Model Input Shape	TFLite Model Output Shape	Keras Model Link (.h5)	Keras Model Size
AlphaPose	Fast-SE-ResNet-101b	N/A	238.1 MB	(1,224,224,3)	(1,56,56,17)	link	211 MB
SimplePose	ResNet-18	link	61.5 MB	(1,224,224,3)	(1,56,56,17)	link	54.3 MB
SimplePose	ResNet-50b	link	135.9 MB	(1,224,224,3)	(1,56,56,17)	link	120 MB
SimplePose	ResNet-101b	link	211.8 MB	(1,224,224,3)	(1,56,56,17)	link	188 MB
SimplePose	ResNet-152b	link	274.3 MB	(1,224,224,3)	(1,56,56,17)	link	243 MB
SimplePose	ResNet(A)-50b	link	136 MB	(1,224,224,3)	(1,56,56,17)	link	120 MB
SimplePose	ResNet(A)-101b	link	211.9 MB	(1,224,224,3)	(1,56,56,17)	link	188 MB
SimplePose	ResNet(A)-152b	link	274.4 MB	(1,224,224,3)	(1,56,56,17)	link	243 MB
SimplePose(Mobile)	ResNet-18	N/A	51.4 MB	(1,224,224,3)	(1,56,56,17)	link	45.4 MB
SimplePose(Mobile)	ResNet-50b	N/A	102.3 MB	(1,224,224,3)	(1,56,56,17)	link	90.5 MB
SimplePose(Mobile)	1.0 MobileNet-224	N/A	20.1 MB	(1,224,224,3)	(1,56,56,17)	link	17.8 MB
SimplePose(Mobile)	1.0 MobileNetV2b-224	N/A	16.4 MB	(1,224,224,3)	(1,56,56,17)	link	14.6 MB
SimplePose(Mobile)	MobileNetV3 Small 224/1.0	N/A	10.5 MB	(1,224,224,3)	(1,56,56,17)	link	9.23 MB
SimplePose(Mobile)	MobileNetV3 Large 224/1.0	N/A	19.1 MB	(1,224,224,3)	(1,56,56,17)	link	16.9 MB
Lightweight OpenPose 2D	MobileNet	N/A	16.4 MB	(1,224,224,3)	(1,28,28,57)	link	14 MB
Lightweight OpenPose 3D	MobileNet	N/A	20.4 MB	(1,224,224,3)	(1,28,28,114)	link	17.5 MB
IBPPose	-	N/A	-	-	-	link	339 MB

Model Source

https://github.com/osmr/imgclsmob/tree/v0.0.460/tensorflow2#coco-keypoints-detection

TFLite Converting Script

https://github.com/osmr/imgclsmob/blob/bb5532e21ff27689e9f1dc133d86933b04396a1c/examples/convert_tf2_to_tfl.py

Sample Command

python convert_tf2_to_tfl.py \
--model lwopenpose2d_mobilenet_cmupan_coco \
--output-dir /home/outputs/pose/tflite;

reco for body "standing" horizontally (standing not on the feet)

Hello,

I am playing with pose estimation models. I am trying to find a model which run on iOS and able to identify join when the body is horizontal.
It seems most models (at least poseNet posenet_mobilenet_v1_100_257x257_multi_kpt_strippe) works better vertically.

do you have any suggestions ?

Thank you

CIContext is able to generate overly

In pixelBufferFromImage(UIImageExtension),

• https://github.com/tucan9389/PoseEstimation-TFLiteSwift/blob/e440b48bddb4459e7bf027b091a6213db4661a02/PoseEstimation-TFLiteSwift/UIImageExtension.swift

CIContext is generated whenever pixelBufferFromImage called.
I recommend to use shared CIContext object(But This isn't just meaning to make single-ton).
Becauase, If CIContext is generated overly then It cause memory issues(zombie object or leak).
So you should use shared CIContext object as possible.

Don't worry about to use shared CIContext object.
This object is thread safe.

Draw Heatmap(ConfidenceMap) each part