README.md

This project is Pose Estimation on iOS with TensorFlow Lite.
If you are interested in iOS + Machine Learning, visit here you can see various DEMOs.

2D pose estimation in real-time	3D pose estimation

Features

• Support 2D pose estimaiton TFLite models
  • Real-time demo with Metal framwork
  • Photo album demo
• Support 3D pose estimation TFLite model
  • Real-time demo with Metal framwork (but realtime model is not ready yet)
  • Real-time pose matching demo
  • Photo album demo
• Render the result keypoints of 2D pose estimation in 2D demo page
• Render the result keypoints of 3D pose estimation with SceneKit
• Render the heatmaps of 2D pose estimation output
  • Part Confidence Maps for typical heatmap based models
  • Part Affinity Fields for OpenPose (2D multi-person)
• Implemented pose-matching with cosine similiarity in 3D pose demo
• Implemented to fix the shoulders' keypoints in 3D pose estimation to pre-process for pose-matching

Models

Source Images

Name	gif	img-0	img-1	img-2
Source	-

Joint Samples

Model Names	gif	img-0	img-1	img-2
PoseNet	-
PEFM CPM	-
PEFM Hourglass	-
OpenPose (multi-person)	-

Heatmap-ConfidenceMap Samples

Model Names	gif	img-0	img-1	img-2
PoseNet	-	-	-	-
PEFM CPM	-
PEFM Hourglass	-
OpenPose (multi-person)	-

Heatmap-PAF Samples

Model Names	gif	img-0	img-1	img-2
OpenPose (PAF x)	-
OpenPose (PAF y)	-

Meta Data

2D

✅ vs ☑️	Name	Size	Inference Time on iPhone11Pro	Post-process Time on iPhone11Pro	PCKh-0.5	multi person vs single person	Model Source	Paper	tflite download
✅	PoseNet	13.3 MB	-	-	-	single	tensorflow/tensorflow	PersonLab
✅	PEFM CPM	2.4 MB	-	-	-	single	edvardHua/PoseEstimationForMobile	Convolutional Pose Machines
✅	PEFM Hourglass v1	1.8 MB	-	-	-	single	edvardHua/PoseEstimationForMobile	Stacked Hourglass Networks
✅	PEFM Hourglass v2	1.7 MB	-	-	-	single	edvardHua/PoseEstimationForMobile	Stacked Hourglass Networks
✅	OpenPose	7.8 MB	-	-	-	multi	ildoonet/tf-pose-estimation	OpenPose
☑️	AlphaPose	-	-	-	-	single	osmr/imgclsmob	RMPE
☑️	SelecSLS	-	-	-	-	single	osmr/imgclsmob	-
☑️	IBPPose	-	-	-	-	single	osmr/imgclsmob	-
☑️	Lightweight OpenPose	-	-	-	-	single	osmr/imgclsmob	OpenPose

3D

✅ vs ☑️	Name	Size	Inference Time on iPhone11Pro	Post-process Time on iPhone11Pro	(metric)	Model Source	Paper	tflite download
✅	Baseline3DPose	137.1 MB	347 ms	79 ms	-	mks0601/3DMPPE_POSENET_RELEASE	Baseline3D	download
✅	LiteBaseline3DPose	16.6 MB	116 ms (cpu only)	19 ms (cpu only)		SangbumChoi/MobileHumanPose	MHP	download

Requirements

• Xcode 11.3+
• iOS 11.0+
• Swift 5
• CocoaPods

gem install cocoapods

Build & Run

1. Install dependencies with cocoapods

cd ~/{PROJECT_PATH}
pod install

2. Open the PoseEstimation-TFLiteSwift.xcworkspace and run the project

See also

• motlabs/awesome-ml-demos-with-ios
• TensorFlow Lite or Tensorflow models provided by:
  • CPM and Hourglass model provided by edvardHua/PoseEstimationForMobile
  • PoseNet model provided by tensorflow/examples
  • OpenPose model provided by ildoonet/tf-pose-estimation
  • Various model provided by osmr/imgclsmob
  • 3DMPPE PoseNet model provided by mks0601/3DMPPE_POSENET_RELEASE
• Pose estimation with Core ML - tucan9389/PoseEstimation-CoreML

License

This repository is licensed under Apache 2.0. Full license text is available in LICENSE.