Beginner: Find yoga pose using Huawei ML kit skeleton detection - Part 1 - Huawei Developers

{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Introduction
In this article, I will explain what is Skeleton detection? How does Skeleton detection work in Android? At the end of this tutorial, we will create the Huawei Skeleton detection in an Android application using Huawei ML Kit.
What is Skeleton detection?
Huawei ML Kit Skeleton detection service detects the human body. So, represents the orientation of a person in a graphical format. Essentially, it’s a set of coordinates that can be connected to describe the position of the person. This service detects and locates key points of the human body such as the top of the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. Currently, full-body and half-body static image recognition and real-time camera stream recognition are supported.
What is the use of Skeleton detection?
Definitely, everyone will have the question like what is the use of it. For example, if you want to develop a fitness application, you can understand and help the user with coordinates from skeleton detection to see if the user has made the exact movements during exercises or you could develop a game about dance movements Using this service and ML kit can understand easily whether the user has done proper excise or not.
How does it work?
You can use skeleton detection over a static image or over a real-time camera stream. Either way, you can get the coordinates of the human body. Of course, when taking them, it’s looking out for critical areas like head, neck, shoulders, elbows, wrists, hips, knees, and ankles. At the same time, both methods will detect multiple human bodies.
There are two attributes to detect skeleton.
1. TYPE_NORMAL
2. TYPE_YOGA
TYPE_NORMAL: If you send the analyzer type as TYPE_NORMAL, perceives skeletal points for normal standing position.
TYPE_YOGA: If you send the analyzer type as TYPE_YOGA, it picks up skeletal points for yoga posture.
Note: The default mode is to detect skeleton points for normal postures.
Integration of Skeleton Detection
1. Configure the application on the AGC.
2. Client application development process.
Configure application on the AGC
This step involves a couple of steps, as follows.
Step 1: We need to register as a developer account in AppGallery Connect. If you are already a developer ignore this step.
Step 2: Create an app by referring to Creating a Project and Creating an App in the Project
Step 3: Set the data storage location based on the current location.
Step 4: Enabling ML. Open AppGallery connect, choose Manage API > ML Kit
Step 5: Generating a Signing Certificate Fingerprint.
Step 6: Configuring the Signing Certificate Fingerprint.
Step 7: Download your agconnect-services.json file, paste it into the app root directory.
Client application development process
This step involves a couple of steps, as follows.
Step 1: Create an Android application in the Android studio (Any IDE which is your favorite).
Step 2: Add the App level Gradle dependencies. Choose inside project Android > app > build.gradle.
1
2apply plugin: 'com.android.application'
apply plugin: 'com.huawei.agconnect'
Root level gradle dependencies.
1
2maven { url 'https://developer.huawei.com/repo/' }
classpath 'com.huawei.agconnect:agcp:1.4.1.300'
Step 3: Add the dependencies in build.gradle
1
2
3implementation 'com.huawei.hms:ml-computer-vision-skeleton:2.0.4.300'
implementation 'com.huawei.hms:ml-computer-vision-skeleton-model:2.0.4.300'
implementation 'com.huawei.hms:ml-computer-vision-yoga-model:2.0.4.300'
To achieve the Skeleton detection example, follow the steps.
1. AGC Configuration
2. Build Android application
Step 1: AGC Configuration
1. Sign in to AppGallery Connect and select My apps.
2. Select the app in which you want to integrate the Huawei ML kit.
3. Navigate to Project Setting > Manage API > ML Kit
Step 2: Build Android application
In this example, I am getting image from the gallery or Camera and getting the skeleton detection and joints points from the ML kit skeleton detection.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147private fun initAnalyzer(analyzerType: Int) {
val setting = MLSkeletonAnalyzerSetting.Factory()
.setAnalyzerType(analyzerType)
.create()
analyzer = MLSkeletonAnalyzerFactory.getInstance().getSkeletonAnalyzer(setting)
imageSkeletonDetectAsync()
}
private fun initFrame(type: Int) {
imageView.invalidate()
val drawable = imageView.drawable as BitmapDrawable
val originBitmap = drawable.bitmap
val maxHeight = (imageView.parent as View).height
val targetWidth = (imageView.parent as View).width
// Update bitmap size
val scaleFactor = (originBitmap.width.toFloat() / targetWidth.toFloat())
.coerceAtLeast(originBitmap.height.toFloat() / maxHeight.toFloat())
val resizedBitmap = Bitmap.createScaledBitmap(
originBitmap,
(originBitmap.width / scaleFactor).toInt(),
(originBitmap.height / scaleFactor).toInt(),
true
)
frame = MLFrame.fromBitmap(resizedBitmap)
initAnalyzer(type)
}
private fun imageSkeletonDetectAsync() {
val task: Task<List<MLSkeleton>>? = analyzer?.asyncAnalyseFrame(frame)
task?.addOnSuccessListener { results ->
// Detection success.
val skeletons: List<MLSkeleton>? = getValidSkeletons(results)
if (skeletons != null && skeletons.isNotEmpty()) {
graphicOverlay?.clear()
val skeletonGraphic = SkeletonGraphic(graphicOverlay, results)
graphicOverlay?.add(skeletonGraphic)
} else {
Log.e(TAG, "async analyzer result is null.")
}
}?.addOnFailureListener { /* Result failure. */ }
}
private fun stopAnalyzer() {
if (analyzer != null) {
try {
analyzer?.stop()
} catch (e: IOException) {
Log.e(TAG, "Failed for analyzer: " + e.message)
}
}
}
override fun onDestroy() {
super.onDestroy()
stopAnalyzer()
}
private fun showPictureDialog() {
val pictureDialog = AlertDialog.Builder(this)
pictureDialog.setTitle("Select Action")
val pictureDialogItems = arrayOf("Select image from gallery", "Capture photo from camera")
pictureDialog.setItems(pictureDialogItems
) { dialog, which ->
when (which) {
0 -> chooseImageFromGallery()
1 -> takePhotoFromCamera()
}
}
pictureDialog.show()
}
fun chooseImageFromGallery() {
val galleryIntent = Intent(Intent.ACTION_PICK, MediaStore.Images.Media.EXTERNAL_CONTENT_URI)
startActivityForResult(galleryIntent, GALLERY)
}
private fun takePhotoFromCamera() {
val cameraIntent = Intent(MediaStore.ACTION_IMAGE_CAPTURE)
startActivityForResult(cameraIntent, CAMERA)
}
public override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
super.onActivityResult(requestCode, resultCode, data)
if (requestCode == GALLERY)
{
if (data != null)
{
val contentURI = data!!.data
try {
val bitmap = MediaStore.Images.Media.getBitmap(this.contentResolver, contentURI)
saveImage(bitmap)
Toast.makeText([email protected], "Image Show!", Toast.LENGTH_SHORT).show()
imageView!!.setImageBitmap(bitmap)
}
catch (e: IOException)
{
e.printStackTrace()
Toast.makeText([email protected], "Failed", Toast.LENGTH_SHORT).show()
}
}
}
else if (requestCode == CAMERA)
{
val thumbnail = data!!.extras!!.get("data") as Bitmap
imageView!!.setImageBitmap(thumbnail)
saveImage(thumbnail)
Toast.makeText([email protected], "Photo Show!", Toast.LENGTH_SHORT).show()
}
}
fun saveImage(myBitmap: Bitmap):String {
val bytes = ByteArrayOutputStream()
myBitmap.compress(Bitmap.CompressFormat.PNG, 90, bytes)
val wallpaperDirectory = File (
(Environment.getExternalStorageDirectory()).toString() + IMAGE_DIRECTORY)
Log.d("fee", wallpaperDirectory.toString())
if (!wallpaperDirectory.exists())
{
wallpaperDirectory.mkdirs()
}
try
{
Log.d("heel", wallpaperDirectory.toString())
val f = File(wallpaperDirectory, ((Calendar.getInstance()
.getTimeInMillis()).toString() + ".png"))
f.createNewFile()
val fo = FileOutputStream(f)
fo.write(bytes.toByteArray())
MediaScannerConnection.scanFile(this, arrayOf(f.getPath()), arrayOf("image/png"), null)
fo.close()
Log.d("TAG", "File Saved::--->" + f.getAbsolutePath())
return f.getAbsolutePath()
}
catch (e1: IOException){
e1.printStackTrace()
}
return ""
}
Result
Tips and Tricks
Check dependencies downloaded properly.
Latest HMS Core APK is required.
If you are taking an image from a camera or gallery make sure your app has camera and storage permission.
Conclusion
In this article, we have learned the integration of the Huawei ML kit, and what is skeleton detection, how it works, what is the use of it, how to get the Joints point from the skeleton detection, types of detections like TYPE_NORMAL and TYPE_YOGA.
Reference
Skeleton Detection
Happy coding