Beginner: Skeleton detection in flutter using Huawei ML Kit - Huawei Developers

{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Introduction
In this article, I will explain what is Skeleton detection? How does Skeleton detection work in Flutter? At the end of this tutorial, we will create the Huawei Skeleton detection in Flutter application using Huawei ML Kit.
What is Skeleton detection?
Huawei ML Kit Skeleton detection service detects the human body. So, represents the orientation of a person in a graphical format. Essentially, it’s a set of coordinates that can be connected to describe the position of the person. This service detects and locates key points of the human body such as the top of the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. Currently, full-body and half-body static image recognition and real-time camera stream recognition are supported.
What is the use of Skeleton detection?
Definitely, everyone will have the question like what is the use of it. For example, if you want to develop a fitness application, you can understand and help the user with coordinates from skeleton detection to see if the user has made the exact movements during exercises or you could develop a game about dance movements Using this service and ML kit can understand easily whether the user has done proper excise or not.
How does it work?
You can use skeleton detection over a static image or over a real-time camera stream. Either way, you can get the coordinates of the human body. Of course, when taking them, it’s looking out for critical areas like head, neck, shoulders, elbows, wrists, hips, knees, and ankles. At the same time, both the methods will detect multiple human bodies.
There are two attributes to detect skeleton.
1. TYPE_NORMAL
2. TYPE_YOGA
TYPE_NORMAL: If you send the analyzer type as TYPE_NORMAL, perceives skeletal points for normal standing position.
TYPE_YOGA: If you send the analyzer type as TYPE_YOGA, it picks up skeletal points for yoga posture.
Note: The default mode is to detect skeleton points for normal postures.
Integration of Skeleton Detection
1. Configure the application on the AGC.
2. Client application development process.
Configure application on the AGC
This step involves a couple of steps, as follows.
Step 1: We need to register as a developer account in AppGallery Connect. If you are already a developer ignore this step.
Step 2: Create an app by referring to Creating a Project and Creating an App in the Project
Step 3: Set the data storage location based on the current location.
Step 4: Enabling ML. Open AppGallery connect, choose Manage API > ML Kit
Step 5: Generating a Signing Certificate Fingerprint.
Step 6: Configuring the Signing Certificate Fingerprint.
Step 7: Download your agconnect-services.json file, paste it into the app root directory.
Client application development process
This step involves a couple of steps as follows.
Step 1: Create a flutter application in the Android studio (Any IDE which is your favorite).
Step 2: Add the App level Gradle dependencies. Choose inside project Android > app > build.gradle.
1
2apply plugin: 'com.android.application'
apply plugin: 'com.huawei.agconnect'
Root level gradle dependencies.
1
2maven { url 'https://developer.huawei.com/repo/' }
classpath 'com.huawei.agconnect:agcp:1.4.1.300'
Step 3: Add the downloaded plugin in pubspec.yaml.
Step 4: Add a downloaded file into the outside project directory. Declare plugin path in pubspec.yaml file under dependencies.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28dependencies:
flutter:
sdk: flutter
huawei_account:
path: ../huawei_account/
huawei_location:
path: ../huawei_location/
huawei_map:
path: ../huawei_map/
huawei_analytics:
path: ../huawei_analytics/
huawei_site:
path: ../huawei_site/
huawei_push:
path: ../huawei_push/
huawei_dtm:
path: ../huawei_dtm/
huawei_ml:
path: ../huawei_ml/
agconnect_crash: ^1.0.0
agconnect_remote_config: ^1.0.0
http: ^0.12.2
camera:
path_provider:
path:
image_picker:
fluttertoast: ^7.1.6
shared_preferences: ^0.5.12+4
To achieve the Skeleton detection example, follow the steps.
1. AGC Configuration
2. Build Flutter application
Step 1: AGC Configuration
1. Sign in to AppGallery Connect and select My apps.
2. Select the app in which you want to integrate the Huawei ML kit.
3. Navigate to Project Setting > Manage API > ML Kit
Step 2: Build Flutter application
In this example, I am getting image from gallery or Camera and getting the skeleton detection and joints points from the ML kit skeleton detection.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128import 'dart:io';
import 'package:flutter/material.dart';
import 'package:huawei_ml/huawei_ml.dart';
import 'package:huawei_ml/skeleton/ml_skeleton_analyzer.dart';
import 'package:huawei_ml/skeleton/ml_skeleton_analyzer_setting.dart';
import 'package:image_picker/image_picker.dart';
class SkeletonDetection extends StatefulWidget {
@override
_SkeletonDetectionState createState() => _SkeletonDetectionState();
}
class _SkeletonDetectionState extends State<SkeletonDetection> {
MLSkeletonAnalyzer analyzer;
MLSkeletonAnalyzerSetting setting;
List<MLSkeleton> skeletons;
double _x = 0;
double _y = 0;
double _score = 0;
@override
void initState() {
// TODO: implement initState
analyzer = new MLSkeletonAnalyzer();
setting = new MLSkeletonAnalyzerSetting();
super.initState();
}
@override
Widget build(BuildContext context) {
return Scaffold(
body: Center(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
children: <Widget>[
_setImageView()
],
),
),
floatingActionButton: FloatingActionButton(
onPressed: () {
_showSelectionDialog(context);
},
child: Icon(Icons.camera_alt),
),
);
}
Future<void> _showSelectionDialog(BuildContext context) {
return showDialog(
context: context,
builder: (BuildContext context) {
return AlertDialog(
title: Text("From where do you want to take the photo?"),
content: SingleChildScrollView(
child: ListBody(
children: <Widget>[
GestureDetector(
child: Text("Gallery"),
onTap: () {
_openGallery(context);
},
),
Padding(padding: EdgeInsets.all(8.0)),
GestureDetector(
child: Text("Camera"),
onTap: () {
_openCamera();
},
)
],
),
));
});
}
File imageFile;
void _openGallery(BuildContext context) async {
var picture = await ImagePicker.pickImage(source: ImageSource.gallery);
this.setState(() {
imageFile = picture;
_skeletonDetection();
});
Navigator.of(context).pop();
}
_openCamera() async {
PickedFile pickedFile = await ImagePicker().getImage(
source: ImageSource.camera,
maxWidth: 800,
maxHeight: 800,
);
if (pickedFile != null) {
imageFile = File(pickedFile.path);
this.setState(() {
imageFile = imageFile;
_skeletonDetection();
});
}
Navigator.of(context).pop();
}
Widget _setImageView() {
if (imageFile != null) {
return Image.file(imageFile, width: 500, height: 500);
} else {
return Text("Please select an image");
}
}
_skeletonDetection() async {
// Create a skeleton analyzer.
analyzer = new MLSkeletonAnalyzer();
// Configure the recognition settings.
setting = new MLSkeletonAnalyzerSetting();
setting.path = imageFile.path;
setting.analyzerType = MLSkeletonAnalyzerSetting.TYPE_NORMAL; // Normal posture.
// Get recognition result asynchronously.
List<MLSkeleton> list = await analyzer.asyncSkeletonDetection(setting);
print("Result data: "+list[0].toJson().toString());
// After the recognition ends, stop the analyzer.
bool res = await analyzer.stopSkeletonDetection();
}
}
Result
Tips and Tricks
Download the latest HMS Flutter plugin.
Check dependencies downloaded properly.
Latest HMS Core APK is required.
If you are taking image from a camera or gallery make sure your app has camera and storage permission
Conclusion
In this article, we have learned the integration of the Huawei ML kit, and what is skeleton detection, how it works, what is the use of it, how to get the Joints point from the skeleton detection, types of detections like TYPE_NORMAL and TYPE_YOGA.
Reference
Skeleton Detection
Happy coding