Introduction of Pose estimation using Huawei HiAI Engine in Android - Huawei Developers

Introduction
In this article, we will learn how to detect human skeletal detection.
The key skeletal features are important for describing human posture and predicting human behavior. Therefore, the recognition of key skeletal features is the basis for a diversity of computer vision tasks, such as motion categorizations, abnormal behavior detection, and auto-navigation. In recent years, improved skeletal feature recognition has been widely applied to the development of deep learning technology, especially domains relating to computer vision.
Pose estimation mainly detects key human body features such as joints and facial features, and provides skeletal information based on such features.
If input a portrait image, users will obtain the coordinate information of 14 key skeletal features of each portrait in it. The algorithm supports real-time processing and returns the result within 70 ms. The result presents posture information regarding head, neck, right and left shoulders, right and left elbows, right and left wrists, and right and left hips, right and left knees, and right and left ankles.
How to integrate Pose Estimation
1. Configure the application on the AGC.
2. Apply for HiAI Engine Library.
3. Client application development process.
Configure application on the AGC
Follow the steps
Step 1: We need to register as a developer account in AppGallery Connect. If you are already a developer ignore this step.
Step 2: Create an app by referring to Creating a Project and Creating an App in the Project
Step 3: Set the data storage location based on the current location.
Step 4: Generating a Signing Certificate Fingerprint.
Step 5: Configuring the Signing Certificate Fingerprint.
Step 6: Download your agconnect-services.json file, paste it into the app root directory.
Apply for HiAI Engine Library
What is Huawei HiAI?
HiAI is Huawei’s AI computing platform. HUAWEI HiAI is a mobile terminal–oriented artificial intelligence (AI) computing platform that constructs three layers of ecology: service capability openness, application capability openness, and chip capability openness. The three-layer open platform that integrates terminals, chips, and the cloud brings more extraordinary experience for users and developers.
How to apply for HiAI Engine?
Follow the steps
Step 1: Navigate to this URL, choose App Service > Development and click HUAWEI HiAI.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Step 2: Click Apply for HUAWEI HiAI kit.
Step 3: Enter required information like Product name and Package name, click Next button.
Step 4: Verify the application details and click Submit button.
Step 5: Click the Download SDK button to open the SDK list.
Step 6: Unzip downloaded SDK and add into your android project under libs folder.
Step 7: Add jar files dependences into app build.gradle file.
Code:
implementation fileTree(include: ['*.aar', '*.jar'], dir: 'libs')
implementation 'com.google.code.gson:gson:2.8.6'
repositories {
flatDir {
dirs 'libs'
}
}
Client application development process
Follow the steps.
Step 1: Create an Android application in the Android studio (Any IDE which is your favorite).
Step 2: Add the App level Gradle dependencies. Choose inside project Android > app > build.gradle.
Code:
apply plugin: 'com.android.application'
apply plugin: 'com.huawei.agconnect'
Root level gradle dependencies.
Code:
maven { url 'https://developer.huawei.com/repo/' }
classpath 'com.huawei.agconnect:agcp:1.4.1.300'
Step 3: Add permission in AndroidManifest.xml.
XML:
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_INTERNAL_STORAGE" />
<uses-permission android:name="android.permission.CAMERA" />
Step 4: Build application.
Java:
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.graphics.Canvas;
import android.graphics.Color;
import android.graphics.Paint;
import android.os.RemoteException;
import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.util.Log;
import android.widget.ImageView;
import android.widget.Toast;
import com.huawei.hiai.pdk.pluginservice.ILoadPluginCallback;
import com.huawei.hiai.pdk.resultcode.HwHiAIResultCode;
import com.huawei.hiai.vision.common.ConnectionCallback;
import com.huawei.hiai.vision.common.VisionBase;
import com.huawei.hiai.vision.common.VisionImage;
import com.huawei.hiai.vision.image.detector.PoseEstimationDetector;
import com.huawei.hiai.vision.visionkit.image.detector.BodySkeletons;
import com.huawei.hiai.vision.visionkit.image.detector.PeConfiguration;
import com.huawei.hiai.vision.visionkit.text.config.VisionTextConfiguration;
import java.io.BufferedInputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;
public class MainActivity extends AppCompatActivity {
private Object mWaitResult = new Object(); // The user establishes a semaphore and waits for the callback information of the bound service
private ImageView mImageView;
private ImageView yogaPose;
[USER=439709]@override[/USER]
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
mImageView = (ImageView) findViewById(R.id.skeleton_img);
yogaPose = (ImageView) findViewById(R.id.yogaPose);
//The application needs to bind the CV service first, and monitor whether the service is successfully connected
VisionBase.init(getApplicationContext(), new ConnectionCallback() {
public void onServiceConnect() { // Listen to the message that the service is successfully bound
Log.d("SkeletonPoint", "HwVisionManager onServiceConnect OK.");
Toast.makeText(getApplicationContext(),"Service binding successfully!",Toast.LENGTH_LONG).show();
synchronized (mWaitResult) {
mWaitResult.notifyAll();
doSkeletonPoint();
}
}
public void onServiceDisconnect() { // Listen to the message that the binding service failed
Log.d("SkeletonPoint", "HwVisionManager onServiceDisconnect OK.");
Toast.makeText(getApplicationContext(),"Service binding failed!",Toast.LENGTH_LONG).show();
synchronized (mWaitResult) {
mWaitResult.notifyAll();
}
}
});
}
[USER=439709]@override[/USER]
protected void onResume() {
super.onResume();
}
[USER=439709]@override[/USER]
protected void onDestroy() {
super.onDestroy();
}
private void doSkeletonPoint() {
// Declare the skeleton detection interface object, and set the plug-in to cross-process mode MODE_OUT (also can be set to the same process mode MODE_IN)
PoseEstimationDetector mPoseEstimationDetector = new PoseEstimationDetector(MainActivity.this);
PeConfiguration config = new PeConfiguration.Builder()
.setProcessMode(VisionTextConfiguration.MODE_OUT)
.build();
mPoseEstimationDetector.setConfiguration(config);
// Currently, the skeleton detection interface accepts input as Bitmap, which is encapsulated into VisionImage. Video streaming will be supported in the future
Bitmap bitmap = null;
VisionImage image = null;
// TODO: Developers need to create a Bitmap here
BufferedInputStream bis = null;
try {
bis = new BufferedInputStream(getAssets().open("0.jpg"));
} catch (IOException e) {
Log.d("SkeletonPoint", e.toString());
Toast.makeText(getApplicationContext(), e.toString(),Toast.LENGTH_LONG).show();
}
bitmap = BitmapFactory.decodeStream(bis);
yogaPose.setImageBitmap(bitmap);
Bitmap bitmap2 = Bitmap.createBitmap(bitmap.getWidth(), bitmap.getHeight(), bitmap.getConfig());
image = VisionImage.fromBitmap(bitmap);
// Query whether the capability supports the installation of plug-ins at the same time. getAvailability() returns -6 to indicate that the current engine supports this ability, but the plug-in needs to be downloaded and installed on the cloud side
int availability = mPoseEstimationDetector.getAvailability();
int installation = HwHiAIResultCode.AIRESULT_UNSUPPORTED; // Indicates that it does not support
if (availability == -6) {
Lock lock = new ReentrantLock();
Condition condition = lock.newCondition();
LoadPluginCallback cb = new LoadPluginCallback(lock, condition);
// Download and install the plugin
mPoseEstimationDetector.loadPlugin(cb);
lock.lock();
try {
condition.await(90, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Log.e("SkeletonPoint", e.getMessage());
} finally {
lock.unlock();
}
installation = cb.mResultCode;
}
// You can call the interface after downloading and installing successfully
if ((availability == HwHiAIResultCode.AIRESULT_SUCCESS)
|| (installation == HwHiAIResultCode.AIRESULT_SUCCESS)) {
// Load model and resources
mPoseEstimationDetector.prepare();
// Skeleton point result returned
List<BodySkeletons> mBodySkeletons = new ArrayList<>();
// The run method is called synchronously. At present, the maximum interface run time is 70 ms, and it is recommended to use another thread to call every frame
// After detect, bitmap will be released
int resultCode = mPoseEstimationDetector.detect(image, mBodySkeletons, null);
Toast.makeText(getApplicationContext(),"resultCode: " + resultCode,Toast.LENGTH_LONG).show();
// Draw a point
if (mBodySkeletons.size() != 0) {
drawPointNew(mBodySkeletons, bitmap2);
mImageView.setImageBitmap(bitmap2);
}
// Release engine
mPoseEstimationDetector.release();
}
}
public static class LoadPluginCallback extends ILoadPluginCallback.Stub {
private int mResultCode = HwHiAIResultCode.AIRESULT_UNKOWN;
private Lock mLock;
private Condition mCondition;
LoadPluginCallback(Lock lock, Condition condition) {
mLock = lock;
mCondition = condition;
}
[USER=439709]@override[/USER]
public void onResult(int resultCode) throws RemoteException {
Log.d("SkeletonPoint", "LoadPluginCallback, onResult: " + resultCode);
mResultCode = resultCode;
mLock.lock();
try {
mCondition.signalAll();
} finally {
mLock.unlock();
}
}
[USER=439709]@override[/USER]
public void onProgress(int i) throws RemoteException {
}
}
private void drawPointNew(List<BodySkeletons> poseEstimationMulPeopleSkeletons, Bitmap bmp) {
if ((poseEstimationMulPeopleSkeletons == null)
|| (poseEstimationMulPeopleSkeletons.size() < 1)) {
return;
}
int humanNum = poseEstimationMulPeopleSkeletons.size();
int points = 14;
int size = humanNum * points;
int[] xArr = new int[size];
int[] yArr = new int[size];
for (int j = 0; (j < humanNum) && (j < 6); j++) {
for (int i = 0; i < points; i++) {
xArr[j * points + i] = (int)((float)poseEstimationMulPeopleSkeletons.get(j).getPosition().get(i).x);
yArr[j * points + i] = (int)((float)poseEstimationMulPeopleSkeletons.get(j).getPosition().get(i).y);
}
}
Paint p = new Paint();
p.setStyle(Paint.Style.FILL_AND_STROKE);
p.setStrokeWidth(5);
p.setColor(Color.GREEN);
Canvas canvas = new Canvas(bmp);
int len = xArr.length;
int[] color = {0xFF000000, 0xFF444444, 0xFF888888, 0xFFCCCCCC, 0xFFFF0000, 0xFF00FF00, 0xFF0000FF,
0xFFFFFF00, 0xFF00FFFF, 0xFFFF00FF, 0xFF8800FF, 0xFF4400FF, 0xFFFFDDDD};
p.setColor(color[4]);
for (int i = 0; i < len; i++) {
canvas.drawCircle(xArr, yArr, 10, p);
}
for (int i = 0; i < humanNum; i++) {
int j = 0;
p.setColor(color[j++]);
if ((xArr[0+points*i]>0) &&(yArr[0+points*i]>0)&&(xArr[1+points*i]>0)&&(yArr[1+points*i]>0)) {
canvas.drawLine(xArr[0+points*i], yArr[0+points*i], xArr[1+points*i], yArr[1+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[1+points*i]>0)&&(yArr[1+points*i]>0)&&(xArr[2+points*i]>0)&&(yArr[2+points*i]>0)) {
canvas.drawLine(xArr[1+points*i], yArr[1+points*i], xArr[2+points*i], yArr[2+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[2+points*i]>0)&&(yArr[2+points*i]>0)&&(xArr[3+points*i]>0)&&(yArr[3+points*i]>0)) {
canvas.drawLine(xArr[2+points*i], yArr[2+points*i], xArr[3+points*i], yArr[3+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[3+points*i]>0)&&(yArr[3+points*i]>0)&&(xArr[4+points*i]>0)&&(yArr[4+points*i]>0)) {
canvas.drawLine(xArr[3+points*i], yArr[3+points*i], xArr[4+points*i], yArr[4+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[1+points*i]>0)&&(yArr[1+points*i]>0)&&(xArr[5+points*i]>0)&&(yArr[5+points*i]>0)) {
canvas.drawLine(xArr[1+points*i], yArr[1+points*i], xArr[5+points*i], yArr[5+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[5+points*i]>0)&&(yArr[5+points*i]>0)&&(xArr[6+points*i]>0)&&(yArr[6+points*i]>0)) {
canvas.drawLine(xArr[5+points*i], yArr[5+points*i], xArr[6+points*i], yArr[6+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[6+points*i]>0)&&(yArr[6+points*i]>0)&&(xArr[7+points*i]>0)&&(yArr[7+points*i]>0)) {
canvas.drawLine(xArr[6+points*i], yArr[6+points*i], xArr[7+points*i], yArr[7+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[1+points*i]>0)&&(yArr[1+points*i]>0)&&(xArr[8+points*i]>0)&&(yArr[8+points*i]>0)) {
canvas.drawLine(xArr[1+points*i], yArr[1+points*i], xArr[8+points*i], yArr[8+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[8+points*i]>0)&&(yArr[8+points*i]>0)&&(xArr[9+points*i]>0)&&(yArr[9+points*i]>0)) {
canvas.drawLine(xArr[8+points*i], yArr[8+points*i], xArr[9+points*i], yArr[9+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[9+points*i]>0)&&(yArr[9+points*i]>0)&&(xArr[10+points*i]>0)&&(yArr[10+points*i]>0)) {
canvas.drawLine(xArr[9+points*i], yArr[9+points*i], xArr[10+points*i], yArr[10+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[1+points*i]>0)&&(yArr[1+points*i]>0)&&(xArr[11+points*i]>0)&&(yArr[11+points*i]>0)) {
canvas.drawLine(xArr[1+points*i], yArr[1+points*i], xArr[11+points*i], yArr[11+points*i], p);
}
p.setColor(color[j++]);
if ((xArr[11+points*i]>0)&&(yArr[11+points*i]>0)&&(xArr[12+points*i]>0)&&(yArr[12+points*i]>0)) {
canvas.drawLine(xArr[11+points*i], yArr[11+points*i], xArr[12+points*i], yArr[12+points*i], p);
}
p.setColor(color[j]);
if ((xArr[12+points*i]>0)&&(yArr[12+points*i]>0)&&(xArr[13+points*i]>0)&&(yArr[13+points*i]>0)) {
canvas.drawLine(xArr[12+points*i], yArr[12+points*i], xArr[13+points*i], yArr[13+points*i], p);
}
}
}
}
Result
Tips and Tricks
This API provides optimal detection results when no more than three portraits are not appear in the image.
This API works better when the proportion of a portrait in an image is high.
At least four skeletal features of the upper part of the body are required for reliable recognition results.
If you are taking Video from a camera or gallery make sure your app has camera and storage permissions.
Add the downloaded huawei-hiai-vision-ove-10.0.4.307.aar, huawei-hiai-pdk-1.0.0.aar file to libs folder.
Check dependencies added properly.
Latest HMS Core APK is required.
Min SDK is 21. Otherwise you will get Manifest merge issue.
Conclusion
In this article, we have learnt what the pose estimation is and how to integrate pose estimation using Huawei HiAI in android with java. We able to detect the image skeleton in the example. It is able to detect head, neck, elbow, knee and ankle.
Reference
Pose Estimation
Apply for Huawei HiAI