Body Skeleton Tracking with Huawei AR Engine - Huawei Developers

Introduction
The use of Augmented Reality (AR) is increasing every day in many areas from shopping to games, from design to education and more. If we ask what augmented reality is, we will get an answer from wikipedia exactly as follows.
What is the AR?
Augmented reality(AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer-generated perceptual information, sometimes across multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory.
However, today we will talk about the advantages of AR Engine, the augmented reality SDK offered by HUAWEI, rather than this classic definition. We will talk about the differences with the ML Kit, which looks similar but different, offered by HUAWEI, and we will develop a demo application using the HUAWEI AR Engine. And in this way, we will learn how to easily integrate augmented reality to our application by using HUAWEI AR Engine.
What is the HUAWEI AR Engine?
HUAWEI AR Engine is a platform for building augmented reality (AR) apps on Android smartphones. It is based on the HiSilicon chipset, and integrates AR core algorithms to provide basic AR capabilities such as motion tracking, environment tracking, body tracking, and face tracking, allowing our app to bridge virtual world with the real world, for a brand new visually interactive user experience. AR Engine accurately understands and provides virtual and physical convergence capabilities for our applications.
AR Engine Advantages
Normally, integrating augmented reality features to our application is a very complicated and laborious task. However, companies have offered SDKs to make this complex work easy. Apple AR Kit, Google AR Core and HUAWEI AR Engine, which is our main topic to develop applications today, are examples of these SDKs. However, there are differences between these SDKs such as performance, capability, and supported devices.
For example, while Google AR Core does not support face tracking and human body tracking, it does support AR Engine and AR Kit. Also, while AR Engine and AR Kit support hand gestures, AR Core does not.
Other advantages of HUAWEI AR Engine enables your device to understand how people move. HUAWEI AR Engine can assist in placing a virtual object or applying special effect on a hand by locating hand locations and recognizing specific gestures. With the depth component, motion tracking capability can track 21 hand skeleton points to implement precise interactive controls and special effect overlays. Regarding body recognition, the capability can track 23 body skeleton points with specific names (Left Hand etc.) to detect human posture in real time. AR Engine supports use with third party applications and the Depth Api. In addition to all these, HUAWEI AR Engine supports these features for both front and back cameras. Also, it is planned to add the feature to provide directions for certain locations in the coming days.
With the AR Engine, HUAWEI mobile phones provide interaction capabilities such as face, gesture, and body recognition, and more than 240 APIs, in addition to the basic motion tracking and environment tracking capabilities.
Differences from HUAWEI ML Kit
HUAWEI AR Engine Body Tracking and ML Kit vision skeleton recognition may look the same. However, there is quite a difference between them. ML Kit provides for some general purpose capabilities while the AR Engine tracks skeleton information in the AR scenario. Skeleton tracking and motion tracking are both enabled in AR Engine. So, AR Engine has various information from the coordinate system. But the service in the ML Kit cannot do the same. Because they served for different purposes.
The service in the AR Engine is used to create an AR app while the service in the ML Kit can only track the skeleton in the image in the smart phone coordinate system. They are different ways to implement these two services. Also, the models are different.
● ● ●
Demo App Development
We will create a simple demo application by using HUAWEI AR Engine’s Body tracking capability. In this demo application, I will try to draw lines that represent the body skeleton on the human body viewed by the camera. First you need to provide software and hardware requirements.
Hardware Requirements
The current version of HUAWEI AR Engine supports only HUAWEI devices. So, you need a HUAWEI phone that supports HUAWEI AR Engine, can be connected to a computer via a USB cable, and whose camera works properly.(You can see the supported devices in the below table)
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Software Requirements
Java JDK (1.8 or later).
Android Studio (3.1 or later).
Latest HUAWEI AR Engine SDK, which is available on HUAWEI Developers.
Latest HUAWEI AR Engine APK, which is available in HUAWEI AppGallery and has been installed on the phone.
After providing the requirements, you need to add OpenGL to graphic rendering and the HUAWEI AR Engine SDK dependencies to your app level build.gradle
Code:
dependencies{
//HUAWEI AR Engine SDK dependency
implementation 'com.huawei.hms:arenginesdk:2.12.0.1'
//opengl graphics library dependency
implementation 'de.javagl:obj:0.3.0'
}
To using the camera you need to add camera permission in your AndroidManifest.xml file.
Code:
<uses-permission android:name="android.permission.CAMERA" />
Now we are ready for development of our app. Before enter to the development we should know the general process of using the HUAWEI AR Engine SDK. Our demo application AR Engine process has to follow steps in the photo.
Note: The general AR Engine usage process will follow steps in the photo. Development steps are not related with steps in the photo.
Now if you have a look at the general process of AR Engine, we can continue to development.
Note: Remember, if you get confused in these processes, take a look at this picture again after development is complete and you will fully understand the general usage process of HUAWEI AR Engine SDK.
First we need to create a class for body rendering shader utilities. We will use this utility class to create vertex and fragment shaders for body rendering. (We will be take advantage of OpenGL computer graphics library functions to create shaders)
Code:
import android.opengl.GLES20;
import android.util.Log;
/**
* This class provides code and programs related to body rendering shader.
*/
class BodyShaderUtil {
private static final String TAG = BodyShaderUtil.class.getSimpleName();
/**
* Newline character.
*/
public static final String LS = System.lineSeparator();
/**
* Code for the vertex shader.
*/
public static final String BODY_VERTEX =
"uniform vec4 inColor;" + LS
+ "attribute vec4 inPosition;" + LS
+ "uniform float inPointSize;" + LS
+ "varying vec4 varColor;" + LS
+ "uniform mat4 inProjectionMatrix;" + LS
+ "uniform float inCoordinateSystem;" + LS
+ "void main() {" + LS
+ " vec4 position = vec4(inPosition.xyz, 1.0);" + LS
+ " if (inCoordinateSystem == 2.0) {" + LS
+ " position = inProjectionMatrix * position;" + LS
+ " }" + LS
+ " gl_Position = position;" + LS
+ " varColor = inColor;" + LS
+ " gl_PointSize = inPointSize;" + LS
+ "}";
/**
* Code for the segment shader.
*/
public static final String BODY_FRAGMENT =
"precision mediump float;" + LS
+ "varying vec4 varColor;" + LS
+ "void main() {" + LS
+ " gl_FragColor = varColor;" + LS
+ "}";
private BodyShaderUtil() {
}
/**
* Create a shader.
*
* @return Shader program.
*/
static int createGlProgram() {
int vertex = loadShader(GLES20.GL_VERTEX_SHADER, BODY_VERTEX);
if (vertex == 0) {
return 0;
}
int fragment = loadShader(GLES20.GL_FRAGMENT_SHADER, BODY_FRAGMENT);
if (fragment == 0) {
return 0;
}
int program = GLES20.glCreateProgram();
if (program != 0) {
GLES20.glAttachShader(program, vertex);
GLES20.glAttachShader(program, fragment);
GLES20.glLinkProgram(program);
int[] linkStatus = new int[1];
GLES20.glGetProgramiv(program, GLES20.GL_LINK_STATUS, linkStatus, 0);
if (linkStatus[0] != GLES20.GL_TRUE) {
Log.e(TAG, "Could not link program " + GLES20.glGetProgramInfoLog(program));
GLES20.glDeleteProgram(program);
program = 0;
}
}
return program;
}
private static int loadShader(int shaderType, String source) {
int shader = GLES20.glCreateShader(shaderType);
if (0 != shader) {
GLES20.glShaderSource(shader, source);
GLES20.glCompileShader(shader);
int[] compiled = new int[1];
GLES20.glGetShaderiv(shader, GLES20.GL_COMPILE_STATUS, compiled, 0);
if (compiled[0] == 0) {
Log.e(TAG, "glError: Could not compile shader " + shaderType);
Log.e(TAG, "glError: " + GLES20.glGetShaderInfoLog(shader));
GLES20.glDeleteShader(shader);
shader = 0;
}
}
return shader;
}
}
Then we need to create an interface to rendering body AR type related data. We will implement this interface in the classes to be displayed. Then, we will call the overridden methods of the classes that implement this interface from the BodyRenderManager class, which will be created from the onCreate method of our activity.
You can see that we passed ARBody type Collection to onDrawFrame method. There are two reasons for this. First reason is the HUAWEI AR Engine can identify two human bodies at a time by default, and it always returns two body objects. The second reason is that we will draw the body skeleton in overridden onDrawFrame methods. We are using HUAWEI ARBody class because it returns the tracking result during body skeleton tracking, including body skeleton data which will be used on drawing.
Code:
import com.huawei.hiar.ARBody;
import java.util.Collection;
/**
* Rendering body AR type related data.
*/
interface BodyRelatedDisplay {
/**
* Init render.
*/
void init();
/**
* Render objects, call per frame.
*
* @param bodies ARBodies.
* @param projectionMatrix Camera projection matrix.
*/
void onDrawFrame(Collection<ARBody> bodies, float[] projectionMatrix);
}
Now we need to create display classes to pass the data to OpenGL ES.
The body skeleton display class:
(We will use this class to pass skeleton data to be rendered and displayed on the screen to OpenGL ES)
Code:
import java.nio.FloatBuffer;
import java.util.Collection;
/**
* Obtain and pass the skeleton data to openGL ES, which will render the data and displays it on the screen.
*/
public class BodySkeletonDisplay implements BodyRelatedDisplay {
private static final String TAG = BodySkeletonDisplay.class.getSimpleName();
// Number of bytes occupied by each 3D coordinate. Float data occupies 4 bytes.
// Each skeleton point represents a 3D coordinate.
private static final int BYTES_PER_POINT = 4 * 3;
private static final int INITIAL_POINTS_SIZE = 150;
private static final float DRAW_COORDINATE = 2.0f;
private int mVbo;
private int mVboSize;
private int mProgram;
private int mPosition;
private int mProjectionMatrix;
private int mColor;
private int mPointSize;
private int mCoordinateSystem;
private int mNumPoints = 0;
private int mPointsNum = 0;
private FloatBuffer mSkeletonPoints;
/**
* Create a body skeleton shader on the GL thread.
* This method is called when {@link BodyRenderManager#onSurfaceCreated}.
*/
@Override
public void init() {
ShaderUtil.checkGlError(TAG, "Init start.");
int[] buffers = new int[1];
GLES20.glGenBuffers(1, buffers, 0);
mVbo = buffers[0];
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVbo);
mVboSize = INITIAL_POINTS_SIZE * BYTES_PER_POINT;
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, mVboSize, null, GLES20.GL_DYNAMIC_DRAW);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
ShaderUtil.checkGlError(TAG, "Before create gl program.");
createProgram();
ShaderUtil.checkGlError(TAG, "Init end.");
}
private void createProgram() {
ShaderUtil.checkGlError(TAG, "Create gl program start.");
mProgram = BodyShaderUtil.createGlProgram();
mColor = GLES20.glGetUniformLocation(mProgram, "inColor");
mPosition = GLES20.glGetAttribLocation(mProgram, "inPosition");
mPointSize = GLES20.glGetUniformLocation(mProgram, "inPointSize");
mProjectionMatrix = GLES20.glGetUniformLocation(mProgram, "inProjectionMatrix");
mCoordinateSystem = GLES20.glGetUniformLocation(mProgram, "inCoordinateSystem");
ShaderUtil.checkGlError(TAG, "Create gl program end.");
}
private void updateBodySkeleton() {
ShaderUtil.checkGlError(TAG, "Update Body Skeleton data start.");
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVbo);
mNumPoints = mPointsNum;
if (mVboSize < mNumPoints * BYTES_PER_POINT) {
while (mVboSize < mNumPoints * BYTES_PER_POINT) {
// If the size of VBO is insufficient to accommodate the new point cloud, resize the VBO.
mVboSize *= 2;
}
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, mVboSize, null, GLES20.GL_DYNAMIC_DRAW);
}
GLES20.glBufferSubData(GLES20.GL_ARRAY_BUFFER, 0, mNumPoints * BYTES_PER_POINT, mSkeletonPoints);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
ShaderUtil.checkGlError(TAG, "Update Body Skeleton data end.");
}
/**
* Update the node data and draw by using OpenGL.
* This method is called when {@link BodyRenderManager#onDrawFrame}.
*
* @param bodies Body data.
* @param projectionMatrix projection matrix.
*/
@Override
public void onDrawFrame(Collection<ARBody> bodies, float[] projectionMatrix) {
for (ARBody body : bodies) {
if (body.getTrackingState() == ARTrackable.TrackingState.TRACKING) {
float coordinate = 1.0f;
if (body.getCoordinateSystemType() == ARCoordinateSystemType.COORDINATE_SYSTEM_TYPE_3D_CAMERA) {
coordinate = DRAW_COORDINATE;
}
findValidSkeletonPoints(body);
updateBodySkeleton();
drawBodySkeleton(coordinate, projectionMatrix);
}
}
}
private void drawBodySkeleton(float coordinate, float[] projectionMatrix) {
ShaderUtil.checkGlError(TAG, "Draw body skeleton start.");
GLES20.glUseProgram(mProgram);
GLES20.glEnableVertexAttribArray(mPosition);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVbo);
// The size of the vertex attribute is 4, and each vertex has four coordinate components.
GLES20.glVertexAttribPointer(
mPosition, 4, GLES20.GL_FLOAT, false, BYTES_PER_POINT, 0);
GLES20.glUniform4f(mColor, 0.0f, 0.0f, 1.0f, 1.0f);
GLES20.glUniformMatrix4fv(mProjectionMatrix, 1, false, projectionMatrix, 0);
// Set the size of the skeleton points.
GLES20.glUniform1f(mPointSize, 30.0f);
GLES20.glUniform1f(mCoordinateSystem, coordinate);
GLES20.glDrawArrays(GLES20.GL_POINTS, 0, mNumPoints);
GLES20.glDisableVertexAttribArray(mPosition);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
ShaderUtil.checkGlError(TAG, "Draw body skeleton end.");
}
private void findValidSkeletonPoints(ARBody arBody) {
int index = 0;
int[] isExists;
int validPointNum = 0;
float[] points;
float[] skeletonPoints;
// Determine whether the data returned by the algorithm is 3D human
// skeleton data or 2D human skeleton data, and obtain valid skeleton points.
if (arBody.getCoordinateSystemType() == ARCoordinateSystemType.COORDINATE_SYSTEM_TYPE_3D_CAMERA) {
isExists = arBody.getSkeletonPointIsExist3D();
points = new float[isExists.length * 3];
skeletonPoints = arBody.getSkeletonPoint3D();
} else {
isExists = arBody.getSkeletonPointIsExist2D();
points = new float[isExists.length * 3];
skeletonPoints = arBody.getSkeletonPoint2D();
}
// Save the three coordinates of each joint point(each point has three coordinates).
for (int i = 0; i < isExists.length; i++) {
if (isExists[i] != 0) {
points[index++] = skeletonPoints[3 * i];
points[index++] = skeletonPoints[3 * i + 1];
points[index++] = skeletonPoints[3 * i + 2];
validPointNum++;
}
}
mSkeletonPoints = FloatBuffer.wrap(points);
mPointsNum = validPointNum;
}
}
The body skeleton line display class:
(And this class will be used to pass the skeleton point connection data to OpenGL ES for rendering on the screen)
Code:
import java.nio.FloatBuffer;
import java.util.Collection;
/**
* Gets the skeleton point connection data and pass it to OpenGL ES for rendering on the screen.
*/
public class BodySkeletonLineDisplay implements BodyRelatedDisplay {
private static final String TAG = BodySkeletonLineDisplay.class.getSimpleName();
// Number of bytes occupied by each 3D coordinate. Float data occupies 4 bytes.
// Each skeleton point represents a 3D coordinate.
private static final int BYTES_PER_POINT = 4 * 3;
private static final int INITIAL_BUFFER_POINTS = 150;
private static final float COORDINATE_SYSTEM_TYPE_3D_FLAG = 2.0f;
private static final int LINE_POINT_RATIO = 6;
private int mVbo;
private int mVboSize = INITIAL_BUFFER_POINTS * BYTES_PER_POINT;
private int mProgram;
private int mPosition;
private int mProjectionMatrix;
private int mColor;
private int mPointSize;
private int mCoordinateSystem;
private int mNumPoints = 0;
private int mPointsLineNum = 0;
private FloatBuffer mLinePoints;
/**
* Constructor.
*/
BodySkeletonLineDisplay() {
}
/**
* Create a body skeleton line shader on the GL thread.
* This method is called when {@link BodyRenderManager#onSurfaceCreated}.
*/
@Override
public void init() {
ShaderUtil.checkGlError(TAG, "Init start.");
int[] buffers = new int[1];
GLES20.glGenBuffers(1, buffers, 0);
mVbo = buffers[0];
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVbo);
ShaderUtil.checkGlError(TAG, "Before create gl program.");
createProgram();
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, mVboSize, null, GLES20.GL_DYNAMIC_DRAW);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
ShaderUtil.checkGlError(TAG, "Init end.");
}
private void createProgram() {
ShaderUtil.checkGlError(TAG, "Create gl program start.");
mProgram = BodyShaderUtil.createGlProgram();
mPosition = GLES20.glGetAttribLocation(mProgram, "inPosition");
mColor = GLES20.glGetUniformLocation(mProgram, "inColor");
mPointSize = GLES20.glGetUniformLocation(mProgram, "inPointSize");
mProjectionMatrix = GLES20.glGetUniformLocation(mProgram, "inProjectionMatrix");
mCoordinateSystem = GLES20.glGetUniformLocation(mProgram, "inCoordinateSystem");
ShaderUtil.checkGlError(TAG, "Create gl program end.");
}
private void drawSkeletonLine(float coordinate, float[] projectionMatrix) {
ShaderUtil.checkGlError(TAG, "Draw skeleton line start.");
GLES20.glUseProgram(mProgram);
GLES20.glEnableVertexAttribArray(mPosition);
GLES20.glEnableVertexAttribArray(mColor);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVbo);
// Set the width of the rendered skeleton line.
GLES20.glLineWidth(18.0f);
// The size of the vertex attribute is 4, and each vertex has four coordinate components.
GLES20.glVertexAttribPointer(
mPosition, 4, GLES20.GL_FLOAT, false, BYTES_PER_POINT, 0);
GLES20.glUniform4f(mColor, 1.0f, 0.0f, 0.0f, 1.0f);
GLES20.glUniformMatrix4fv(mProjectionMatrix, 1, false, projectionMatrix, 0);
// Set the size of the points.
GLES20.glUniform1f(mPointSize, 100.0f);
GLES20.glUniform1f(mCoordinateSystem, coordinate);
GLES20.glDrawArrays(GLES20.GL_LINES, 0, mNumPoints);
GLES20.glDisableVertexAttribArray(mPosition);
GLES20.glDisableVertexAttribArray(mColor);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
ShaderUtil.checkGlError(TAG, "Draw skeleton line end.");
}
/**
* Rendering lines between body bones.
* This method is called when {@link BodyRenderManager#onDrawFrame}.
*
* @param bodies Bodies data.
* @param projectionMatrix Projection matrix.
*/
@Override
public void onDrawFrame(Collection<ARBody> bodies, float[] projectionMatrix) {
for (ARBody body : bodies) {
if (body.getTrackingState() == ARTrackable.TrackingState.TRACKING) {
float coordinate = 1.0f;
if (body.getCoordinateSystemType() == ARCoordinateSystemType.COORDINATE_SYSTEM_TYPE_3D_CAMERA) {
coordinate = COORDINATE_SYSTEM_TYPE_3D_FLAG;
}
updateBodySkeletonLineData(body);
drawSkeletonLine(coordinate, projectionMatrix);
}
}
}
/**
* Update body connection data.
*/
private void updateBodySkeletonLineData(ARBody body) {
findValidConnectionSkeletonLines(body);
ShaderUtil.checkGlError(TAG, "Update body skeleton line data start.");
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVbo);
mNumPoints = mPointsLineNum;
if (mVboSize < mNumPoints * BYTES_PER_POINT) {
while (mVboSize < mNumPoints * BYTES_PER_POINT) {
// If the storage space is insufficient, allocate double the space.
mVboSize *= 2;
}
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, mVboSize, null, GLES20.GL_DYNAMIC_DRAW);
}
GLES20.glBufferSubData(GLES20.GL_ARRAY_BUFFER, 0, mNumPoints * BYTES_PER_POINT, mLinePoints);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
ShaderUtil.checkGlError(TAG, "Update body skeleton line data end.");
}
private void findValidConnectionSkeletonLines(ARBody arBody) {
mPointsLineNum = 0;
int[] connections = arBody.getBodySkeletonConnection();
float[] linePoints = new float[LINE_POINT_RATIO * connections.length];
float[] coors;
int[] isExists;
if (arBody.getCoordinateSystemType() == ARCoordinateSystemType.COORDINATE_SYSTEM_TYPE_3D_CAMERA) {
coors = arBody.getSkeletonPoint3D();
isExists = arBody.getSkeletonPointIsExist3D();
} else {
coors = arBody.getSkeletonPoint2D();
isExists = arBody.getSkeletonPointIsExist2D();
}
// Filter out valid skeleton connection lines based on the returned results,
// which consist of indexes of two ends, for example, [p0,p1;p0,p3;p0,p5;p1,p2].
// The loop takes out the 3D coordinates of the end points of the valid connection
// line and saves them in sequence.
for (int j = 0; j < connections.length; j += 2) {
if (isExists[connections[j]] != 0 && isExists[connections[j + 1]] != 0) {
linePoints[mPointsLineNum * 3] = coors[3 * connections[j]];
linePoints[mPointsLineNum * 3 + 1] = coors[3 * connections[j] + 1];
linePoints[mPointsLineNum * 3 + 2] = coors[3 * connections[j] + 2];
linePoints[mPointsLineNum * 3 + 3] = coors[3 * connections[j + 1]];
linePoints[mPointsLineNum * 3 + 4] = coors[3 * connections[j + 1] + 1];
linePoints[mPointsLineNum * 3 + 5] = coors[3 * connections[j + 1] + 2];
mPointsLineNum += 2;
}
}
mLinePoints = FloatBuffer.wrap(linePoints);
}
}
More details, you can check https://forums.developer.huawei.com/forumPortal/en/topicview?tid=0202387404360340481&fid=0101187876626530001&channelname=HuoDong59&ha_source=xda

---

How much time it takes to integrate this service ?