Huawei AR Engine Face Tracking Feature - Huawei Developers

Here I will try to explain the Facial Expression Tracking feature of HUAWEI AR Engine as much as I can by developing a demo application. In addition, if you want to learn about the Body Tracking feature offered by the HUAWEI AR Engine and the comparison I made with its competitors, I recommend you to read the 1st article of this series that I wrote before.
This feature of the HUAWEI AR Engine provides meticulous control over the virtual character’s facial expressions by providing the calculated values of the facial poses and the parameter values corresponding to the expressions in real time. It provides this capability in order to track and obtain facial image information, comprehend facial expressions in real time, and convert the facial expressions into various expression parameters, thereby enabling the expressions of virtual characters to be controlled. In addition, AR Engine supports the recognition of 64 types of facial expressions covering eyes, eyebrows, eyeballs, mouth and tongue.
Also the Face Mesh feature of HUAWEI AR Engine, calculates the pose and mesh model data of a face in real time. The mesh model data changes to account for facial movements.
By providing high-precision face mesh modeling and tracking capabilities, HUAWEI AR Engine delivers a highly-realistic mesh model in real time, after obtaining face image information. The mesh model changes its location and shape in accordance with the face, for accurate real time responsivity.
Also, AR Engine provides a mesh with more than 4,000 vertices and 7,000 triangles to precisely outline face contours, and enhance the overall user experience.
Now I will develop a demo application and try to explain this feature and what it provides in more detail.
The figure below shows the general usage process of HUAWEI AR Engine SDK. We will start this process with ARSession, which we will start in Activity’s onResume function.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
While developing this application, we will start with the Engine Functionality section that you can see in the figure below. Then we will develop the render manager class, after that we will complete this article by writing the activity, ie UI part.
While developing this demo application, we will use the OpenGL library for rendering, as in my other article. For this, we will create a class called FaceRenderManager that implements OpenGL’s GLSurfaceView.Renderer interface. We will create and render shaders in the onSurfaceCreated method of this interface. First of all, we will start with the Face Geometry drawing part.
1- Face Geometry
In this section, we will draw the Face Geometry features.
a. Create and Attach Shaders
First of all, we define our vertex shader and fragment shader programs that we will use.
Code:
private static final String LS = System.lineSeparator();
private static final String FACE_GEOMETRY_VERTEX =
"attribute vec2 inTexCoord;" + LS
+ "uniform mat4 inMVPMatrix;" + LS
+ "uniform float inPointSize;" + LS
+ "attribute vec4 inPosition;" + LS
+ "uniform vec4 inColor;" + LS
+ "varying vec4 varAmbient;" + LS
+ "varying vec4 varColor;" + LS
+ "varying vec2 varCoord;" + LS
+ "void main() {" + LS
+ " varAmbient = vec4(1.0, 1.0, 1.0, 1.0);" + LS
+ " gl_Position = inMVPMatrix * vec4(inPosition.xyz, 1.0);" + LS
+ " varColor = inColor;" + LS
+ " gl_PointSize = inPointSize;" + LS
+ " varCoord = inTexCoord;" + LS
+ "}";
private static final String FACE_GEOMETRY_FRAGMENT =
"precision mediump float;" + LS
+ "uniform sampler2D inTexture;" + LS
+ "varying vec4 varColor;" + LS
+ "varying vec2 varCoord;" + LS
+ "varying vec4 varAmbient;" + LS
+ "void main() {" + LS
+ " vec4 objectColor = texture2D(inTexture, vec2(varCoord.x, 1.0 - varCoord.y));" + LS
+ " if(varColor.x != 0.0) {" + LS
+ " gl_FragColor = varColor * varAmbient;" + LS
+ " }" + LS
+ " else {" + LS
+ " gl_FragColor = objectColor * varAmbient;" + LS
+ " }" + LS
+ "}";
Well, we wrote our shader programs. These are our fragment and vertex shader programs to providing the code for certain programmable stages of the face rendering pipeline.
Now it is time to create shader programs. For this, we add the following code. When we call this method by giving the required shader type and shader source code as parameters, we first create an empty shader object, then provide the source code and compile it and get a referenced integer value. Then we will continue with the next steps with this referenced value.
Code:
private static int loadShader(int shaderType, String source) {
int shader = GLES20.glCreateShader(shaderType);
if (0 != shader) {
GLES20.glShaderSource(shader, source);
GLES20.glCompileShader(shader);
int[] compiled = new int[1];
GLES20.glGetShaderiv(shader, GLES20.GL_COMPILE_STATUS, compiled, 0);
if (compiled[0] == 0) {
Log.e(TAG, "glError: Could not compile shader " + shaderType);
Log.e(TAG, "GLES20 Error: " + GLES20.glGetShaderInfoLog(shader));
GLES20.glDeleteShader(shader);
shader = 0;
}
}
return shader;
}
We will now call this method to create both the vertex shader and the fragment shader to create the face geometry. For this, we will pass the FACE_GEOMETRY_VERTEX and FACE_GEOMETRY_FRAGMENT source codes to this method. Then we add the compiled shaders to the program object we created, to be linked later. After that, we link this program object to use it.
Code:
private static int createGlProgram() {
int vertex = loadShader(GLES20.GL_VERTEX_SHADER, FACE_GEOMETRY_VERTEX);
if (vertex == 0) {
return 0;
}
int fragment = loadShader(GLES20.GL_FRAGMENT_SHADER, FACE_GEOMETRY_FRAGMENT);
if (fragment == 0) {
return 0;
}
int program = GLES20.glCreateProgram();
if (program != 0) {
GLES20.glAttachShader(program, vertex);
GLES20.glAttachShader(program, fragment);
GLES20.glLinkProgram(program);
int[] linkStatus = new int[1];
GLES20.glGetProgramiv(program, GLES20.GL_LINK_STATUS, linkStatus, 0);
if (linkStatus[0] != GLES20.GL_TRUE) {
Log.e(TAG, "Could not link program: " + GLES20.glGetProgramInfoLog(program));
GLES20.glDeleteProgram(program);
program = 0;
}
}
return program;
}
Now let’s call the method we wrote and initialize the values we will use. We initialized these values because while drawing the frame (from the onDrawFrame method), we will use these values to draw the points.
Code:
private int mProgram;
private int mPositionAttribute;
private int mColorUniform;
private int mModelViewProjectionUniform;
private int mPointSizeUniform;
private int mTextureUniform;
private int mTextureCoordAttribute;
private void createProgram() {
ShaderUtil.checkGlError(TAG, "Create gl program start.");
mProgram = createGlProgram();
mPositionAttribute = GLES20.glGetAttribLocation(mProgram, "inPosition");
mColorUniform = GLES20.glGetUniformLocation(mProgram, "inColor");
mModelViewProjectionUniform = GLES20.glGetUniformLocation(mProgram, "inMVPMatrix");
mPointSizeUniform = GLES20.glGetUniformLocation(mProgram, "inPointSize");
mTextureUniform = GLES20.glGetUniformLocation(mProgram, "inTexture");
mTextureCoordAttribute = GLES20.glGetAttribLocation(mProgram, "inTexCoord");
ShaderUtil.checkGlError(TAG, "Create gl program end.");
}
Now we will start to create OpenGL ES regarding face geometry, including creating shader programs, using the functions we have written. In the next steps, we will call this function from the onSurfaceCreated function of the GLSurfaceView.Renderer interface to create OpenGL ES when the surface is created so that we can create OpenGL ES.
b. OpenGL Initialization for Face Geometry
First, we create 2 buffer objects. These buffers will hold our vertice information and our Triangle information. Then we will get visuals by updating them.
We bind the first buffer object we created for Vertex attributes to the array buffer. We specify the size of the buffer and do not put any data in it for now. Then we tell OpenGL that with DYNAMIC_DRAW we will frequently update the values in this buffer and therefore do not optimize these values. Finally, we unbind for optimization.
Code:
private static final int BUFFER_OBJECT_NUMBER = 2;
private int mVerticeId;
private int mVerticeBufferSize = 8000; // Initialize the size of the vertex VBO.
private int mTriangleId;
private int mTriangleBufferSize = 5000; // Initialize the size of the triangle VBO.
void init(Context context) {
ShaderUtil.checkGlError(TAG, "Init start.");
//Create Buffer objects
int[] buffers = new int[BUFFER_OBJECT_NUMBER];
GLES20.glGenBuffers(BUFFER_OBJECT_NUMBER, buffers, 0);
mVerticeId = buffers[0];
mTriangleId = buffers[1];
//Bind Array Buffer and set parameters
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVerticeId);
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, mVerticeBufferSize * BYTES_PER_POINT, null, GLES20.GL_DYNAMIC_DRAW);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
}
Then, by adding the following code to the above init function, we bind the second buffer objects we created to the vertex array index binding point with GL_ELEMENT_ARRAY_BUFFER.
Code:
void init(Context context) {
//...
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, mTriangleId);
// Each floating-point number occupies 4 bytes.
GLES20.glBufferData(GLES20.GL_ELEMENT_ARRAY_BUFFER, mTriangleBufferSize * 4, null, GLES20.GL_DYNAMIC_DRAW);
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, 0);
//...
}
After adding them, we add the following code to the init function, create a texture object and bind it to the GL_TEXTURE_2D point.
Code:
int[] texNames = new int[1];
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glGenTextures(1, texNames, 0);
mTextureName = texNames[0];
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureName);
Now that we have binded the texture object, we add the following code to the init () function and call the createProgram () function that we created in the section “a”, attach the shaders and set the texture parameters.
Code:
void init(Context context) {
//...
int[] texNames = new int[1];
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glGenTextures(1, texNames, 0);
mTextureName = texNames[0];
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureName);
createProgram();
Bitmap textureBitmap;
try (InputStream inputStream = context.getAssets().open("face_geometry.png")) {
textureBitmap = BitmapFactory.decodeStream(inputStream);
} catch (IllegalArgumentException | IOException e) {
Log.e(TAG, "Open bitmap error!");
return;
}
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR_MIPMAP_LINEAR);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, textureBitmap, 0);
GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);
ShaderUtil.checkGlError(TAG, "Init end.");
}
The final version of the init () function:
Code:
void init(Context context) {
ShaderUtil.checkGlError(TAG, "Init start.");
int[] texNames = new int[1];
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glGenTextures(1, texNames, 0);
mTextureName = texNames[0];
int[] buffers = new int[BUFFER_OBJECT_NUMBER];
GLES20.glGenBuffers(BUFFER_OBJECT_NUMBER, buffers, 0);
mVerticeId = buffers[0];
mTriangleId = buffers[1];
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVerticeId);
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, mVerticeBufferSize * BYTES_PER_POINT, null, GLES20.GL_DYNAMIC_DRAW);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, mTriangleId);
// Each floating-point number occupies 4 bytes.
GLES20.glBufferData(GLES20.GL_ELEMENT_ARRAY_BUFFER, mTriangleBufferSize * 4, null,
GLES20.GL_DYNAMIC_DRAW);
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, 0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureName);
createProgram();
Bitmap textureBitmap;
try (InputStream inputStream = context.getAssets().open("face_geometry.png")) {
textureBitmap = BitmapFactory.decodeStream(inputStream);
} catch (IllegalArgumentException | IOException e) {
Log.e(TAG, "Open bitmap error!");
return;
}
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR_MIPMAP_LINEAR);
GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, textureBitmap, 0);
GLES20.glGenerateMipmap(GLES20.GL_TEXTURE_2D);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);
ShaderUtil.checkGlError(TAG, "Init end.");
}
c. Draw Face Display
In this section, there are some steps for face drawing operations, that is, to update the face geometry data in buffer. First, we will get the face geometry. For this, we will use the ARFaceGeometry class offered by HUAWEI AR Engine. We can obtain this class using ARSession’s getAllTrackables () function. This way we can get the face (s) on the camera. Example: ArSession.getAllTrackables (ARFace.class)
After obtaining the ARFaceGeometry class, we will use this class to get the vertices, texture coordinates, triangle number and triangle indices of the faces seen by the camera. Then, using these data, we will update the data inside the buffer objects we created earlier and specified with “mVerticeId” and “mTriangleId”.
Now let’s create a function called updateFaceGeometryData that takes an object of type ARFaceGeometry as a parameter. And let’s write the codes of the processes mentioned in the next paragraph into this function.
Code:
private void updateFaceGeometryData(ARFaceGeometry faceGeometry) {
ShaderUtil.checkGlError(TAG, "Before update data.");
FloatBuffer faceVertices = faceGeometry.getVertices();
// Obtain the number of geometric vertices of a face.
mPointsNum = faceVertices.limit() / 3;
FloatBuffer textureCoordinates = faceGeometry.getTextureCoordinates();
// Obtain the number of geometric texture coordinates of the
// face (the texture coordinates are two-dimensional).
int texNum = textureCoordinates.limit() / 2;
Log.d(TAG, "Update face geometry data: texture coordinates size:" + texNum);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVerticeId);
if (mVerticeBufferSize < (mPointsNum + texNum) * BYTES_PER_POINT) {
while (mVerticeBufferSize < (mPointsNum + texNum) * BYTES_PER_POINT) {
// If the capacity of the vertex VBO buffer is insufficient, expand the capacity.
mVerticeBufferSize *= 2;
}
GLES20.glBufferData(GLES20.GL_ARRAY_BUFFER, mVerticeBufferSize, null, GLES20.GL_DYNAMIC_DRAW);
}
GLES20.glBufferSubData(GLES20.GL_ARRAY_BUFFER, 0, mPointsNum * BYTES_PER_POINT, faceVertices);
GLES20.glBufferSubData(GLES20.GL_ARRAY_BUFFER, mPointsNum * BYTES_PER_POINT, texNum * BYTES_PER_COORD,
textureCoordinates);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
mTrianglesNum = faceGeometry.getTriangleCount();
IntBuffer faceTriangleIndices = faceGeometry.getTriangleIndices();
Log.d(TAG, "update face geometry data: faceTriangleIndices.size: " + faceTriangleIndices.limit());
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, mTriangleId);
if (mTriangleBufferSize < mTrianglesNum * BYTES_PER_POINT) {
while (mTriangleBufferSize < mTrianglesNum * BYTES_PER_POINT) {
// If the capacity of the vertex VBO buffer is insufficient, expand the capacity.
mTriangleBufferSize *= 2;
}
GLES20.glBufferData(GLES20.GL_ELEMENT_ARRAY_BUFFER, mTriangleBufferSize, null, GLES20.GL_DYNAMIC_DRAW);
}
GLES20.glBufferSubData(GLES20.GL_ELEMENT_ARRAY_BUFFER, 0, mTrianglesNum * BYTES_PER_POINT, faceTriangleIndices);
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, 0);
ShaderUtil.checkGlError(TAG, "After update data.");
}
As you know, model, view and projection matrices are required for 3D rendering on the screen. So we need to create the matrix of the face for the model. For this, we will use the ARFace class of HUAWEI AR Engine. With the getPose () function of this class, we will obtain the ARPose type object. We obtain the model view matrix from this object. We obtain the projection matrix from the ARCamera object of the HUAWEI AR Engine and multiply them. In this way, we obtain the matrix to update the model view projection (MVP) data. Now let’s do them with the following function.
Code:
private static final float PROJECTION_MATRIX_NEAR = 0.1f;
private static final float PROJECTION_MATRIX_FAR = 100.0f;
// The size of the MVP matrix is 4 x 4.
private float[] mModelViewProjections = new float[16];
private void updateModelViewProjectionData(ARCamera camera, ARFace face) {
// The size of the projection matrix is 4 * 4.
float[] projectionMatrix = new float[16];
camera.getProjectionMatrix(projectionMatrix, 0, PROJECTION_MATRIX_NEAR, PROJECTION_MATRIX_FAR);
ARPose facePose = face.getPose();
// The size of viewMatrix is 4 * 4.
float[] facePoseViewMatrix = new float[16];
facePose.toMatrix(facePoseViewMatrix, 0);
Matrix.multiplyMM(mModelViewProjections, 0, projectionMatrix, 0, facePoseViewMatrix, 0);
}
As the last step of the face drawing phase, we complete the drawing with the following function. With this function, we will draw the geometric features of the face using the values we have created / defined up to this stage.
Note: These drawing functions will be called for each frame.
Code:
private void drawFaceGeometry() {
ShaderUtil.checkGlError(TAG, "Before draw.");
Log.d(TAG, "Draw face geometry: mPointsNum: " + mPointsNum + " mTrianglesNum: " + mTrianglesNum);
GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, mTextureName);
GLES20.glUniform1i(mTextureUniform, 0);
ShaderUtil.checkGlError(TAG, "Init texture.");
GLES20.glEnable(GLES20.GL_DEPTH_TEST);
GLES20.glEnable(GLES20.GL_CULL_FACE);
// Draw point.
GLES20.glUseProgram(mProgram);
GLES20.glEnableVertexAttribArray(mPositionAttribute);
GLES20.glEnableVertexAttribArray(mTextureCoordAttribute);
GLES20.glEnableVertexAttribArray(mColorUniform);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, mVerticeId);
GLES20.glVertexAttribPointer(mPositionAttribute, POSITION_COMPONENTS_NUMBER, GLES20.GL_FLOAT, false, BYTES_PER_POINT, 0);
GLES20.glVertexAttribPointer(mTextureCoordAttribute, TEXCOORD_COMPONENTS_NUMBER, GLES20.GL_FLOAT, false, BYTES_PER_COORD, 0);
GLES20.glUniform4f(mColorUniform, 1.0f, 0.0f, 0.0f, 1.0f);
GLES20.glUniformMatrix4fv(mModelViewProjectionUniform, 1, false, mModelViewProjections, 0);
GLES20.glUniform1f(mPointSizeUniform, 5.0f); // Set the size of Point to 5.
GLES20.glDrawArrays(GLES20.GL_POINTS, 0, mPointsNum);
GLES20.glDisableVertexAttribArray(mColorUniform);
GLES20.glBindBuffer(GLES20.GL_ARRAY_BUFFER, 0);
ShaderUtil.checkGlError(TAG, "Draw point.");
// Draw triangles.
GLES20.glEnableVertexAttribArray(mColorUniform);
// Clear the color and use the texture color to draw triangles.
GLES20.glUniform4f(mColorUniform, 0.0f, 0.0f, 0.0f, 0.0f);
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, mTriangleId);
// The number of input triangle points
GLES20.glDrawElements(GLES20.GL_TRIANGLES, mTrianglesNum * 3, GLES20.GL_UNSIGNED_INT, 0);
GLES20.glBindBuffer(GLES20.GL_ELEMENT_ARRAY_BUFFER, 0);
GLES20.glDisableVertexAttribArray(mColorUniform);
ShaderUtil.checkGlError(TAG, "Draw triangles.");
GLES20.glDisableVertexAttribArray(mTextureCoordAttribute);
GLES20.glDisableVertexAttribArray(mPositionAttribute);
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);
GLES20.glDisable(GLES20.GL_DEPTH_TEST);
GLES20.glDisable(GLES20.GL_CULL_FACE);
ShaderUtil.checkGlError(TAG, "Draw after.");
}
More details, you can check https://forums.developer.huawei.com/forumPortal/en/topicview?tid=0202387445597520482&fid=0101187876626530001&channelname=HuoDong59&ha_source=xda