About This Document
The COVID-19 outbreak has changed the shopping habits of consumers. A great proportion of consumers are turning to live streaming e-commerce, a trend that is currently enjoying great success. To achieve a high transaction volume, a live streamer needs to spend a lot of time preparing information about products, discounts, and usage tips. The background used in the live streaming room is another deciding factor. A background that doesn't change is likely to bore the audience, especially when their interest in the products is weak, and cause them to leave the live stream. Unless the live streamer is incredibly charismatic, it's very difficult to keep the audience interested in all products being advertised. As a result, the audience size may decrease during the course of the live stream.
With the image segmentation service launched by HUAWEI ML Kit, various static and dynamic backgrounds are available and can be changed in real time according to product categories or custom requirements, making live streaming more lively and interesting. This service can crop out the portrait of the live streamer based on semantic segmentation. For example, when introducing a household product, the live streamer can change to a home background in real time. This helps give the audience a more immersive shopping experience.
Functional Description
This demo is developed based on the image segmentation and hand keypoint detection services launched by HUAWEI ML Kit, and demonstrates the function of allowing the user to change the background by waving his/her hand. To avoid misoperations, the background only changes when the hand waving motion reaches a certain threshold. Once custom backgrounds are loaded, the user can wave their hand over the phone from right to left to go back to the last background, or wave their hand from left to right to go to the next background. Dynamic backgrounds are also supported. If you want to allow users to change the background with custom hand gestures or implement other hand gesture effects, you can integrate HUAWEI ML Kit's hand keypoint detection service to develop such features.
Now, let's take a detailed look at the development process.
Development Process
1. Add the AppGallery Connect plug-in and the Maven repository.
Code:
buildscript {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
dependencies {
...
classpath 'com.huawei.agconnect:agcp:1.4.1.300'
}
}
allprojects {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
}
2. Perform integration in full SDK mode.
Code:
dependencies{
// Import the base SDK for image segmentation.
implementation 'com.huawei.hms:ml-computer-vision-segmentation:2.0.4.300'
// Import the multiclass segmentation model package.
implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-multiclass-model:2.0.4.300'
// Import the portrait segmentation model package.
implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-body-model:2.0.4.300'
// Import the base SDK for gesture detection.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.4.300'
// Import the hand keypoint detection model package.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.4.300'
}
3. Add configurations to the file header.
Add apply plugin: 'com.huawei.agconnect' after apply plugin: 'com.android.application'.
4. Add the function of automatically updating the machine learning model.
Code:
Add the following configurations to the AndroidManifest.xml file:
<manifest
...
<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value="imgseg,handkeypoint" />
...
</manifest>
5. Create an image segmentation analyzer.
Code:
MLImageSegmentationAnalyzer imageSegmentationAnalyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer(); // Image segmentation analyzer.
MLHandKeypointAnalyzer handKeypointAnalyzer = MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer(); // Gesture detection analyzer.
MLCompositeAnalyzer analyzer = new MLCompositeAnalyzer.Creator()
.add(imageSegmentationAnalyzer)
.add(handKeypointAnalyzer)
.create();
6. Create a detection result processing class.
Code:
public class ImageSegmentAnalyzerTransactor implements MLAnalyzer.MLTransactor<MLImageSegmentation> {
@Override
public void transactResult(MLAnalyzer.Result<MLImageSegmentation> results) {
SparseArray<MLImageSegmentation> items = results.getAnalyseList();
// Process the detection result as required.
// Other detection-related APIs provided by ML Kit cannot be called.
}
@Override
public void destroy() {
// Callback method used to release resources when detection completes.
}
}
public class HandKeypointTransactor implements MLAnalyzer.MLTransactor<List<MLHandKeypoints>> {
@Override
public void transactResult(MLAnalyzer.Result<List<MLHandKeypoints>> results) {
SparseArray<List<MLHandKeypoints>> analyseList = results.getAnalyseList();
// Process the detection result as required.
// Other detection-related APIs provided by ML Kit cannot be called.
}
@Override
public void destroy() {
// Callback method used to release resources when detection completes.
}
}
7. Set the detection result processor, and bind it to the analyzer.
Code:
imageSegmentationAnalyzer.setTransactor(new ImageSegmentAnalyzerTransactor());
handKeypointAnalyzer.setTransactor(new HandKeypointTransactor());
8. Create a LensEngine instance.
Code:
Context context = this.getApplicationContext();
LensEngine lensEngine = new LensEngine.Creator(context,analyzer)
// Use the front- or rear-facing camera. LensEngine.BACK_LENS indicates the rear-facing camera, and LensEngine.FRONT_LENS indicates the front-facing camera.
.setLensType(LensEngine.FRONT_LENS)
.applyDisplayDimension(1280, 720)
.applyFps(20.0f)
.enableAutomaticFocus(true)
.create();
9. Start the camera, read video streams, and perform detection.
Code:
// Implement other logic of the SurfaceView control by yourself.
SurfaceView mSurfaceView = new SurfaceView(this);
try {
lensEngine.run(mSurfaceView.getHolder());
} catch (IOException e) {
// Handle exceptions.
}
10. After the detection is complete, stop the analyzer, and release detection resources.
Code:
if (analyzer != null) {
try {
analyzer.stop();
} catch (IOException e) {
// Handle exceptions.
}
}
if (lensEngine != null) {
lensEngine.release();
}
Summary
ML Kit's background changing function can be quickly developed by importing packages, performing gesture detection, analyzing, and processing the detection result. Performing image segmentation allows you to do even more. For example, you can apply front-end rendering technologies to prevent bullet comments from covering the people on the screen or use graphical assets to create exquisite images of various sizes. With semantic segmentation, you can accurately segment any object from an image. In addition to portraits, food items, pets, buildings, landscapes, flowers, and grasses can also be segmented. With these functions integrated in your app, users no longer need to use professional photo editing software.
Reference
Official website of Huawei Developers
Development Guide
HMS Core official community on Reddit
Demo and sample code
Discussions on Stack Overflow
Related
In this article I will talk about HUAWEI Scene Kit. HUAWEI Scene Kit is a lightweight rendering engine that features high performance and low consumption. It provides advanced descriptive APIs for us to edit, operate, and render 3D materials. Scene Kit adopts physically based rendering (PBR) pipelines to achieve realistic rendering effects. With this Kit, we only need to call some APIs to easily load and display complicated 3D objects on Android phones.
It was announced before with just SceneView feature. But, in the Scene Kit SDK 5.0.2.300 version, they have announced Scene Kit with new features FaceView and ARView. With these new features, the Scene Kit has made the integration of Plane Detection and Face Tracking features much easier.
At this stage, the following question may come to your mind “since there are ML Kit and AR Engine, why are we going to use Scene Kit?” Let’s give the answer to this question with an example.
Differences Between Scene Kit and AR Engine or ML Kit:
For example, we have a Shopping application. And let’s assume that our application has a feature in the glasses purchasing part that the user can test the glasses using AR to see how the glasses looks like in real. Here, we do not need to track facial gestures using the Facial expression tracking feature provided by AR Engine. All we have to do is render a 3D object on the user’s eye. Face Tracking is enough for this. So if we used AR Engine, we would have to deal with graphics libraries like OpenGL. But by using the Scene Kit FaceView, we can easily add this feature to our application without dealing with any graphics library. Because the feature here is a basic feature and the Scene Kit provides this to us.
So what distinguishes AR Engine or ML Kit from Scene Kit is AR Engine and ML Kit provide more detailed controls. However, Scene Kit only provides the basic features (I’ll talk about these features later). For this reason, its integration is much simpler.
Let’s examine what these features provide us.SceneView
With SceneView, we are able to load and render 3D materials in common scenes.
It allows us to:
Load and render 3D materials.
Load the cubemap texture of a skybox to make the scene look larger and more impressive than it actually is.
Load lighting maps to mimic real-world lighting conditions through PBR pipelines.
Swipe on the screen to view rendered materials from different angles.
ARView:
ARView uses the plane detection capability of AR Engine, together with the graphics rendering capability of Scene Kit, to provide us with the capability of loading and rendering 3D materials in common AR scenes.
With ARView, we can:
Load and render 3D materials in AR scenes.
Set whether to display the lattice plane (consisting of white lattice points) to help select a plane in a real-world view.
Tap an object placed onto the lattice plane to select it. Once selected, the object will change to red. Then we can move, resize, or rotate it.
FaceView:
FaceView can use the face detection capability provided by ML Kit or AR Engine to dynamically detect faces. Along with the graphics rendering capability of Scene Kit, FaceView provides us with superb AR scene rendering dedicated for faces.
With FaceView we can:
Dynamically detect faces and apply 3D materials to the detected faces.
As I mentioned above ARView uses the plane detection capability of AR Engine and the FaceView uses the face detection capability provided by either ML Kit or AR Engine. When using the FaceView feature, we can use the SDK we want by specifying which SDK to use in the layout.
Here, we should consider the devices to be supported when choosing the SDK. You can see the supported devices in the table below. Also for more detailed information you can visit this page. (In addition to the table on this page, the Scene Kit’s SceneView feature also supports P40 Lite devices.)
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Also, I think it is useful to mention some important working principles of Scene Kit:
Scene Kit
Provides a Full-SDK, which we can integrate into our app to access 3D graphics rendering capabilities, even though our app runs on phones without HMS Core.
Uses the Entity Component System (ECS) to reduce coupling and implement multi-threaded parallel rendering.
Adopts real-time PBR pipelines to make rendered images look like in a real world.
Supports the general-purpose GPU Turbo to significantly reduce power consumption.
Demo App
Let’s learn in more detail by integrating these 3 features of the Scene Kit with a demo application that we will develop in this section.
To configure the Maven repository address for the HMS Core SDK add the below code to project level build.gradle.
Go to
〉 project level build.gradle > buildscript > repositories
〉project level build.gradle > allprojects > repositories
Code:
maven { url 'https://developer.huawei.com/repo/' }
After that go to
module level build.gradle > dependencies
then add build dependencies for the Full-SDK of Scene Kit in the dependencies block.
Code:
implementation 'com.huawei.scenekit:full-sdk:5.0.2.302'
Note: When adding build dependencies, replace the version here “full-sdk: 5.0.2.302” with the latest Full-SDK version. You can find all the SDK and Full-SDK version numbers in Version Change History.
Then click the Sync Now as shown below
After the build is successfully completed, add the following line to the manifest.xml file for Camera permission.
Code:
<uses-permission android:name="android.permission.CAMERA" />
Now our project is ready to development. We can use all the functionalities of Scene Kit.
Let’s say this demo app is a shopping app. And I want to use Scene Kit features in this application. We’ll use the Scene Kit’s ARView feature in the “office” section of our application to test how a plant and a aquarium looks on our desk.
And in the sunglasses section, we’ll use the FaceView feature to test how sunglasses look on our face.
Finally, we will use the SceneView feature in the shoes section of our application. We’ll test a shoe to see how it looks.
We will need materials to test these properties, let’s get these materials first. I will use 3D models that you can download from the links below. You can use the same or different materials if you want.
Capability: ARView, Used Models: Plant , Aquarium
Capability: FaceView, Used Model: Sunglasses
Capability: SceneView, Used Model: Shoe
Note: I used 3D models in “.glb” format as asset in ARView and FaceView features. However, these links I mentioned contain 3D models in “.gltf” format. I converted “.gltf” format files to “.glb” format. Therefore, you can obtain a 3D model in “.glb” format by uploading all the files (textures, scene.bin and scene.gltf) of the 3D models downloaded from these links to an online converter website. You can use any online conversion website for the conversion process.
All materials must be stored in the assets directory. Thus, we place the materials under app> src> main> assets in our project. After placing it, our file structure will be as follows.
After adding the materials, we will start by adding the ARView feature first. Since we assume that there are office supplies in the activity where we will use the ARView feature, let’s create an activity named OfficeActivity and first develop its layout.
Note: Activities must extend the Activity class. Update the activities that extend the AppCompatActivity with Activity”
Example: It should be “OfficeActivity extends Activity”.
ARView
In order to use the ARView feature of the Scene Kit, we add the following ARView code to the layout (activity_office.xml file).
Code:
<com.huawei.hms.scene.sdk.ARView
android:id="@+id/ar_view"
android:layout_width="match_parent"
android:layout_height="match_parent">
</com.huawei.hms.scene.sdk.ARView>
Overview of the activity_office.xml file:
Code:
<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:gravity="bottom"
tools:context=".OfficeActivity">
<com.huawei.hms.scene.sdk.ARView
android:id="@+id/ar_view"
android:layout_width="match_parent"
android:layout_height="match_parent"/>
<LinearLayout
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_alignParentBottom="true"
android:layout_centerInParent="true"
android:layout_centerHorizontal="true"
android:layout_centerVertical="true"
android:gravity="bottom"
android:layout_marginBottom="30dp"
android:orientation="horizontal">
<Button
android:id="@+id/button_flower"
android:layout_width="110dp"
android:layout_height="wrap_content"
android:onClick="onButtonFlowerToggleClicked"
android:text="Load Flower"/>
<Button
android:id="@+id/button_aquarium"
android:layout_width="110dp"
android:layout_height="wrap_content"
android:onClick="onButtonAquariumToggleClicked"
android:text="Load Aquarium"/>
</LinearLayout>
</RelativeLayout>
We specified 2 buttons, one for the aquarium and the other for loading a plant. Now, let’s do the initializations from OfficeActivity and activate the ARView feature in our application. First, let’s override the onCreate () function to obtain the ARView and the button that will trigger the code of object loading.
Code:
private ARView mARView;
private Button mButtonFlower;
private boolean isLoadFlowerResource = false;
private boolean isLoadAquariumResource = false;
private Button mButtonAquarium;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_office);
mARView = findViewById(R.id.ar_view);
mButtonFlower = findViewById(R.id.button_flower);
mButtonAquarium = findViewById(R.id.button_aquarium);
Toast.makeText(this, "Please move the mobile phone slowly to find the plane", Toast.LENGTH_LONG).show();
}
Then add the method that will be triggered when the buttons are clicked. Here we will check the loading status of the object. We will clean or load the object according to the its situation.
For plant button:
Code:
public void onButtonFlowerToggleClicked(View view) {
mARView.enablePlaneDisplay(true);
if (!isLoadFlowerResource) {
// Load 3D model.
mARView.loadAsset("ARView/flower.glb");
float[] scale = new float[] { 0.15f, 0.15f, 0.15f };
float[] rotation = new float[] { 0.707f, 0.0f, -0.500f, 0.0f };
// (Optional) Set the initial status.
mARView.setInitialPose(scale, rotation);
isLoadFlowerResource = true;
mButtonFlower.setText("Clear Flower");
} else {
// Clear the resources loaded in the ARView.
mARView.clearResource();
mARView.loadAsset("");
isLoadFlowerResource = false;
mButtonFlower.setText("Load Flower");
}
}
For the aquarium button:
Code:
public void onButtonAquariumToggleClicked(View view) {
mARView.enablePlaneDisplay(true);
if (!isLoadAquariumResource) {
// Load 3D model.
mARView.loadAsset("ARView/aquarium.glb");
float[] scale = new float[] { 0.015f, 0.015f, 0.015f };
float[] rotation = new float[] { 0.0f, 0.0f, 0.0f, 0.0f };
// (Optional) Set the initial status.
mARView.setInitialPose(scale, rotation);
isLoadAquariumResource = true;
mButtonAquarium.setText("Clear Aquarium");
} else {
// Clear the resources loaded in the ARView.
mARView.clearResource();
mARView.loadAsset("");
isLoadAquariumResource = false;
mButtonAquarium.setText("Load Aquarium");
}
}
Now let’s talk about what we do with the codes here, line by line. First, we set the ARView.enablePlaneDisplay() function to true, and if a plane is defined in the real world, the program will appear a lattice plane here.
Code:
mARView.enablePlaneDisplay(true);
Then we check whether the object has been loaded or not. If it is not loaded, we specify the path to the 3D model we selected with the mARView.loadAsset() function and load it. (assets> ARView> flower.glb)
Code:
mARView.loadAsset("ARView/flower.glb");
Then we create and initialize scale and rotation arrays for the starting position. For now, we are entering hardcoded values here. For the future versions, by holding the screen, etc. We can set a starting position.
Note: The Scene Kit ARView feature already allows us to move, adjust the size and change the direction of the object we have created on the screen. For this, we should select the object we created and move our finger on the screen to change the position, size or direction of the object.
Here we can adjust the direction or size of the object by adjusting the rotation and scale values.(These values will be used as parameter of setInitialPose() function)
Note: These values can be changed according to used model. To find the appropriate values, you should try yourself. For details of these values see the document of ARView setInitialPose() function.
Code:
float[] scale = new float[] { 0.15f, 0.15f, 0.15f };
float[] rotation = new float[] { 0.707f, 0.0f, -0.500f, 0.0f };
Then we set the scale and rotation values we created as the starting position.
Code:
mARView.setInitialPose(scale, rotation);
After this process, we set the boolean value to indicate that the object has been created and we update the text of the button.
Code:
isLoadResource = true;
mButton.setText(R.string.btn_text_clear_resource);
If the object is already loaded, we clear the resource and load the empty object so that we remove the object from the screen.
Code:
mARView.clearResource();
mARView.loadAsset("");
Then we set the boolean value again and done by updating the button text.
Code:
isLoadResource = false;
mButton.setText(R.string.btn_text_load);
Finally, we should not forget to override the following methods as in the code to ensure synchronization.
Code:
@Override
protected void onPause() {
super.onPause();
mARView.onPause();
}
@Override
protected void onResume() {
super.onResume();
mARView.onResume();
}
@Override
protected void onDestroy() {
super.onDestroy();
mARView.destroy();
}
The overview of OfficeActivity.java should be as follows.
Code:
import android.app.Activity;
import android.os.Bundle;
import android.view.View;
import android.widget.Button;
import android.widget.Toast;
import com.huawei.hms.scene.sdk.ARView;
public class OfficeActivity extends Activity {
private ARView mARView;
private Button mButtonFlower;
private boolean isLoadFlowerResource = false;
private boolean isLoadAquariumResource = false;
private Button mButtonAquarium;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_office);
mARView = findViewById(R.id.ar_view);
mButtonFlower = findViewById(R.id.button_flower);
mButtonAquarium = findViewById(R.id.button_aquarium);
Toast.makeText(this, "Please move the mobile phone slowly to find the plane", Toast.LENGTH_LONG).show();
}
/**
* Synchronously call the onPause() method of the ARView.
*/
@Override
protected void onPause() {
super.onPause();
mARView.onPause();
}
/**
* Synchronously call the onResume() method of the ARView.
*/
@Override
protected void onResume() {
super.onResume();
mARView.onResume();
}
/**
* If quick rebuilding is allowed for the current activity, destroy() of ARView must be invoked synchronously.
*/
@Override
protected void onDestroy() {
super.onDestroy();
mARView.destroy();
}
public void onButtonFlowerToggleClicked(View view) {
mARView.enablePlaneDisplay(true);
if (!isLoadFlowerResource) {
// Load 3D model.
mARView.loadAsset("ARView/flower.glb");
float[] scale = new float[] { 0.15f, 0.15f, 0.15f };
float[] rotation = new float[] { 0.707f, 0.0f, -0.500f, 0.0f };
// (Optional) Set the initial status.
mARView.setInitialPose(scale, rotation);
isLoadFlowerResource = true;
mButtonFlower.setText("Clear Flower");
} else {
// Clear the resources loaded in the ARView.
mARView.clearResource();
mARView.loadAsset("");
isLoadFlowerResource = false;
mButtonFlower.setText("Load Flower");
}
}
public void onButtonAquariumToggleClicked(View view) {
mARView.enablePlaneDisplay(true);
if (!isLoadAquariumResource) {
// Load 3D model.
mARView.loadAsset("ARView/aquarium.glb");
float[] scale = new float[] { 0.015f, 0.015f, 0.015f };
float[] rotation = new float[] { 0.0f, 0.0f, 0.0f, 0.0f };
// (Optional) Set the initial status.
mARView.setInitialPose(scale, rotation);
isLoadAquariumResource = true;
mButtonAquarium.setText("Clear Aquarium");
} else {
// Clear the resources loaded in the ARView.
mARView.clearResource();
mARView.loadAsset("");
isLoadAquariumResource = false;
mButtonAquarium.setText("Load Aquarium");
}
}
}
In this way, we added the ARView feature of Scene Kit to our application. We can now use the ARView feature. Now let’s test the ARView part on a device that supports the Scene Kit ARView feature.
Let’s place plants and aquariums on our table as below and see how it looks.
In order for ARView to recognize the ground, first you need to turn the camera slowly until the plane points you see in the photo appear on the screen. After the plane points appear on the ground, we specify that we will add plants by clicking the load flower button. Then we can add the plant by clicking the point on the screen where we want to add the plant. When we do the same by clicking the aquarium button, we can add an aquarium.
I placed an aquarium and plants on my table. You can test how it looks by placing plants or aquariums on your table or anywhere. You can see how it looks in the photo below.
Note: “Clear Flower” and “Clear Aquarium” buttons will remove the objects we have placed on the screen.
After creating the objects, we select the object we want to move, change its size or direction as you can see in the picture below. Under normal conditions, the color of the selected object will turn into red. (The color of some models doesn’t change. For example, when the aquarium model is selected, the color of the model doesn’t change to red.)
If we want to change the size of the object after selecting it, we can zoom in out by using our two fingers. In the picture above you can see that I changed plants sizes. Also we can move the selected object by dragging it. To change its direction, we can move our two fingers in a circular motion.
More information, you can visit https://forums.developer.huawei.com/forumPortal/en/topicview?tid=0201380649772650451&fid=0101187876626530001
Is Scene Kit free of charge?
Introduction
Black Friday is coming. Have you already piled the products you want to buy into your online shopping cart? What if you're feeling a little more frugal and want to save your hard-earned cash instead? Fortunately, there is Crazy Shopping Cart, a fast-paced and addictive virtual shopping game that integrates HUAWEI ML Kit's hand keypoint detection service to allow you to control a virtual shopping cart using hand gestures. In this post, I'll show you how the game's hand gesture control feature was developed using ML Kit.
How to Play
Players perform hand gestures to control the left-right movement of a shopping cart and collect as many items as possible. The cart speeds up every 15 seconds.
Looks fun right? Let's take a look at how the game's hand gesture control system is developed.
Development Process
1. Configure the Maven repository address.
Open the build.gradle file in the root directory of your Android Studio project.
Code:
buildscript {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
dependencies {
...
classpath 'com.huawei.agconnect:agcp:1.4.1.300'
}
}
allprojects {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
}
2. Perform integration in full SDK mode.
Code:
dependencies{
// Import the base SDK.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.4.300'
// Import the hand keypoint detection model package.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.4.300'
}
After integrating the SDK, add the following configuration to the file header:
Add apply plugin: 'com.huawei.agconnect' after apply plugin: 'com.android.application'.
3. Create a hand keypoint analyzer.
Code:
MLHandKeypointAnalyzer analyzer =MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer();
4. Create the detection result processing class HandKeypointTransactor.
Code:
public class HandKeypointTransactor implements MLAnalyzer.MLTransactor<List<MLHandKeypoints>> {
@Override
public void transactResult(MLAnalyzer.Result<List<MLHandKeypoints>> results) {
SparseArray<List<MLHandKeypoints>> analyseList = results.getAnalyseList();
// Process the detection result as required.
// Other detection-related APIs provided by ML Kit cannot be called.
}
@Override
public void destroy() {
// Callback method used to release resources when detection completes.
}
}
5. Set the detection result processor, and bind it to the analyzer.
Code:
analyzer.setTransactor(new HandKeypointTransactor());
6. Create a LensEngine instance.
Code:
LensEngine lensEngine = new LensEngine.Creator(getApplicationContext(), analyzer)
.setLensType(LensEngine.BACK_LENS)
.applyDisplayDimension(1280, 720)
.applyFps(20.0f)
.enableAutomaticFocus(true)
.create();
7. Call the run method to start the camera and read video streams for detection.
Code:
// Implement other logic of the SurfaceView control by yourself.
SurfaceView mSurfaceView = findViewById(R.id.surface_view);
try {
lensEngine.run(mSurfaceView.getHolder());
} catch (IOException e) {
// Handle exceptions.
}
8. After the detection is complete, stop the analyzer, and release detection resources.
Code:
if (analyzer != null) {
analyzer.stop();
}
if (lensEngine != null) {
lensEngine.release();
}
Summary
As you can see, the development process is really simple and fast. In addition to adding hand gesture control to your mobile games, ML Kit's hand keypoint detection service also has many other useful applications. For example, users can add cute or funny special effects to their videos when using short-video apps with this service integrated. Also, users of smart home apps with this service integrated can customize the hand gestures they use to remotely control appliances.
Try the service out for yourself now by integrating HUAWEI ML Kit with your own apps.
Reference
Official website of Huawei Developers
Development Guide
HMS Core official community on Reddit
Demo and sample code
Discussions on Stack Overflow
What's it like to date a programmer?
John is a Huawei programmer. His girlfriend Jenny, a teacher, has an interesting answer to that question: "Thanks to my programmer boyfriend, my course ranked among the most popular online courses at my school".
Let's go over how this came to be. Due to COVID-19, the school where Jenny taught went entirely online. Jenny, who was new to live streaming, wanted her students to experience the full immersion of traveling to Tokyo, New York, Paris, the Forbidden City, Catherine Palace, and the Louvre Museum, so that they could absorb all of the relevant geographic and historical knowledge related to those places. But how to do so?
Jenny was stuck on this issue, but John quickly came to her rescue.
After analyzing her requirements in detail, John developed a tailored online course app that brings its users an uncannily immersive experience. It enables users to change the background while live streaming. The video imagery within the app looks true-to-life, as each pixel is labeled, and the entire body image — down to a single strand of hair — is completely cut out.
Actual Effects
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
How to ImplementChanging live-streaming backgrounds by gesture can be realized by using image segmentation and hand gesture recognition in HUAWEI ML Kit.
The image segmentation service segments specific elements from static images or dynamic video streams, with 11 types of image elements supported: human bodies, sky scenes, plants, foods, cats and dogs, flowers, water, sand, buildings, mountains, and others.
The hand gesture recognition service offers two capabilities: hand keypoint detection and hand gesture recognition. Hand keypoint detection is capable of detecting 21 hand keypoints (including fingertips, knuckles, and wrists) and returning positions of the keypoints. The hand gesture recognition capability detects and returns the positions of all rectangular areas of the hand from images and videos, as well as the type and confidence of a gesture. This capability can recognize 14 different gestures, including the thumbs-up/down, OK sign, fist, finger heart, and number gestures from 1 to 9. Both capabilities support detection from static images and real-time video streams.
Development Process1. Add the AppGallery Connect plugin and the Maven repository.
Code:
buildscript {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
dependencies {
...
classpath 'com.huawei.agconnect:agcp:1.4.1.300'
}
}
allprojects {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
}
2. Integrate required services in the full SDK mode.
Code:
dependencies{
// Import the basic SDK of image segmentation.
implementation 'com.huawei.hms:ml-computer-vision-segmentation:2.0.4.300'
// Import the multiclass segmentation model package.
implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-multiclass-model:2.0.4.300'
// Import the human body segmentation model package.
implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-body-model:2.0.4.300'
// Import the basic SDK of hand gesture recognition.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.4.300'
// Import the model package of hand keypoint detection.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.4.300'
}
3. Add configurations in the file header.
Add apply plugin: 'com.huawei.agconnect' after apply plugin: 'com.android.application'.
4. Automatically update the machine learning model.
Add the following statements to the AndroidManifest.xml file:
Code:
<manifest
...
<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value="imgseg,handkeypoint" />
...
</manifest>
5. Create an image segmentation analyzer.
Code:
MLImageSegmentationAnalyzer imageSegmentationAnalyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer();// Image segmentation analyzer.
MLHandKeypointAnalyzer handKeypointAnalyzer = MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer();// Hand gesture recognition analyzer.
MLCompositeAnalyzer analyzer = new MLCompositeAnalyzer.Creator()
.add(imageSegmentationAnalyzer)
.add(handKeypointAnalyzer)
.create();
6. Create a class for processing the recognition result.
Code:
public class ImageSegmentAnalyzerTransactor implements MLAnalyzer.MLTransactor<MLImageSegmentation> {
@Override
public void transactResult(MLAnalyzer.Result<MLImageSegmentation> results) {
SparseArray<MLImageSegmentation> items = results.getAnalyseList();
// Process the recognition result as required. Note that only the detection results are processed.
// Other detection-related APIs provided by ML Kit cannot be called.
}
@Override
public void destroy() {
// Callback method used to release resources when the detection ends.
}
}
public class HandKeypointTransactor implements MLAnalyzer.MLTransactor<List<MLHandKeypoints>> {
@Override
public void transactResult(MLAnalyzer.Result<List<MLHandKeypoints>> results) {
SparseArray<List<MLHandKeypoints>> analyseList = results.getAnalyseList();
// Process the recognition result as required. Note that only the detection results are processed.
// Other detection-related APIs provided by ML Kit cannot be called.
}
@Override
public void destroy() {
// Callback method used to release resources when the detection ends.
}
}
7. Set the detection result processor to bind the analyzer to the result processor.
Code:
imageSegmentationAnalyzer.setTransactor(new ImageSegmentAnalyzerTransactor());
handKeypointAnalyzer.setTransactor(new HandKeypointTransactor());
8. Create a LensEngine object.
Code:
Context context = this.getApplicationContext();
LensEngine lensEngine = new LensEngine.Creator(context,analyzer)
// Set the front or rear camera mode. LensEngine.BACK_LENS indicates the rear camera, and LensEngine.FRONT_LENS indicates the front camera.
.setLensType(LensEngine.FRONT_LENS)
.applyDisplayDimension(1280, 720)
.applyFps(20.0f)
.enableAutomaticFocus(true)
.create();
9. Start the camera, read video streams, and start recognition.
Code:
// Implement other logics of the SurfaceView control by yourself.
SurfaceView mSurfaceView = new SurfaceView(this);
try {
lensEngine.run(mSurfaceView.getHolder());
} catch (IOException e) {
// Exception handling logic.
}
10. Stop the analyzer and release the recognition resources when recognition ends.
Code:
if (analyzer != null) {
try {
analyzer.stop();
} catch (IOException e) {
// Exception handling.
}
}
if (lensEngine != null) {
lensEngine.release();
}
For more details, as follows:
Our official website
Our Development Documentation page, to find the documents you need
Experience the easy-integration process on Codelabs
GitHub to download demos and sample codes
Stack Overflow to solve any integration problem
Original Source
muraliameakula said:
What's it like to date a programmer?
John is a Huawei programmer. His girlfriend Jenny, a teacher, has an interesting answer to that question: "Thanks to my programmer boyfriend, my course ranked among the most popular online courses at my school".
Let's go over how this came to be. Due to COVID-19, the school where Jenny taught went entirely online. Jenny, who was new to live streaming, wanted her students to experience the full immersion of traveling to Tokyo, New York, Paris, the Forbidden City, Catherine Palace, and the Louvre Museum, so that they could absorb all of the relevant geographic and historical knowledge related to those places. But how to do so?
Jenny was stuck on this issue, but John quickly came to her rescue.
After analyzing her requirements in detail, John developed a tailored online course app that brings its users an uncannily immersive experience. It enables users to change the background while live streaming. The video imagery within the app looks true-to-life, as each pixel is labeled, and the entire body image — down to a single strand of hair — is completely cut out.
Actual Effects
How to ImplementChanging live-streaming backgrounds by gesture can be realized by using image segmentation and hand gesture recognition in HUAWEI ML Kit.
The image segmentation service segments specific elements from static images or dynamic video streams, with 11 types of image elements supported: human bodies, sky scenes, plants, foods, cats and dogs, flowers, water, sand, buildings, mountains, and others.
The hand gesture recognition service offers two capabilities: hand keypoint detection and hand gesture recognition. Hand keypoint detection is capable of detecting 21 hand keypoints (including fingertips, knuckles, and wrists) and returning positions of the keypoints. The hand gesture recognition capability detects and returns the positions of all rectangular areas of the hand from images and videos, as well as the type and confidence of a gesture. This capability can recognize 14 different gestures, including the thumbs-up/down, OK sign, fist, finger heart, and number gestures from 1 to 9. Both capabilities support detection from static images and real-time video streams.
Development Process1. Add the AppGallery Connect plugin and the Maven repository.
Code:
buildscript {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
dependencies {
...
classpath 'com.huawei.agconnect:agcp:1.4.1.300'
}
}
allprojects {
repositories {
google()
jcenter()
maven {url 'https://developer.huawei.com/repo/'}
}
}
2. Integrate required services in the full SDK mode.
Code:
dependencies{
// Import the basic SDK of image segmentation.
implementation 'com.huawei.hms:ml-computer-vision-segmentation:2.0.4.300'
// Import the multiclass segmentation model package.
implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-multiclass-model:2.0.4.300'
// Import the human body segmentation model package.
implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-body-model:2.0.4.300'
// Import the basic SDK of hand gesture recognition.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint:2.0.4.300'
// Import the model package of hand keypoint detection.
implementation 'com.huawei.hms:ml-computer-vision-handkeypoint-model:2.0.4.300'
}
3. Add configurations in the file header.
Add apply plugin: 'com.huawei.agconnect' after apply plugin: 'com.android.application'.
4. Automatically update the machine learning model.
Add the following statements to the AndroidManifest.xml file:
Code:
<manifest
...
<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value="imgseg,handkeypoint" />
...
</manifest>
5. Create an image segmentation analyzer.
Code:
MLImageSegmentationAnalyzer imageSegmentationAnalyzer = MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer();// Image segmentation analyzer.
MLHandKeypointAnalyzer handKeypointAnalyzer = MLHandKeypointAnalyzerFactory.getInstance().getHandKeypointAnalyzer();// Hand gesture recognition analyzer.
MLCompositeAnalyzer analyzer = new MLCompositeAnalyzer.Creator()
.add(imageSegmentationAnalyzer)
.add(handKeypointAnalyzer)
.create();
6. Create a class for processing the recognition result.
Code:
public class ImageSegmentAnalyzerTransactor implements MLAnalyzer.MLTransactor<MLImageSegmentation> {
@Override
public void transactResult(MLAnalyzer.Result<MLImageSegmentation> results) {
SparseArray<MLImageSegmentation> items = results.getAnalyseList();
// Process the recognition result as required. Note that only the detection results are processed.
// Other detection-related APIs provided by ML Kit cannot be called.
}
@Override
public void destroy() {
// Callback method used to release resources when the detection ends.
}
}
public class HandKeypointTransactor implements MLAnalyzer.MLTransactor<List<MLHandKeypoints>> {
@Override
public void transactResult(MLAnalyzer.Result<List<MLHandKeypoints>> results) {
SparseArray<List<MLHandKeypoints>> analyseList = results.getAnalyseList();
// Process the recognition result as required. Note that only the detection results are processed.
// Other detection-related APIs provided by ML Kit cannot be called.
}
@Override
public void destroy() {
// Callback method used to release resources when the detection ends.
}
}
7. Set the detection result processor to bind the analyzer to the result processor.
Code:
imageSegmentationAnalyzer.setTransactor(new ImageSegmentAnalyzerTransactor());
handKeypointAnalyzer.setTransactor(new HandKeypointTransactor());
8. Create a LensEngine object.
Code:
Context context = this.getApplicationContext();
LensEngine lensEngine = new LensEngine.Creator(context,analyzer)
// Set the front or rear camera mode. LensEngine.BACK_LENS indicates the rear camera, and LensEngine.FRONT_LENS indicates the front camera.
.setLensType(LensEngine.FRONT_LENS)
.applyDisplayDimension(1280, 720)
.applyFps(20.0f)
.enableAutomaticFocus(true)
.create();
9. Start the camera, read video streams, and start recognition.
Code:
// Implement other logics of the SurfaceView control by yourself.
SurfaceView mSurfaceView = new SurfaceView(this);
try {
lensEngine.run(mSurfaceView.getHolder());
} catch (IOException e) {
// Exception handling logic.
}
10. Stop the analyzer and release the recognition resources when recognition ends.
Code:
if (analyzer != null) {
try {
analyzer.stop();
} catch (IOException e) {
// Exception handling.
}
}
if (lensEngine != null) {
lensEngine.release();
}
For more details, as follows:
Our official website
Our Development Documentation page, to find the documents you need
Experience the easy-integration process on Codelabs
GitHub to download demos and sample codes
Stack Overflow to solve any integration problem
Original Source
Click to expand...
Click to collapse
can i integrate any video background?
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Emojis are a must-have tool in today's online communications as they help add color to text-based chatting and allow users to better express the emotions behind their words. Since the number of preset emojis is always limited, many apps now allow users to create their own custom emojis to keep things fresh and exciting.
For example, in a social media app, users who do not want to show their faces when making video calls can use an animated character to protect their privacy, with their facial expressions applied to the character; in a live streaming or e-commerce app, virtual streamers with realistic facial expressions are much more likely to attract watchers; in a video or photo shooting app, users can control the facial expressions of an animated character when taking a selfie, and then share the selfie via social media; and in an educational app for kids, a cute animated character with detailed facial expressions will make online classes much more fun and engaging for students.
I myself am developing such a messaging app. When chatting with friends and wanting to express themselves in ways other than words, users of my app can take a photo to create an emoji of themselves, or of an animated character they have selected. The app will then identify users' facial expressions, and apply their facial expressions to the emoji. In this way, users are able to create an endless amount of unique emojis. During the development of my app, I used the capabilities provided by HMS Core AR Engine to track users' facial expressions and convert the facial expressions into parameters, which greatly reduced the development workload. Now I will show you how I managed to do this.
ImplementationAR Engine provides apps with the ability to track and recognize facial expressions in real time, which can then be converted into facial expression parameters and used to accurately control the facial expressions of virtual characters.
Currently, AR Engine provides 64 facial expressions, including eyelid, eyebrow, eyeball, mouth, and tongue movements. It supports 21 eye-related movements, including eyeball movement and opening and closing the eyes; 28 mouth movements, including opening the mouth, puckering, pulling, or licking the lips, and moving the tongue; as well as 5 eyebrow movements, including raising or lowering the eyebrows.
DemoFacial expression based emoji
Development ProcedureRequirements on the Development EnvironmentJDK: 1.8.211 or later
Android Studio: 3.0 or later
minSdkVersion: 26 or later
targetSdkVersion: 29 (recommended)
compileSdkVersion: 29 (recommended)
Gradle version: 6.1.1 or later (recommended)
Make sure that you have downloaded the AR Engine APK from AppGallery and installed it on the device.
Test device: see Software and Hardware Requirements of AR Engine Features
If you need to use multiple HMS Core kits, use the latest versions required for these kits.
Preparations1. Before getting started, you will need to register as a Huawei developer and complete identity verification on HUAWEI Developers. You can click here to find out the detailed registration and identity verification procedure.
2. Before development, integrate the AR Engine SDK via the Maven repository into your development environment.
3. The procedure for configuring the Maven repository address in Android Studio varies for Gradle plugin earlier than 7.0, Gradle plugin 7.0, and Gradle plugin 7.1 or later. You need to configure it according to the specific Gradle plugin version.
4. Take Gradle plugin 7.0 as an example:
Open the project-level build.gradle file in your Android Studio project and configure the Maven repository address.
Go to buildscript > repositories and configure the Maven repository address for the SDK.
Code:
buildscript {
repositories {
google()
jcenter()
maven {url "https://developer.huawei.com/repo/" }
}
}
Open the project-level settings.gradle file and configure the Maven repository address for the HMS Core SDK.
Code:
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
repositories {
google()
jcenter()
maven {url "https://developer.huawei.com/repo/" }
}
}
}
5. Add the following build dependency in the dependencies block.
Code:
dependencies {
implementation 'com.huawei.hms:arenginesdk:{version}
}
App Development1. Check whether AR Engine has been installed on the current device. If yes, your app can run properly. If not, you need to prompt the user to install it, for example, by redirecting the user to AppGallery. The sample code is as follows:
Code:
boolean isInstallArEngineApk =AREnginesApk.isAREngineApkReady(this);
if (!isInstallArEngineApk) {
// ConnectAppMarketActivity.class is the activity for redirecting users to AppGallery.
startActivity(new Intent(this, com.huawei.arengine.demos.common.ConnectAppMarketActivity.class));
isRemindInstall = true;
}
2. Create an AR scene. AR Engine supports five scenes, including motion tracking (ARWorldTrackingConfig), face tracking (ARFaceTrackingConfig), hand recognition (ARHandTrackingConfig), human body tracking (ARBodyTrackingConfig), and image recognition(ARImageTrackingConfig).
The following takes creating a face tracking scene by calling ARFaceTrackingConfig as an example.
Code:
// Create an ARSession object.
mArSession = new ARSession(this);
// Select a specific Config to initialize the ARSession object based on the application scenario.
ARFaceTrackingConfig config = new ARFaceTrackingConfig(mArSession);
Set scene parameters using the config.setXXX method.
Code:
// Set the camera opening mode, which can be external or internal. The external mode can only be used in ARFace. Therefore, you are advised to use the internal mode.
mArConfig.setImageInputMode(ARConfigBase.ImageInputMode.EXTERNAL_INPUT_ALL);
3. Set the AR scene parameters for face tracking and start face tracking.
Code:
mArSession.configure(mArConfig);
mArSession.resume();
4. Initialize the FaceGeometryDisplay class to obtain the facial geometric data and render the data on the screen.
Code:
public class FaceGeometryDisplay {
// Initialize the OpenGL ES rendering related to face geometry, including creating the shader program.
void init(Context context) {...
}
}
5. Initialize the onDrawFrame method in the FaceGeometryDisplay class, and call face.getFaceGeometry() to obtain the face mesh.
Code:
public void onDrawFrame(ARCamera camera, ARFace face) {
ARFaceGeometry faceGeometry = face.getFaceGeometry();
updateFaceGeometryData(faceGeometry);
updateModelViewProjectionData(camera, face);
drawFaceGeometry();
faceGeometry.release();
}
6. Initialize updateFaceGeometryData() in the FaceGeometryDisplay class.
Pass the face mesh data for configuration and set facial expression parameters using OpenGl ES.
Code:
private void updateFaceGeometryData (ARFaceGeometry faceGeometry) {
FloatBuffer faceVertices = faceGeometry.getVertices();
FloatBuffer textureCoordinates =faceGeometry.getTextureCoordinates();
// Obtain an array consisting of face mesh texture coordinates, which is used together with the vertex data returned by getVertices() during rendering.
}
7. Initialize the FaceRenderManager class to manage facial data rendering.
Code:
public class FaceRenderManager implements GLSurfaceView.Renderer {
public FaceRenderManager(Context context, Activity activity) {
mContext = context;
mActivity = activity;
}
// Set ARSession to obtain the latest data.
public void setArSession(ARSession arSession) {
if (arSession == null) {
LogUtil.error(TAG, "Set session error, arSession is null!");
return;
}
mArSession = arSession;
}
// Set ARConfigBase to obtain the configuration mode.
public void setArConfigBase(ARConfigBase arConfig) {
if (arConfig == null) {
LogUtil.error(TAG, "setArFaceTrackingConfig error, arConfig is null.");
return;
}
mArConfigBase = arConfig;
}
// Set the camera opening mode.
public void setOpenCameraOutsideFlag(boolean isOpenCameraOutsideFlag) {
isOpenCameraOutside = isOpenCameraOutsideFlag;
}
...
@Override
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
mFaceGeometryDisplay.init(mContext);
}
}
8. Implement the face tracking effect by calling methods like setArSession and setArConfigBase of FaceRenderManager in FaceActivity.
Code:
public class FaceActivity extends BaseActivity {
@Override
protected void onCreate(Bundle savedInstanceState) {
mFaceRenderManager = new FaceRenderManager(this, this);
mFaceRenderManager.setDisplayRotationManage(mDisplayRotationManager);
mFaceRenderManager.setTextView(mTextView);
glSurfaceView.setRenderer(mFaceRenderManager);
glSurfaceView.setRenderMode(GLSurfaceView.RENDERMODE_CONTINUOUSLY);
}
}
ConclusionEmojis allow users to express their moods and excitement in a way words can't. Instead of providing users with a selection of the same old boring preset emojis that have been used a million times, you can now make your app more fun by allowing users to create emojis themselves! Users can easily create an emoji with their own smiles, simply by facing the camera, selecting an animated character they love, and smiling. With such an ability to customize emojis, users will be able to express their feelings in a more personalized and interesting manner. If you have any interest in developing such an app, AR Engine is a great choice for you. With accurate facial tracking capabilities, it is able to identify users' facial expressions in real time, convert the facial expressions into parameters, and then apply them to virtual characters. Integrating the capability can help you considerably streamline your app development process, leaving you with more time to focus on how to provide more interesting features to users and improve your app's user experience.
ReferenceAR Engine Sample Code
Face Tracking Capability
Augmented reality (AR) is now widely used in a diverse range of fields, to facilitate fun and immersive experiences and interactions. Many features like virtual try-on, 3D gameplay, and interior design, among many others, depend on this technology. For example, many of today's video games use AR to keep gameplay seamless and interactive. Players can create virtual characters in battle games, and make them move as if they are extensions of the player's body. With AR, characters can move and behave like real people, hiding behind a wall, for instance, to escape detection by the enemy. Another common application is adding elements like pets, friends, and objects to photos, without compromising the natural look in the image.
However, AR app development is still hindered by the so-called pass-through problem, which you may have encountered during the development. Examples include a ball moving too fast and then passing through the table, a player being unable to move even when there are no obstacles around, or a fast-moving bullet passing through and then missing its target. You may also have found that the virtual objects that your app applies to the physical world look as if they were pasted on the screen, instead of blending into the environment. This can to a large extent undermine the user experience and may lead directly to user churn. Fortunately there is environment mesh in HMS Core AR Engine, a toolkit that offers powerful AR capabilities and streamlines your app development process, to resolve these issues once and for all. After being integrated with this toolkit, your app will enjoy better perception of the 3D space in which a virtual object is placed, and perform collision detection using the reconstructed mesh. This ensures that users are able to interact with virtual objects in a highly realistic and natural manner, and that virtual characters will be able to move around 3D spaces with greater ease. Next we will show you how to implement this capability.
Demo
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
ImplementationAR Engine uses the real time computing to output the environment mesh, which includes the device orientation in a real space, and 3D grid for the current camera view. AR Engine is currently supported on mobile phone models with rear ToF cameras, and only supports the scanning of static scenes. After being integrated with this toolkit, your app will be able to use environment meshes to accurately recognize the real world 3D space where a virtual character is located, and allow for the character to be placed anywhere in the space, whether it is a horizontal surface, vertical surface, or curved surface that can be reconstructed. You can use the reconstructed environment mesh to implement virtual and physical occlusion and collision detection, and even hide virtual objects behind physical ones, to effectively prevent pass-through.
Environment mesh technology has a wide range of applications. For example, it can be used to provide users with more immersive and refined virtual-reality interactions during remote collaboration, video conferencing, online courses, multi-player gaming, laser beam scanning (LBS), metaverse, and more.
Integration ProcedureEnsure that you have met the following requirements on the development environment:
JDK: 1.8.211 or later
Android Studio: 3.0 or later
minSdkVersion: 26 or later
targetSdkVersion: 29 (recommended)
compileSdkVersion: 29 (recommended)
Gradle version: 6.1.1 or later (recommended)
Make sure that you have downloaded the AR Engine APK from AppGallery and installed it on the device.
If you need to use multiple HMS Core kits, use the latest versions required for these kits.
Preparations1. Before getting started, you will need to register as a Huawei developer and complete identity verification on the HUAWEI Developers website. You can click here to find out the detailed registration and identity verification procedure.
2. Before development, integrate the AR Engine SDK via the Maven repository into your development environment.
3. The procedure for configuring the Maven repository address in Android Studio varies for Gradle plugin earlier than 7.0, Gradle plugin 7.0, and Gradle plugin 7.1 or later. You need to configure it according to the specific Gradle plugin version.
4. The following takes Gradle plugin 7.0 as an example:
Open the project-level build.gradle file in your Android Studio project and configure the Maven repository address.
Go to buildscript > repositories and configure the Maven repository address for the SDK.
Code:
buildscript {
repositories {
google()
jcenter()
maven {url "https://developer.huawei.com/repo/" }
}
}
Open the project-level settings.gradle file and configure the Maven repository address for the HMS Core SDK.
Code:
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
repositories {
google()
jcenter()
maven {url "https://developer.huawei.com/repo/" }
}
}
}
5. Add the following build dependency in the dependencies block.
Code:
dependencies {
implementation 'com.huawei.hms:arenginesdk:{version}
}
Development ProcedureInitialize the HitResultDisplay class to draw virtual objects based on the specified parameters.
Code:
Public class HitResultDisplay implements SceneMeshComponenDisplay{
// Initialize VirtualObjectData.
VirtualObjectData mVirtualObject = new VirtualObjectData();
// Pass the context to mVirtualObject in the init method.
Public void init(Context context){
mVirtualObject.init(context);
/ / Pass the material attributes.
mVirtualObject.setMaterialProperties();
}
// Pass ARFrame in the onDrawFrame method to obtain the estimated lighting.
Public void onDrawFrame(ARFrame arframe){
// Obtain the estimated lighting.
ARLightEstimate le = arframe.getLightEstimate();
// Obtain the pixel intensity of the current camera field of view.
lightIntensity = le.getPixelIntensity();
// Pass data to methods in mVirtualObject.
mVirtualObject.draw(…,…,lightIntensity,…);
// Pass the ARFrame object in the handleTap method to obtain the coordinate information.
handleTap(arframe);
}
// Implement the handleTap method.
Private void handleTap(ARFrame frame){
// Call hitTest with the ARFrame object.
List<ARHitResult> hitTestResults = frame.hitTest(tap);
// Check whether a surface is hit and whether it is hit in a plane polygon.
For(int i = 0;i<hitTestResults.size();i++){
ARHitResult hitResultTemp = hitTestResults.get(i);
Trackable = hitResultTemp.getTrackable();
If(trackable instanceof ARPoint && ((ARPoint) trackable).getOrientationMode() == ARPoint.OrientationMode.ESTIMATED_SURFACE_NORMAL){
isHasHitFlag = true;
hitResult = hitResultTemp;
}
}
}
}
2. Initialize the SceneMeshDisplay class to render the scene network.
Code:
Public class SceneMeshDiaplay implements SceneMeshComponenDisplay{
// Implement openGL operations in init.
Public void init(Context context){}
// Obtain the current environment mesh in the onDrawFrame method.
Public void onDrawFrame(ARFrame arframe){
ARSceneMesh arSceneMesh = arframe.acquireSceneMesh();
// Create a method for updating data and pass arSceneMesh to the method.
updateSceneMeshData(arSceneMesh);
// Release arSceneMesh when it is no longer in use.
arSceneMesh.release();
}
// Implement this method to update data.
Public void updateSceneMeshData(ARSceneMesh sceneMesh){
// Obtain an array containing mesh vertex coordinates of the environment mesh in the current view.
FloatBuffer meshVertices = sceneMesh.getVertices();
// Obtain an array containing the indexes of the vertices in the mesh triangle plane in the current view.
IntBuffer meshTriangleIndices = sceneMesh.getTriangleIndices();
}
}
3. Initialize the SceneMeshRenderManager class to provide render managers for external scenes, including render managers for virtual objects.
Code:
public class SceneMeshRenderManager implements GLSurfaceView.Render{
// Initialize the class for updating network data and performing rendering.
private SceneMeshDisplay mSceneMesh = new SceneMeshDisplay();
// Initialize the class for drawing virtual objects.
Private HitResultDisplay mHitResultDisplay = new HitResultDisplay();
// Implement the onSurfaceCreated() method.
public void onSurfaceCreated(){
// Pass context to the mSceneMesh and mHitResultDisplay classes.
mSceneMesh.init(mContext);
mHitResultDisplay.init(mContext);
}
// Implement the onDrawFrame() method.
public void onDrawFrame(){
// Configure camera using the ARSession object.
mArSession.setCameraTexTureName();
ARFrame arFrame = mArSession.update();
ARCamera arCamera = arframe.getCamera();
// Pass the data required for the SceneMeshDisplay class.
mSceneMesh.onDrawFrame(arframe,viewmtxs,projmtxs);
}
}
4. Initialize the SceneMeshActivity class to implement display functions.
Code:
public class SceneMeshActivity extends BaseActivity{
// Provides render managers for external scenes, including those for virtual objects.
private ScemeMeshRenderManager mSceneMeshRenderManager;
// Manages the entire running status of AR Engine.
private ARSession mArSession;
// Initialize some classes and objects.
protected void onCreate(Bundle savedInstanceState){
mSceneMeshRenderManager = new SceneMeshRenderManager();
}
// Initialize ARSession in the onResume method.
protected void onResume(){
// Initialize ARSession.
mArSession = new ARSession(this.getApplicationContext());
// Create an ARWorldTrackingConfig object based on the session parameters.
ARConfigBase config = new ARWorldTrackingConfig(mArSession);
// Pass ARSession to SceneMeshRenderManager.
mSceneMeshRenderManager.setArSession(mArSession);
// Enable the mesh, and call the setEnableItem method using config.
config.setEnableItem(ARConfigBase.ENABLE_MESH | ARConfigBase.ENABLE_DEPTH);
}
}
ConclusionAR bridges the real and the virtual worlds, to make jaw-dropping interactive experiences accessible to all users. That is why so many mobile app developers have opted to build AR capabilities into their apps. Doing so can give your app a leg up over the competition.
When developing such an app, you will need to incorporate a range of capabilities, such as hand recognition, motion tracking, hit test, plane detection, and lighting estimate. Fortunately, you do not have to do any of this on your own. Integrating an SDK can greatly streamline the process, and provide your app with many capabilities that are fundamental to seamless and immersive AR interactions. If you are not sure how to deal with the pass-through issue, or your app is not good at presenting virtual objects naturally in the real world, AR Engine can do a lot of heavy lifting for you. After being integrated with this toolkit, your app will be able to better perceive the physical environments around virtual objects, and therefore give characters the freedom to move around as if they are navigating real spaces.
ReferencesAR Engine Development Guide
Software and Hardware Requirements of AR Engine Features
AR Engine Sample Code