Hello everyone, in this article we will create a demo application using the core features of Panorama Kit to help you better understand the implementation of this kit. This app will simply allow users to quickly display interactive viewing of panoramic images in simulated 3D space.
What is Panorama Kit and Why Should We Use It?
By integrating the HUAWEI Panorama Kit, your app can quickly display interactive viewing of 360-degree spherical or cylindrical panoramic images in simulated 3D space, delivering immersive 360-degree showcases and interactions to users.
Features
360-degree spherical panorama, partially 360-degree spherical panorama, and cylindrical panorama
Parsing and display of panorama images in JPG, JPEG, and PNG formats
Interactive viewing of panoramic videos
Panorama view adjustment by swiping the screen or rotating the mobile phone to any degree
Interactive viewing of 360-degree spherical panoramas shot by mainstream panoramic cameras
Flexible use of panorama services, such as displaying panoramas in a certain area of your app, instead of in a window outside the app, made possible with in-app panorama display APIs
Integration Preparations
Panorama SDK can be directly called by devices, without connecting to AppGallery Connect so you don’t need agconnect-services.json. You just need the add Maven repository address in the project-level build.gradle file.
Software Requirements
Android Studio 3.0 or later
Java SDK 1.7 or later
HMS Core (APK) 4.0.0.300 or later
HMS Core SDK 4.0.0.300 or later
Android 5.0 or later
Supported Locations
To see all supported locations, please refer to HUAWEI Panorama Kit Supported Locations
Configuration
Add Maven repository address in the project-level build.gradle file
Code:
buildscript {
repositories {
...
maven { url 'https://developer.huawei.com/repo/' }
}
...
}
allprojects {
repositories {
...
maven { url 'https://developer.huawei.com/repo/' }
}
}
...
And add build dependencies in the app-level build.gradle file.
Code:
implementation 'com.huawei.hms:panorama:4.0.4.301'
Keep in mind that the SDK version used in this article may be not the latest one. If you want to use the latest SDK version, please refer to Version Change History.
Finally, add read and write permissions in the AndroidManifest.xml.
Code:
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
Remember, in Android 6.0 or later, you need to dynamically request the permissions.
Implementation
There are 3 different ways to call Panorama View;
Load a panorama image without a specified panorama type.
Load a panorama image with a specified panorama type.
Start an activity, which shows how to call panorama image display using the PanoramaLocalInterface.
Now we’ll go through each of these ways to implement Panorama Kit.
Load a panorama image without a specified panorama type
If you don’t specify a panorama type, users will be able to move both horizontally and vertically. But if suitable, it’s best that you specify a panorama type to give your users a better viewing experience.
Code:
private fun loadImageWithoutType() {
Panorama.getInstance().loadImageInfoWithPermission(
this,
Uri.parse("android.resource://$packageName/${R.raw.pano1}")
)
.setResultCallback(ResultCallbackImpl([email protected]))
}
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Load a panorama image with a specified panorama type
Here we specify the panorama type of IMAGE_TYPE_RING to improve the viewing experience of this particular image.
Code:
private fun loadImageWithType() {
Panorama.getInstance().loadImageInfoWithPermission(
this,
Uri.parse("android.resource://$packageName/${R.raw.pano1}"),
PanoramaInterface.IMAGE_TYPE_RING
).setResultCallback(ResultCallbackImpl([email protected]))
}
Load panorama image inside a custom activity
Finally, if you want to show panorama view in a custom activity of yours, you can do so by sending the necessary data detailed in the code below and calling the local interface api of Panorama Kit in that specific activity.
Code:
private fun panoramaWithLocalInterface() {
val intent = Intent([email protected], LocalInterfaceActivity::class.java)
intent.apply {
data = Uri.parse("android.resource://$packageName/${R.raw.pano1}");
putExtra("PanoramaType", PanoramaInterface.IMAGE_TYPE_SPHERICAL)
}
startActivity(intent)
}
Init local interface and set image uri and type, obtain the view and add it to your layout. And listen on the changeInputButton event to change the control mode.
Code:
private fun callLocalApi(imageUri: Uri?, imageType: Int) {
mLocalInterface = Panorama.getInstance().getLocalInstance(this)
if (mLocalInterface.init() == 0 && mLocalInterface.setImage(imageUri, imageType) == 0) {
val view: View = mLocalInterface.view
relativeLayout.addView(view)
view.setOnTouchListener([email protected])
changeInputButton.apply {
bringToFront()
setOnClickListener([email protected])
}
} else {
Log.e(TAG, "local api error")
}
}
Code:
override fun onClick(v: View?) {
if (v?.id == R.id.changeInputButton) {
if (mIsGyroEnabled) {
mIsGyroEnabled = false
mLocalInterface.setControlMode(PanoramaInterface.CONTROL_TYPE_TOUCH)
} else {
mIsGyroEnabled = true
mLocalInterface.setControlMode(PanoramaInterface.CONTROL_TYPE_POSE)
}
}
}
Send the view touch control event to the HMS Core Panorama SDK. (This step assumes that the touch mode is enabled.)
Code:
@SuppressLint("ClickableViewAccessibility")
override fun onTouch(v: View?, event: MotionEvent?): Boolean {
mLocalInterface.let {
mLocalInterface.updateTouchEvent(event)
}
return true
}
Keep in mind: After loading a panoramic image, you can call the setImage API to replace it with another one, if necessary.
Code:
mLocalInterface.setImage(resourceUri, imageType)
And that’s it! These were the 3 ways of implementing Panorama Kit into your app. You can find the full source code of this demo application in the link below.
https://github.com/tolunayozturk/hms-panoramakitdemo
As always, thank you for reading this article and using HMS. Please let me know if you have any questions!
How much time it takes to integrate Panoroma Kit in android project and can you give me one use case ?
Related
Introduction
Sometimes, we can't help taking photos to keep our unforgettable moments in life. But most of us are not professional photographers or models, so our photographs can end up falling short of our expectations. So, how can we produce more impressive snaps? If you have an image processing app on your phone, it can automatically detect faces in a photo, and you can then adjust the image until you're happy with it. So, after looking around online, I found HUAWEI ML Kit's face detection capability. By integrating this capability, you can add beautification functions to your apps. Have a try!
Application Scenarios
ML Kit's face detection capability detects up to 855 facial keypoints and returns the coordinates for the face's contour, eyebrows, eyes, nose, mouth, and ears, as well as the angle of the face. Once you've integrated this capability, you can quickly create beauty apps and enable users to add fun facial effects and features to their images.
Face detection also detects whether the subject's eyes are open, whether they're wearing glasses or a hat, whether they have a beard, and even their gender and age. This is useful if you want to add a parental control function to your app which prevents children from getting too close to their phone, or staring at the screen for too long.
In addition, face detection can detect up to seven facial expressions, including smiling, neutral, angry, disgusted, frightened, sad, and surprised faces. This is great if you want to create apps such as smile-cameras.
You can integrate any of these capabilities as needed. At the same time, face detection supports image and video stream detection, cross-frame face tracking, and multi-face detection. It really is powerful! Now, let's see how to integrate this capability.
Face Detection Development
1. Preparations
You can find detailed information about the preparations you need to make on the HUAWEI Developers-Development Process. Here, we'll just look at the most important procedures.
1.1 Configure the Maven Repository Address in the Project-Level build.gradle File
Code:
buildscript {
repositories {
...
maven {url 'https://developer.huawei.com/repo/'}
}
}
dependencies {
...
classpath 'com.huawei.agconnect:agcp:1.3.1.300'
}
allprojects {
repositories {
...
maven {url 'https://developer.huawei.com/repo/'}
}
}
1.2 Add Configurations to the File Header
After integrating the SDK, add the following configuration to the file header:
Code:
apply plugin: 'com.android.application'
apply plugin: 'com.huawei.agconnect'
1.3 Configure SDK Dependencies in the App-Level build.gradle File
Code:
dependencies{
// Import the base SDK.
implementation 'com.huawei.hms:ml-computer-vision-face:2.0.1.300'
// Import the contour and keypoint detection model package.
implementation 'com.huawei.hms:ml-computer-vision-face-shape-point-model:2.0.1.300'
// Import the facial expression detection model package.
implementation 'com.huawei.hms:ml-computer-vision-face-emotion-model:2.0.1.300'
// Import the facial feature detection model package.
implementation 'com.huawei.hms:ml-computer-vision-face-feature-model:2.0.1.300'
}
1.4 Add these Statements to the AndroidManifest.xml File so the Machine Learning Model can Update Automatically
Code:
<manifest
...
<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value= "face"/>
...
</manifest>
1.5 Apply for Camera Permission
Code:
<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" />
2. Code Development
2.1 Create a Face Analyzer by Using the Default Parameter Configurations
Code:
analyzer = MLAnalyzerFactory.getInstance().getFaceAnalyzer();
2.2 Create an MLFrame Object by Using the android.graphics.Bitmap for the Analyzer to Detect Images
Code:
MLFrame frame = MLFrame.fromBitmap(bitmap);
2.3 Call the asyncAnalyseFrame Method to Perform Face Detection
Code:
Task<List<MLFace>> task = analyzer.asyncAnalyseFrame(frame);
task.addOnSuccessListener(new OnSuccessListener<List<MLFace>>() {
@Override
public void onSuccess(List<MLFace> faces) {
// Detection success. Obtain the face keypoints.
}
}).addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(Exception e) {
// Detection failure.
}
});
2.4 Use the Progress Bar to Process the Face in the Image
Call the magnifyEye and smallFaceMesh methods to implement the eye-enlarging algorithm and face-shaping algorithm.
Code:
private SeekBar.OnSeekBarChangeListener onSeekBarChangeListener = new SeekBar.OnSeekBarChangeListener() {
@Override
public void onProgressChanged(SeekBar seekBar, int progress, boolean fromUser) {
switch (seekBar.getId()) {
case R.id.seekbareye: // When the progress bar of the eye enlarging changes, ...
case R.id.seekbarface: // When the progress bar of the face shaping changes, ...
}
}
2.5 Release the Analyzer After the Detection is Complete
Code:
try {
if (analyzer != null) {
analyzer.stop();
}
} catch (IOException e) {
Log.e(TAG, "e=" + e.getMessage());
}
Demo
Now, let's see what it can do. Pretty cool, right?
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
What makes a beautiful game loaded with top notch graphics, what makes virtual reality meeting rooms, what makes moving objects and space representation?
I am sure, this would have made you wonder!!
This is possible with 3D image rendering.
3D image rendering is a process to convert any 3D model (which is also done using computer software) into a 2D view on a compute/mobile screen, however ensuring the actual feel of the 3D model.
There are many tools and software to do the conversions on a system but very few supports the same process on an Android Mobile device.
Huawei Scene Kit is a wonderful set of API’s which allows 3D models to display on a mobile screen while converting the 3D models adopting physical based rendering (PBR) pipelines to achieve realistic rendering effects.
Features
Uses physical based rendering for 3D scenes with quality immersion to provide realistic effects.
Offers API's for different scenarios for easier 3D scene construction which makes the development super easy.
Uses less power for equally powerful scene construction using a 3D graphics rendering framework and algorithms to provide high performance experience.
How does Huawei Scene Kit Works
The SDK of Scene Kit, after being integrated into your app, will send a 3D materials loading request to the Scene Kit APK in HMS Core (APK). Then the Scene Kit APK will verify, parse, and render the materials.
Scene Kit also provides a Full-SDK, which you can integrate into your app to access 3D graphics rendering capabilities, even though your app runs on phones without HMS Core.
Scene Kit uses the Entity Component System (ECS) to reduce coupling and implement multi-threaded parallel rendering.
Real-time PBR pipelines are adopted to make rendered images look like in a real world.
The general-purpose GPU Turbo is supported to significantly reduce power consumption.
Huawei Scene Kit supports 3 types of different rendering mechanism
SceneView
ARView
FaceView
Development Overview
Prerequisite
Must have a Huawei Developer Account
Must have Android Studio 3.0 or later
Must have a Huawei phone with HMS Core 4.0.2.300 or later
EMUI 3.0 or later
Software Requirements
Java SDK 1.7 or later
Android 5.0 or later
Supported Devices
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Supported 3D Formates
Materials to be rendered: glTF, glb
glTF textures: png, jpeg
Skybox materials: dds (cubemap)
Lighting maps: dds (cubemap)
Note: These materials allow for a resolution of up to 4K.
Preparation
Create an app or project in the Huawei app gallery connect.
Provide the SHA Key and App Package name of the project in App Information Section and enable the required API.
Create an Android project.
Note: Scene Kit SDK can be directly called by devices, without connecting to AppGallery Connect and hence it is not mandatory to download and integrate the agconnect-services.json.
Integration
Add below to build.gradle (project)file, under buildscript/repositories and allprojects/repositories.
Code:
Maven {url 'http://developer.huawei.com/repo/'}
Add below to build.gradle (app) file, under dependencies.
To use the SDK of Scene Kit, add the following dependencies:
Code:
dependences {
implementation 'com.huawei.scenekit:sdk:{5.0.2.302}'
}
To use the Full-SDK, add the following dependencies:
Code:
dependencies {
implementation 'com.huawei.scenekit:full-sdk:{5.0.2.302}'
Adding permissions
Code:
<uses-permission android:name="android.permission.CAMERA " />
Development Process
This article focus on demonstrating the capabilities of Huawei’s Scene Kit: Scene View API’s.
Here is the example which explains how we can integrate a simple 3D Model (Downloaded from the free website: https://sketchfab.com/ ) with Scene View API’s to showcase the space view of AMAZON Jungle.
Download and add supported 3D model into the App’s asset folder.
Main Activity
Launcher activity for the application which has a button to navigate and display the space view.
Code:
xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:background="@drawable/dbg"
android:orientation="vertical">
<Button
android:layout_width="match_parent"
android:layout_height="match_parent"
android:background="@drawable/btn"
android:onClick="onBtnSceneViewDemoClicked"
android:text="@string/btn_text_scene_kit_demo"
android:textColor="@color/upsdk_white"
android:textSize="20sp"
android:textStyle="bold" />
</LinearLayout>
Space View Activity
It’s a java class which holds the logic to render the 3D models using the API’s and display them in a surface view.
Code:
import android.content.Context;
import android.graphics.Canvas;
import android.util.AttributeSet;
import android.view.MotionEvent;
import android.view.SurfaceHolder;
import com.huawei.hms.scene.sdk.SceneView;
public class SceneSampleView extends SceneView {
/* Constructor - used in new mode.*/
public SceneSampleView(Context context) {
super(context);
}
public SceneSampleView(Context context, AttributeSet attributeSet) {
super(context, attributeSet);
}
@Override
public void surfaceCreated(SurfaceHolder holder) {
super.surfaceCreated(holder);
// Loads the model of a scene by reading files from assets.
loadScene("SceneView/scene.gltf");
}
@Override
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
super.surfaceChanged(holder, format, width, height);
}
@Override
public void surfaceDestroyed(SurfaceHolder holder) {
super.surfaceDestroyed(holder);
}
@Override
public boolean onTouchEvent(MotionEvent motionEvent) {
return super.onTouchEvent(motionEvent);
}
@Override
public void onDraw(Canvas canvas) {
super.onDraw(canvas);
}
}
Results
Conclusion
We learnt a simple application which demonstrate the Scene View API’s and showcase how easy it is to have the 3D models shown on the applications.
More information, you can visit https://forums.developer.huawei.com/forumPortal/en/topicview?tid=0201382255415980474&fid=0101187876626530001
1. Overview
Camera functions on today's phones are so powerful that they now involve advanced imaging processing algorithms, as well as the camera hardware itself. But even so, users often need to continually adjust the image parameters, a process that can be extremely frustrating, even when it results in an ideal shot. For the most of us who are not professional photographers, images often end up falling woefully short of expectations. Given these universal challenges, there's incredible demand for apps that can enhance image effects automatically, by accounting for the user's surroundings. HUAWEI ML Kit's scene detection service helps bring such apps to life, by detecting 102 visual features, including beaches, sky scenes, food, night scenes, plants, and common buildings. The service can automatically adjust the parameters corresponding to the image matrix, helping build proactive, intelligent, and efficient apps. Let's take a look at how it does this.
2. Effects
For a city nightscape, the service will accurately detect the night scene, and enhance the brightness in bright areas, as well as the darkness in dark areas, rendering a highly-pleasing contrast.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
For a sky image, the service will brighten the sky with an enhanced matrix.
The effects are similar for photos of flowers and plants.
The effects tend to vary, depending on the specific image. Users are free to apply their favorite effects, or mix-and-max them with filters.
Now, let's take a look at how the service can be integrated.
1. Development Process
3.1 Create a scene detection analyzer instance.
Code:
MLSceneDetectionAnalyzer analyzer = MLSceneDetectionAnalyzerFactory.getInstance().getSceneDetectionAnalyzer();
3.2 Create an MLFrame object by using android.graphics.Bitmap. JPG, JPEG, PNG, and BMP images are currently supported.
Code:
MLFrame frame = new MLFrame.Creator().setBitmap(bitmap).create();
3.3 Scene detection
Code:
Task<List<MLSceneDetection>> task = this.analyzer.asyncAnalyseFrame(frame);
task.addOnSuccessListener(new OnSuccessListener<List<MLSceneDetection>>() {
@Override
public void onSuccess(List<MLSceneDetection> sceneInfos) {
// Processing logic for scene detection success.
}
}).addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(Exception e) {
// Processing logic for scene detection failure.
if (e instanceof MLException) {
MLException exception = (MLException) e;
// Obtain the result code.
int errCode = exception.getErrCode();
// Obtain the error information.
String message = exception.getMessage();
} else {
// Other errors.
}
}
});
3.4 Stop the analyzer to release detection resources when the detection ends.
Code:
if (analyzer != null) {
analyzer.stop();
}
The Maven repository address.
buildscript {
repositories {
maven { url 'https://developer.huawei.com/repo/' }
}
}
allprojects {
repositories {
maven { url 'https://developer.huawei.com/repo/' }
}
}
Import the SDK.
Code:
dependencies {
// Scene detection SDK.
implementation 'com.huawei.hms:ml-computer-vision-scenedetection:2.0.3.300'
// Scene detection model.
implementation 'com.huawei.hms:ml-computer-vision-scenedetection-model:2.0.3.300'
}
Manifest files
Code:
<manifest
...
<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value="1" />
...
</manifest>
Permissions
Code:
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.autofocus" />
Submit a dynamic permission application.
Code:
if (!(ActivityCompat.checkSelfPermission(this, Manifest.permission.CAMERA) == PackageManager.PERMISSION_GRANTED)) {
requestCameraPermission();
}
4. Learn More
ML Kit's scene detection service also comes equipped with intelligent features, such as smart album management and image searching, which allows users to find and sort images in a refreshingly natural and intuitive way.
Reference
Official website of Huawei Developers
Development Guide
HMS Core official community on Reddit
Demo and sample code
Discussions on Stack Overflow
Is there any restriction using the Kit ?
Very nice
It's so rewarding to set the model you've created into motion. If only there were an easy way to do this… well, actually there is!
I had long sought out this kind of solution, and then voila! I got my hands on motion capture, a capability from HMS Core 3D Modeling Kit, which comes with technologies like human body detection, model acceleration, and model compression, as well as a monocular human pose estimation algorithm from the deep learning perspective.
Crucially, this capability does NOT require advanced devices — a mobile phone with an RGB camera is good enough on its own. The camera captures 3D data from 24 key skeletal points on the body, which the capability uses to seamlessly animate a model.
What makes the motion capture capability even better is its straightforward integration process, which I'd like to share with you.
Application ScenariosMotion capture is ideal for 3D content creation for gaming, film & TV, and healthcare, among other similar fields. It can be used to animate characters and create videos for user generated content (UGC) games, animate virtual streamers in real time, and provide injury rehab, to cite just a few examples.
Integration ProcessPreparationsRefer to the official instructions to complete all necessary preparations.
Configuring the Project
Before developing the app, there are a few more things you'll need to do: Configure app information in AppGallery Connect; make sure that the Maven repository address of the 3D Modeling SDK has been configured in the project, and that the SDK has been integrated.
1. Create a motion capture engine
Java:
// Set necessary parameters as needed.
Modeling3dMotionCaptureEngineSetting setting = new Modeling3dMotionCaptureEngineSetting.Factory()
// Set the detection mode.
// Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON_QUATERNION: skeleton point quaternions of a human pose.
// Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON: skeleton point coordinates of a human pose.
.setAnalyzeType(Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON_QUATERNION
| Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON)
.create();
Modeling3dMotionCaptureEngine engine = Modeling3dMotionCaptureEngineFactory.getInstance().getMotionCaptureEngine(setting);
Modeling3dFrame encapsulates video frame or static image data sourced from a camera, as well as related data processing logic.
Customize the logic for processing the input video frames, to convert them to the Modeling3dFrame object for detection. The video frame format can be NV21.
Use android.graphics.Bitmap to convert the input image to the Modeling3dFrame object for detection. The image format can be JPG, JPEG, or PNG.
Java:
// Create a Modeling3dFrame object using a bitmap.
Modeling3dFrame frame = Modeling3dFrame.fromBitmap(bitmap);
// Create a Modeling3dFrame object using a video frame.
Modeling3dFrame.Property property = new Modeling3dFrame.Property.Creator().setFormatType(ImageFormat.NV21)
// Set the frame width.
.setWidth(width)
// Set the frame height.
.setHeight(height)
// Set the video frame rotation angle.
.setQuadrant(quadant)
// Set the video frame number.
.setItemIdentity(framIndex)
.create();
Modeling3dFrame frame = Modeling3dFrame.fromByteBuffer(byteBuffer, property);
2. Call the asynchronous or synchronous API for motion detection.
Sample code for calling the asynchronous API asyncAnalyseFrame:
Java:
Task<List<Modeling3dMotionCaptureSkeleton>> task = engine.asyncAnalyseFrame(frame);
task.addOnSuccessListener(new OnSuccessListener<List<Modeling3dMotionCaptureSkeleton>>() {
@Override
public void onSuccess(List<Modeling3dMotionCaptureSkeleton> results) {
// Detection success.
}
}).addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(Exception e) {
// Detection failure.
}
});
Sample code for calling the synchronous API analyseFrame
Java:
SparseArray<Modeling3dMotionCaptureSkeleton> sparseArray = engine.analyseFrame(frame);
for (int i = 0; i < sparseArray.size(); i++) {
// Process the detection result.
};
3. Stop the motion capture engine to release detection resources, once the detection is complete
Java:
try {
if (engine != null) {
engine.stop();
}
} catch (IOException e) {
// Handle exceptions.
}
Result
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
To learn more, please visit:
>> 3D Modeling Kit official website
>> 3D Modeling Kit Development Guide
>> Reddit to join developer discussions
>> GitHub to download the sample code
>> Stack Overflow to solve integration problems
Follow our official account for the latest HMS Core-related news and updates.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Emojis are a must-have tool in today's online communications as they help add color to text-based chatting and allow users to better express the emotions behind their words. Since the number of preset emojis is always limited, many apps now allow users to create their own custom emojis to keep things fresh and exciting.
For example, in a social media app, users who do not want to show their faces when making video calls can use an animated character to protect their privacy, with their facial expressions applied to the character; in a live streaming or e-commerce app, virtual streamers with realistic facial expressions are much more likely to attract watchers; in a video or photo shooting app, users can control the facial expressions of an animated character when taking a selfie, and then share the selfie via social media; and in an educational app for kids, a cute animated character with detailed facial expressions will make online classes much more fun and engaging for students.
I myself am developing such a messaging app. When chatting with friends and wanting to express themselves in ways other than words, users of my app can take a photo to create an emoji of themselves, or of an animated character they have selected. The app will then identify users' facial expressions, and apply their facial expressions to the emoji. In this way, users are able to create an endless amount of unique emojis. During the development of my app, I used the capabilities provided by HMS Core AR Engine to track users' facial expressions and convert the facial expressions into parameters, which greatly reduced the development workload. Now I will show you how I managed to do this.
ImplementationAR Engine provides apps with the ability to track and recognize facial expressions in real time, which can then be converted into facial expression parameters and used to accurately control the facial expressions of virtual characters.
Currently, AR Engine provides 64 facial expressions, including eyelid, eyebrow, eyeball, mouth, and tongue movements. It supports 21 eye-related movements, including eyeball movement and opening and closing the eyes; 28 mouth movements, including opening the mouth, puckering, pulling, or licking the lips, and moving the tongue; as well as 5 eyebrow movements, including raising or lowering the eyebrows.
DemoFacial expression based emoji
Development ProcedureRequirements on the Development EnvironmentJDK: 1.8.211 or later
Android Studio: 3.0 or later
minSdkVersion: 26 or later
targetSdkVersion: 29 (recommended)
compileSdkVersion: 29 (recommended)
Gradle version: 6.1.1 or later (recommended)
Make sure that you have downloaded the AR Engine APK from AppGallery and installed it on the device.
Test device: see Software and Hardware Requirements of AR Engine Features
If you need to use multiple HMS Core kits, use the latest versions required for these kits.
Preparations1. Before getting started, you will need to register as a Huawei developer and complete identity verification on HUAWEI Developers. You can click here to find out the detailed registration and identity verification procedure.
2. Before development, integrate the AR Engine SDK via the Maven repository into your development environment.
3. The procedure for configuring the Maven repository address in Android Studio varies for Gradle plugin earlier than 7.0, Gradle plugin 7.0, and Gradle plugin 7.1 or later. You need to configure it according to the specific Gradle plugin version.
4. Take Gradle plugin 7.0 as an example:
Open the project-level build.gradle file in your Android Studio project and configure the Maven repository address.
Go to buildscript > repositories and configure the Maven repository address for the SDK.
Code:
buildscript {
repositories {
google()
jcenter()
maven {url "https://developer.huawei.com/repo/" }
}
}
Open the project-level settings.gradle file and configure the Maven repository address for the HMS Core SDK.
Code:
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
repositories {
google()
jcenter()
maven {url "https://developer.huawei.com/repo/" }
}
}
}
5. Add the following build dependency in the dependencies block.
Code:
dependencies {
implementation 'com.huawei.hms:arenginesdk:{version}
}
App Development1. Check whether AR Engine has been installed on the current device. If yes, your app can run properly. If not, you need to prompt the user to install it, for example, by redirecting the user to AppGallery. The sample code is as follows:
Code:
boolean isInstallArEngineApk =AREnginesApk.isAREngineApkReady(this);
if (!isInstallArEngineApk) {
// ConnectAppMarketActivity.class is the activity for redirecting users to AppGallery.
startActivity(new Intent(this, com.huawei.arengine.demos.common.ConnectAppMarketActivity.class));
isRemindInstall = true;
}
2. Create an AR scene. AR Engine supports five scenes, including motion tracking (ARWorldTrackingConfig), face tracking (ARFaceTrackingConfig), hand recognition (ARHandTrackingConfig), human body tracking (ARBodyTrackingConfig), and image recognition(ARImageTrackingConfig).
The following takes creating a face tracking scene by calling ARFaceTrackingConfig as an example.
Code:
// Create an ARSession object.
mArSession = new ARSession(this);
// Select a specific Config to initialize the ARSession object based on the application scenario.
ARFaceTrackingConfig config = new ARFaceTrackingConfig(mArSession);
Set scene parameters using the config.setXXX method.
Code:
// Set the camera opening mode, which can be external or internal. The external mode can only be used in ARFace. Therefore, you are advised to use the internal mode.
mArConfig.setImageInputMode(ARConfigBase.ImageInputMode.EXTERNAL_INPUT_ALL);
3. Set the AR scene parameters for face tracking and start face tracking.
Code:
mArSession.configure(mArConfig);
mArSession.resume();
4. Initialize the FaceGeometryDisplay class to obtain the facial geometric data and render the data on the screen.
Code:
public class FaceGeometryDisplay {
// Initialize the OpenGL ES rendering related to face geometry, including creating the shader program.
void init(Context context) {...
}
}
5. Initialize the onDrawFrame method in the FaceGeometryDisplay class, and call face.getFaceGeometry() to obtain the face mesh.
Code:
public void onDrawFrame(ARCamera camera, ARFace face) {
ARFaceGeometry faceGeometry = face.getFaceGeometry();
updateFaceGeometryData(faceGeometry);
updateModelViewProjectionData(camera, face);
drawFaceGeometry();
faceGeometry.release();
}
6. Initialize updateFaceGeometryData() in the FaceGeometryDisplay class.
Pass the face mesh data for configuration and set facial expression parameters using OpenGl ES.
Code:
private void updateFaceGeometryData (ARFaceGeometry faceGeometry) {
FloatBuffer faceVertices = faceGeometry.getVertices();
FloatBuffer textureCoordinates =faceGeometry.getTextureCoordinates();
// Obtain an array consisting of face mesh texture coordinates, which is used together with the vertex data returned by getVertices() during rendering.
}
7. Initialize the FaceRenderManager class to manage facial data rendering.
Code:
public class FaceRenderManager implements GLSurfaceView.Renderer {
public FaceRenderManager(Context context, Activity activity) {
mContext = context;
mActivity = activity;
}
// Set ARSession to obtain the latest data.
public void setArSession(ARSession arSession) {
if (arSession == null) {
LogUtil.error(TAG, "Set session error, arSession is null!");
return;
}
mArSession = arSession;
}
// Set ARConfigBase to obtain the configuration mode.
public void setArConfigBase(ARConfigBase arConfig) {
if (arConfig == null) {
LogUtil.error(TAG, "setArFaceTrackingConfig error, arConfig is null.");
return;
}
mArConfigBase = arConfig;
}
// Set the camera opening mode.
public void setOpenCameraOutsideFlag(boolean isOpenCameraOutsideFlag) {
isOpenCameraOutside = isOpenCameraOutsideFlag;
}
...
@Override
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
mFaceGeometryDisplay.init(mContext);
}
}
8. Implement the face tracking effect by calling methods like setArSession and setArConfigBase of FaceRenderManager in FaceActivity.
Code:
public class FaceActivity extends BaseActivity {
@Override
protected void onCreate(Bundle savedInstanceState) {
mFaceRenderManager = new FaceRenderManager(this, this);
mFaceRenderManager.setDisplayRotationManage(mDisplayRotationManager);
mFaceRenderManager.setTextView(mTextView);
glSurfaceView.setRenderer(mFaceRenderManager);
glSurfaceView.setRenderMode(GLSurfaceView.RENDERMODE_CONTINUOUSLY);
}
}
ConclusionEmojis allow users to express their moods and excitement in a way words can't. Instead of providing users with a selection of the same old boring preset emojis that have been used a million times, you can now make your app more fun by allowing users to create emojis themselves! Users can easily create an emoji with their own smiles, simply by facing the camera, selecting an animated character they love, and smiling. With such an ability to customize emojis, users will be able to express their feelings in a more personalized and interesting manner. If you have any interest in developing such an app, AR Engine is a great choice for you. With accurate facial tracking capabilities, it is able to identify users' facial expressions in real time, convert the facial expressions into parameters, and then apply them to virtual characters. Integrating the capability can help you considerably streamline your app development process, leaving you with more time to focus on how to provide more interesting features to users and improve your app's user experience.
ReferenceAR Engine Sample Code
Face Tracking Capability