Static biometric verification is a feature of HMS Core ML Kit, which captures faces in real time and can determine whether a face belongs to a real person or not, without prompting the user to move their head or face. In this way, the service helps deliver a convenient user experience that wins positive feedback.
Technical PrinciplesStatic biometric verification requires an RGB camera and is able to differentiate between a real person's face and a spoof attack (such as an image or screenshot of a face and a face mask), through details (such as the moiré pattern or reflection on a paper photo) in the image captured by the camera. The service supports data from a wide array of scenarios, including different lighting conditions, face accessories, genders, hairstyles, and mask materials. The service analyzes a face's surroundings to detect suspicious environments.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
The static biometric verification model adopts the lightweight convolutional module, and linear computation is converted to a single convolutional module or a fully connected layer in the inference phase through reparameterization. The MindSpore Lite inference framework is used for model deployment, which crops operators. The model's package size is then shrunk, making it more convenient for integration.
Application ScenariosLiveness detection is usually used before face verification. For example, when a user uses facial recognition to unlock their phone, liveness detection first determines whether the captured face is real or not. If yes, face verification will then check whether the face matches the one recorded in the system. These two technologies complement one another to protect a user's device from unauthorized access.
So it's safe to say that static biometric verification provides rigid protection for our apps, and I'm here to illustrate how this can be integrated.
Integration ProcedurePreparations
The detailed preparations are all provided in the official document for the service.
Two modes are available to call the service:
Call ModeLiveness Detection ProcessLiveness Detection UIFunctionDefault View ModeProcessed by ML KitProvided by ML KitDetermines whether a face is real or not.Customized View ModeProcessed by ML KitCustomDetermines whether a face is real or not.
Default View Mode
1. Create a callback to obtain the static biometric verification result
Java:
private MLLivenessCapture.Callback callback = new MLLivenessCapture.Callback() {
@Override
public void onSuccess(MLLivenessCaptureResult result) {
// Callback when verification is successful. The result indicates whether the face is of a real person.
}
@Override
public void onFailure(int errorCode) {
// Callback when verification fails. For example, the camera is abnormal (CAMERA_ERROR). Add the processing logic to deal with the failure.
}
};
2. Create a static biometric verification instance and start verification
Java:
MLLivenessCapture capture = MLLivenessCapture. getInstance();
capture.startDetect(activity, callback);
Customized View Mode
1. Create an MLLivenessDetectView instance and load it to the activity layout
Java:
/**
* i. Bind the camera preview screen to the remote view and set the liveness detection area.
* In the camera preview stream, static biometric verification determines whether a face is in the middle of the image. To improve the pass rate, you are advised to place the face frame in the middle of the screen and set the liveness detection area to be slightly larger than the face frame.
* ii. Set whether to detect the mask.
* iii. Set the result callback.
* iv. Load MLLivenessDetectView to the activity.
*/
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_liveness_custom_detection);
mPreviewContainer = findViewById(R.id.surface_layout);
// ObtainLLivenessDetectView
mlLivenessDetectView = new MLLivenessDetectView.Builder()
.setContext(this)
// Set whether to detect the mask.
.setOptions(MLLiveness DetectView.DETECT_MASK)
// Set the rectangle of the face frame relative to MLLivenessDetectView.
.setFaceRect(new Rect(0, 0, 0, 200))
// Set the result callback.
.setDetectCallback(new OnMLLivenessDetectCallback() {
@Override
public void onCompleted(MLLivenessCaptureResult result) {
// Callback when verification is complete.
}
@Override
public void onError(int error) {
// Callback when an error occurs during verification.
}
@Override
public void onInfo(int infoCode, Bundle bundle) {
// Callback when the verification prompt message is received. This message can be displayed on the UI.
// if(infoCode==MLLivenessDetectInfo.NO_FACE_WAS_DETECTED){
// No face is detected.
// }
// ...
}
@Override
public void onStateChange(int state, Bundle bundle) {
// Callback when the verification status changes.
// if(state==MLLivenessDetectStates.START_DETECT_FACE){
// Start face detection.
// }
// ...
}
}).build();
mPreviewContainer.addView(mlInteractiveLivenessDetectView);
mlInteractiveLivenessDetectView.onCreate(savedInstanceState);
}
2. Set a lifecycle listener for MLLivenessDetectView
Java:
@Override
protected void onDestroy() {
super.onDestroy();
mlLivenessDetectView.onDestroy();
}
@Override
protected void onPause() {
super.onPause();
mlLivenessDetectView.onPause();
}
@Override
protected void onResume() {
super.onResume();
mlLivenessDetectView.onResume();
}
@Override
protected void onStart() {
super.onStart();
mlLivenessDetectView.onStart();
}
@Override
protected void onStop() {
super.onStop();
mlLivenessDetectView.onStop();
}
To learn more, please visit:
>> HUAWEI Developers official website
>> Development Guide
>> Reddit to join developer discussions
>> GitHub to download the sample code
>> Stack Overflow to solve integration problems
Follow our official account for the latest HMS Core-related news and updates.
Related
"John, have you seen my glasses?"
Our old friend John, a programmer at Huawei, has a grandpa who despite his old age, is an avid reader. Leaning back, struggling to make out what was written on the newspaper through his glasses, but unable to take his eyes off the text — this was how my grandpa used to read, John explained.
Reading this way was harmful on his grandpa's vision, and it occurred to John that the ears could take over the role of "reading" from the eyes. He soon developed a text-reading app that followed this logic, recognizing and then reading out text from a picture. Thanks to this app, John's grandpa now can ”read” from the comfort of his rocking chair, without having to strain his eyes.
How to Implement
The user takes a picture of a text passage. The app then automatically identifies the location of the text within the picture, and adjusts the shooting angle to an angle directly facing the text.
The app recognizes and extracts the text from the picture.
The app converts the recognized text into audio output by leveraging text-to-speech technology.
These functions are easy to implement, when relying on three services in HUAWEI ML Kit: document skew correction, text recognition, and text to speech (TTS).
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Preparations
1. Configure the Huawei Maven repository address.
2. Add the build dependencies for the HMS Core SDK.
Code:
dependencies {
// Import the base SDK.
implementation 'com.huawei.hms:ml-computer-voice-tts:2.1.0.300'
// Import the bee voice package.
implementation 'com.huawei.hms:ml-computer-voice-tts-model-bee:2.1.0.300'
// Import the eagle voice package.
implementation 'com.huawei.hms:ml-computer-voice-tts-model-eagle:2.1.0.300'
// Import a PDF file analyzer.
implementation 'com.itextpdf:itextg:5.5.10'
}
Tap PREVIOUS or NEXT to turn to the previous or next page. Tap speak to start reading; tap it again to pause reading.
Development process
1. Create a TTS engine by using the custom configuration class MLTtsConfig. Here, on-device TTS is used as an example.
Java:
private void initTts() {
// Set authentication information for your app to download the model package from the server of Huawei.
MLApplication.getInstance().setApiKey(AGConnectServicesConfig.
fromContext(getApplicationContext()).getString("client/api_key"));
// Create a TTS engine by using MLTtsConfig.
mlTtsConfigs = new MLTtsConfig()
// Set the text converted from speech to English.
.setLanguage(MLTtsConstants.TTS_EN_US)
// Set the speaker with the English male voice (eagle).
.setPerson(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_MALE_EAGLE)
// Set the speech speed whose range is (0, 5.0]. 1.0 indicates a normal speed.
.setSpeed(.8f)
// Set the volume whose range is (0, 2). 1.0 indicates a normal volume.
.setVolume(1.0f)
// Set the TTS mode to on-device.
.setSynthesizeMode(MLTtsConstants.TTS_OFFLINE_MODE);
mlTtsEngine = new MLTtsEngine(mlTtsConfigs);
// Update the configuration when the engine is running.
mlTtsEngine.updateConfig(mlTtsConfigs);
// Pass the TTS callback function to the TTS engine to perform TTS.
mlTtsEngine.setTtsCallback(callback);
// Create an on-device TTS model manager.
manager = MLLocalModelManager.getInstance();
isPlay = false;
}
2. Create a TTS callback function for processing the TTS result.
Java:
MLTtsCallback callback = new MLTtsCallback() {
@Override
public void onError(String taskId, MLTtsError err) {
// Processing logic for TTS failure.
}
@Override
public void onWarn(String taskId, MLTtsWarn warn) {
// Alarm handling without affecting service logic.
}
@Override
// Return the mapping between the currently played segment and text. start: start position of the audio segment in the input text; end (excluded): end position of the audio segment in the input text.
public void onRangeStart(String taskId, int start, int end) {
// Process the mapping between the currently played segment and text.
}
@Override
// taskId: ID of a TTS task corresponding to the audio.
// audioFragment: audio data.
// offset: offset of the audio segment to be transmitted in the queue. One TTS task corresponds to a TTS queue.
// range: text area where the audio segment to be transmitted is located; range.first (included): start position; range.second (excluded): end position.
public void onAudioAvailable(String taskId, MLTtsAudioFragment audioFragment, int offset,
Pair<Integer, Integer> range, Bundle bundle) {
// Audio stream callback API, which is used to return the synthesized audio data to the app.
}
@Override
public void onEvent(String taskId, int eventId, Bundle bundle) {
// Callback method of a TTS event. eventId indicates the event name.
boolean isInterrupted;
switch (eventId) {
case MLTtsConstants.EVENT_PLAY_START:
// Called when playback starts.
break;
case MLTtsConstants.EVENT_PLAY_STOP:
// Called when playback stops.
isInterrupted = bundle.getBoolean(MLTtsConstants.EVENT_PLAY_STOP_INTERRUPTED);
break;
case MLTtsConstants.EVENT_PLAY_RESUME:
// Called when playback resumes.
break;
case MLTtsConstants.EVENT_PLAY_PAUSE:
// Called when playback pauses.
break;
// Pay attention to the following callback events when you focus on only the synthesized audio data but do not use the internal player for playback.
case MLTtsConstants.EVENT_SYNTHESIS_START:
// Called when TTS starts.
break;
case MLTtsConstants.EVENT_SYNTHESIS_END:
// Called when TTS ends.
break;
case MLTtsConstants.EVENT_SYNTHESIS_COMPLETE:
// TTS is complete. All synthesized audio streams are passed to the app.
isInterrupted = bundle.getBoolean(MLTtsConstants.EVENT_SYNTHESIS_INTERRUPTED);
break;
default:
break;
}
}
};
3. Extract text from a PDF file.
Java:
private String loadText(String path) {
String result = "";
try {
PdfReader reader = new PdfReader(path);
result = result.concat(PdfTextExtractor.getTextFromPage(reader,
mCurrentPage.getIndex() + 1).trim() + System.lineSeparator());
reader.close();
} catch (IOException e) {
showToast(e.getMessage());
}
// Obtain the position of the header.
int header = result.indexOf(System.lineSeparator());
// Obtain the position of the footer.
int footer = result.lastIndexOf(System.lineSeparator());
if (footer != 0){
// Do not display the text in the header and footer.
return result.substring(header, footer - 5);
}else {
return result;
}
}
4. Perform TTS in on-device mode.
Java:
// Create an MLTtsLocalModel instance to set the speaker so that the language model corresponding to the speaker can be downloaded through the model manager.
MLTtsLocalModel model = new MLTtsLocalModel.Factory(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_MALE_EAGLE).create();
manager.isModelExist(model).addOnSuccessListener(new OnSuccessListener<Boolean>() {
@Override
public void onSuccess(Boolean aBoolean) {
// If the model is not downloaded, call the download API. Otherwise, call the TTS API of the on-device engine.
if (aBoolean) {
String source = loadText(mPdfPath);
// Call the speak API to perform TTS. source indicates the text to be synthesized.
mlTtsEngine.speak(source, MLTtsEngine.QUEUE_APPEND);
if (isPlay){
// Pause playback.
mlTtsEngine.pause();
tv_speak.setText("speak");
}else {
// Resume playback.
mlTtsEngine.resume();
tv_speak.setText("pause");
}
isPlay = !isPlay;
} else {
// Call the API for downloading the on-device TTS model.
downloadModel(MLTtsConstants.TTS_SPEAKER_OFFLINE_EN_US_MALE_EAGLE);
showToast("The offline model has not been downloaded!");
}
}
}).addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(Exception e) {
showToast(e.getMessage());
}
});
5. Release resources when the current UI is destroyed.
Java:
@Override
protected void onDestroy() {
super.onDestroy();
try {
if (mParcelFileDescriptor != null) {
mParcelFileDescriptor.close();
}
if (mCurrentPage != null) {
mCurrentPage.close();
}
if (mPdfRenderer != null) {
mPdfRenderer.close();
}
if (mlTtsEngine != null){
mlTtsEngine.shutdown();
}
} catch (IOException e) {
e.printStackTrace();
}
}
Other Applicable Scenarios
TTS can be used across a broad range of scenarios. For example, you could integrate it into an education app to read bedtime stories to children, or integrate it into a navigation app, which could read out instructions aloud.
For more details, you can go to:
Reddit to join our developer discussion
GitHub to download demos and sample codes
Stack Overflow to solve any integration problems
Original Source
Well explained will it supports all languages?
Displaying products with 3D models is something too great to ignore for an e-commerce app. Using those fancy gadgets, such an app can leave users with the first impression upon products in a fresh way!
The 3D model plays an important role in boosting user conversion. It allows users to carefully view a product from every angle, before they make a purchase. Together with the AR technology, which gives users an insight into how the product will look in reality, the 3D model brings a fresher online shopping experience that can rival offline shopping.
Despite its advantages, the 3D model has yet to be widely adopted. The underlying reason for this is that applying current 3D modeling technology is expensive:
Technical requirements: Learning how to build a 3D model is time-consuming.
Time: It takes at least several hours to build a low polygon model for a simple object, and even longer for a high polygon one.
Spending: The average cost of building a simple model can be more than one hundred dollars, and even higher for building a complex one.
Luckily, 3D object reconstruction, a capability in 3D Modeling Kit newly launched in HMS Core, makes 3D model building straightforward. This capability automatically generates a 3D model with a texture for an object, via images shot from different angles with a common RGB-Cam. It gives an app the ability to build and preview 3D models. For instance, when an e-commerce app has integrated 3D object reconstruction, it can generate and display 3D models of shoes. Users can then freely zoom in and out on the models for a more immersive shopping experience.
Actual Effect
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Technical Solutions
3D object reconstruction is implemented on both the device and cloud. RGB images of an object are collected on the device and then uploaded to the cloud. Key technologies involved in the on-cloud modeling process include object detection and segmentation, feature detection and matching, sparse/dense point cloud computing, and texture reconstruction. Finally, the cloud outputs an OBJ file (a commonly used 3D model file format) of the generated 3D model with 40,000 to 200,000 patches.
Preparations1. Configuring a Dependency on the 3D Modeling SDK
Open the app-level build.gradle file and add a dependency on the 3D Modeling SDK in the dependencies block.
Code:
// Build a dependency on the 3D Modeling SDK.
implementation 'com.huawei.hms:modeling3d-object-reconstruct:1.0.0.300'
2. Configuring AndroidManifest.xml
Open the AndroidManifest.xml file in the main folder. Add the following information before <application> to apply for the storage read and write permissions and camera permission.
Code:
/<!-- Permission to read data from and write data into storage. -->
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<!-- Permission to use the camera. -->
<uses-permission android:name="android.permission.CAMERA" />
Development Procedure1. Configuring the Storage Permission Application
In the onCreate() method of MainActivity, check whether the storage read and write permissions have been granted; if not, apply for them by using requestPermissions.
Code:
/if (EasyPermissions.hasPermissions(MainActivity.this, PERMISSIONS)) {
Log.i(TAG, "Permissions OK");
} else {
EasyPermissions.requestPermissions(MainActivity.this, "To use this app, you need to enable the permission.",
RC_CAMERA_AND_EXTERNAL_STORAGE, PERMISSIONS);
}
Check the application result. If the permissions are not granted, prompt the user to grant them.
Code:
@Override
public void onPermissionsGranted(int requestCode, @NonNull List<String> perms) {
Log.i(TAG, "permissions = " + perms);
if (requestCode == RC_CAMERA_AND_EXTERNAL_STORAGE && PERMISSIONS.length == perms.size()) {
initView();
initListener();
}
}
@Override
public void onPermissionsDenied(int requestCode, @NonNull List<String> perms) {
if (EasyPermissions.somePermissionPermanentlyDenied(this, perms)) {
new AppSettingsDialog.Builder(this)
.setRequestCode(RC_CAMERA_AND_EXTERNAL_STORAGE)
.setRationale("To use this app, you need to enable the permission.")
.setTitle("Insufficient permissions")
.build()
.show();
}
}
2. Creating a 3D Object Reconstruction Configurator
Code:
/// Set the PICTURE mode.
Modeling3dReconstructSetting setting = new Modeling3dReconstructSetting.Factory()
.setReconstructMode(Modeling3dReconstructConstants.ReconstructMode.PICTURE)
.create();
3. Creating a 3D Object Reconstruction Engine and Initializing the Task
Call getInstance() of Modeling3dReconstructEngine and pass the current context to create an instance of the 3D object reconstruction engine.
Code:
// Create an engine.
modeling3dReconstructEngine = Modeling3dReconstructEngine.getInstance(mContext);
Use the engine to initialize the task.
Code:
// Initialize the 3D object reconstruction task.
modeling3dReconstructInitResult = modeling3dReconstructEngine.initTask(setting);
// Obtain the task ID.
String taskId = modeling3dReconstructInitResult.getTaskId();
4. Creating a Listener Callback to Process the Image Upload Result
Create a listener callback that allows you to configure the operations triggered upon upload success and failure.
Code:
// Create an upload listener callback.
private final Modeling3dReconstructUploadListener uploadListener = new Modeling3dReconstructUploadListener() {
@Override
public void onUploadProgress(String taskId, double progress, Object ext) {
// Upload progress.
}
@Override
public void onResult(String taskId, Modeling3dReconstructUploadResult result, Object ext) {
if (result.isComplete()) {
isUpload = true;
ScanActivity.this.runOnUiThread(new Runnable() {
@Override
public void run() {
progressCustomDialog.dismiss();
Toast.makeText(ScanActivity.this, getString(R.string.upload_text_success), Toast.LENGTH_SHORT).show();
}
});
TaskInfoAppDbUtils.updateTaskIdAndStatusByPath(new Constants(ScanActivity.this).getCaptureImageFile() + manager.getSurfaceViewCallback().getCreateTime(), taskId, 1);
}
}
@Override
public void onError(String taskId, int errorCode, String message) {
isUpload = false;
runOnUiThread(new Runnable() {
@Override
public void run() {
progressCustomDialog.dismiss();
Toast.makeText(ScanActivity.this, "Upload failed." + message, Toast.LENGTH_SHORT).show();
LogUtil.e("taskid" + taskId + "errorCode: " + errorCode + " errorMessage: " + message);
}
});
}
};
5. Passing the Upload Listener Callback to the Engine to Upload Images
Pass the upload listener callback to the engine. Call uploadFile(),
pass the task ID obtained in step 3 and the path of the images to be uploaded. Then, upload the images to the cloud server.
Code:
// Pass the listener callback to the engine.
modeling3dReconstructEngine.setReconstructUploadListener(uploadListener);
// Start uploading.
modeling3dReconstructEngine.uploadFile(taskId, filePath);
6. Querying the Task Status
Call getInstance of Modeling3dReconstructTaskUtils to create a task processing instance. Pass the current context.
Code:
// Create a task processing instance.
modeling3dReconstructTaskUtils = Modeling3dReconstructTaskUtils.getInstance(Modeling3dDemo.getApp());
Call queryTask of the task processing instance to query the status of the 3D object reconstruction task.
Code:
// Query the task status, which can be: 0 (images to be uploaded); 1: (image upload completed); 2: (model being generated); 3( model generation completed); 4: (model generation failed).
Modeling3dReconstructQueryResult queryResult = modeling3dReconstructTaskUtils.queryTask(task.getTaskId());
7. Creating a Listener Callback to Process the Model File Download Result
Create a listener callback that allows you to configure the operations triggered upon download success and failure.
Code:
// Create a download listener callback.
private Modeling3dReconstructDownloadListener modeling3dReconstructDownloadListener = new Modeling3dReconstructDownloadListener() {
@Override
public void onDownloadProgress(String taskId, double progress, Object ext) {
((Activity) mContext).runOnUiThread(new Runnable() {
@Override
public void run() {
dialog.show();
}
});
}
@Override
public void onResult(String taskId, Modeling3dReconstructDownloadResult result, Object ext) {
((Activity) mContext).runOnUiThread(new Runnable() {
@Override
public void run() {
Toast.makeText(getContext(), "Download complete", Toast.LENGTH_SHORT).show();
TaskInfoAppDbUtils.updateDownloadByTaskId(taskId, 1);
dialog.dismiss();
}
});
}
@Override
public void onError(String taskId, int errorCode, String message) {
LogUtil.e(taskId + " <---> " + errorCode + message);
((Activity) mContext).runOnUiThread(new Runnable() {
@Override
public void run() {
Toast.makeText(getContext(), "Download failed." + message, Toast.LENGTH_SHORT).show();
dialog.dismiss();
}
});
}
};
8. Passing the Download Listener Callback to the Engine to Download the File of the Generated Model
Pass the download listener callback to the engine. Call downloadModel, pass the task ID obtained in step 3 and the path for saving the model file to download it.
Code:
/ Pass the download listener callback to the engine.
modeling3dReconstructEngine.setReconstructDownloadListener(modeling3dReconstructDownloadListener);
// Download the model file.
modeling3dReconstructEngine.downloadModel(appDb.getTaskId(), appDb.getFileSavePath());
More Information
The object should have rich texture, be medium-sized, and a rigid body. The object should not be reflective, transparent, or semi-transparent. The object types include goods (like plush toys, bags, and shoes), furniture (like sofas), and cultural relics (such as bronzes, stone artifacts, and wooden artifacts).
The object dimension should be within the range from 15 x 15 x 15 cm to 150 x 150 x 150 cm. (A larger dimension requires a longer time for modeling.)
3D object reconstruction does not support modeling for the human body and face.
Ensure the following requirements are met during image collection: Put a single object on a stable plane in pure color. The environment shall not be dark or dazzling. Keep all images in focus, free from blur caused by motion or shaking. Ensure images are taken from various angles including the bottom, flat, and top (it is advised that you upload more than 50 images for an object). Move the camera as slowly as possible. Do not change the angle during shooting. Lastly, ensure the object-to-image ratio is as big as possible, and all parts of the object are present.
These are all about the sample code of 3D object reconstruction. Try to integrate it into your app and build your own 3D models!
ReferencesFor more details, you can go to:
3D Modeling Kit official website
3D Moedling Kit Development Documentation page, to find the documents you need
Reddit to join our developer discussion
GitHub to download 3D Modeling Kit sample codes
Stack Overflow to solve any integration problems
Text to speech (TTS) is highly sought after by audio/video editors, thanks to its ability to automatically turn text into naturally sounding speech, as a low cost alternative to human dubbing. It can be used on all kinds of video, regardless of whether the video is long or short.
I recently stumbled upon the AI dubbing capability of HMS Core Audio Editor Kit, which does just that. It is able to turn input text into speech with just a tap, and comes loaded with a selection of smooth, naturally-sounding male and female timbres.
This is ideal for developing apps that involve e-books, creating audio content, and editing audio/video. Below describes how I integrated this capability.
Making PreparationsComplete all necessary preparations by following the official guide.
Configuring the Project1. Set the app authentication information
The information can be set via an API key or access token (recommended).
Use setAccessToken to set an access token during app initialization.
Java:
HAEApplication.getInstance().setAccessToken("your access token");
Or, use setApiKey to set an API key during app initialization. The API key needs to be set only once.
Java:
HAEApplication.getInstance().setApiKey("your ApiKey");
2. Initialize the runtime environment
Initialize HuaweiAudioEditor, and create a timeline and necessary lanes.
Java:
// Create a HuaweiAudioEditor instance.
HuaweiAudioEditor mEditor = HuaweiAudioEditor.create(mContext);
// Initialize the runtime environment of HuaweiAudioEditor.
mEditor.initEnvironment();
// Create a timeline.
HAETimeLine mTimeLine = mEditor.getTimeLine();
// Create a lane.
HAEAudioLane audioLane = mTimeLine.appendAudioLane();
Import audio.
Java:
// Add an audio asset to the end of the lane.
HAEAudioAsset audioAsset = audioLane.appendAudioAsset("/sdcard/download/test.mp3", mTimeLine.getCurrentTime());
3. Integrate AI dubbing.
Call HAEAiDubbingEngine to implement AI dubbing.
Java:
// Configure the AI dubbing engine.
HAEAiDubbingConfig haeAiDubbingConfig = new HAEAiDubbingConfig()
// Set the volume.
.setVolume(volumeVal)
// Set the speech speed.
.setSpeed(speedVal)
// Set the speaker.
.setType(defaultSpeakerType);
// Create a callback for an AI dubbing task.
HAEAiDubbingCallback callback = new HAEAiDubbingCallback() {
@Override
public void onError(String taskId, HAEAiDubbingError err) {
// Callback when an error occurs.
}
@Override
public void onWarn(String taskId, HAEAiDubbingWarn warn) {}
@Override
public void onRangeStart(String taskId, int start, int end) {}
@Override
public void onAudioAvailable(String taskId, HAEAiDubbingAudioInfo haeAiDubbingAudioFragment, int i, Pair<Integer, Integer> pair, Bundle bundle) {
// Start receiving and then saving the file.
}
@Override
public void onEvent(String taskId, int eventID, Bundle bundle) {
// Synthesis is complete.
if (eventID == HAEAiDubbingConstants.EVENT_SYNTHESIS_COMPLETE) {
// The AI dubbing task has been complete. That is, the synthesized audio data is completely processed.
}
}
@Override
public void onSpeakerUpdate(List<HAEAiDubbingSpeaker> speakerList, List<String> lanList,
List<String> lanDescList) { }
};
// AI dubbing engine.
HAEAiDubbingEngine mHAEAiDubbingEngine = new HAEAiDubbingEngine(haeAiDubbingConfig);
// Set the listener for the playback process of an AI dubbing task.
mHAEAiDubbingEngine.setAiDubbingCallback(callback);
// Convert text to speech and play the speech. In the method, text indicates the text to be converted to speech, and mode indicates the mode for playing the converted audio.
String taskId = mHAEAiDubbingEngine.speak(text, mode);
// Pause playback.
mHAEAiDubbingEngine.pause();
// Resume playback.
mHAEAiDubbingEngine.resume();
// Stop AI dubbing.
mHAEAiDubbingEngine.stop();
ResultIn the demo below, I successfully implement the AI dubbing function in app. Now, I can converts text into emotionally expressive speech, with default and custom timbres.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
To learn more, please visit:
>> Audio Editor Kit official website
>> Audio Editor Kit Development Guide
>> Reddit to join developer discussions
>> GitHub to download the sample code
>> Stack Overflow to solve integration problems
Follow our official account for the latest HMS Core-related news and updates.
Now that spring has arrived, it's time to get out and stretch your legs! As programmers, many of us are used to being seated for hours and hours at time, which can lead to back pain and aches. We're all aware that building the workout plan, and keeping track of health indicators round-the-clock can have enormous benefits for body, mind, and soul.
Fortunately, AR Engine makes that remarkably easy. It comes with face tracking capabilities, and will soon support body tracking as well. Thanks to core AR algorithms, AR Engine is able to monitor heart rate, respiratory rate, facial health status, and heart rate waveform signals in real time during your workouts. You can also use it to build an app, for example, to track the real-time workout status, perform real-time health check for patients, or to monitor real-time health indicators of vulnerable users, like the elderly or the disabled. With AR Engine, you can make your health or fitness app more engaging and visually immersive than you might have believed possible.
Advantages and Device Model Restrictions1. Monitors core health indicators like heart rate, respiratory rate, facial health status, and heart rate waveform signals in real time.
2. Enables devices to better understand their users. Thanks to technologies like Simultaneous Localization and Mapping (SLAM) and 3D reconstruction, AR Engine renders images to build 3D human faces on mobile phones, resulting in seamless virtual-physical cohesion.
3. Supports all of the device models listed in Software and Hardware Requirements of AR Engine Features.
Demo IntroductionA simple demo is available to give you a grasp of how to integrate AR Engine, and use its human body and face tracking capabilities.
ENABLE_HEALTH_DEVICE: indicates whether to enable health check.
HealthParameter: health check parameter, including heart rate, respiratory rate, age and gender probability based on facial features, and heart rate waveform signals.
FaceDetectMode: face detection mode, including health rate checking, respiratory rate checking, real-time health checking, and all of the three above.
Effect
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
The following details how you can run the demo using the source code.
Key Steps1. Add the Huawei Maven repository to the project-level build.gradle file.
Java:
buildscript {
repositories {
maven { url 'http://developer.huawei.com/repo/'}
}
dependencies {
...
// Add the AppGallery Connect plugin configuration.
classpath 'com.huawei.agconnect:agcp:1.4.2.300'
}
}allprojects {
repositories {
maven { url 'http://developer.huawei.com/repo/'}
}
}
2. Add dependencies on the SDK to the app-level build.gradle file.
Java:
implementation 'com.huawei.hms:arenginesdk:3.7.0.3'
3. Declare system permissions in the AndroidManifest.xml file.
Java:
<uses-permission android:name="android.permission.CAMERA" />
4. Check whether AR Engine has been installed on the current device. If yes, the app can run properly. If not, the app automatically redirects the user to AppGallery to install AR Engine.
Java:
boolean isInstallArEngineApk = AREnginesApk.isAREngineApkReady(this);
if (!isInstallArEngineApk && isRemindInstall) {
Toast.makeText(this, "Please agree to install.", Toast.LENGTH_LONG).show();
finish();
}
if (!isInstallArEngineApk) {
startActivity(new Intent(this, ConnectAppMarketActivity.class));
isRemindInstall = true;
}
return AREnginesApk.isAREngineApkReady(this);
Key Code1. Call ARFaceTrackingConfig and create an ARSession object. Then, set the human face detection mode, configure AR parameters for motion tracking, and enable motion tracking.
Java:
mArSession = new ARSession(this);
mArFaceTrackingConfig = new ARFaceTrackingConfig(mArSession);
mArFaceTrackingConfig.setEnableItem(ARConfigBase.ENABLE_HEALTH_DEVICE);
mArFaceTrackingConfig
.setFaceDetectMode(ARConfigBase.FaceDetectMode.HEALTH_ENABLE_DEFAULT.getEnumValue());
2. Call FaceHealthServiceListener to add your app and pass the health check status and progress. Call handleProcessProgressEvent() to obtain the health check progress.
Java:
mArSession.addServiceListener(new FaceHealthServiceListener() {
@Override
public void handleEvent(EventObject eventObject) {
if (!(eventObject instanceof FaceHealthCheckStateEvent)) {
return;
}
final FaceHealthCheckState faceHealthCheckState =
((FaceHealthCheckStateEvent) eventObject).getFaceHealthCheckState();
runOnUiThread(new Runnable() {
@Override
public void run() {
mHealthCheckStatusTextView.setText(faceHealthCheckState.toString());
}
});
}
@Override
public void handleProcessProgressEvent(final int progress) {
mHealthRenderManager.setHealthCheckProgress(progress);
runOnUiThread(new Runnable() {
@Override
public void run() {
setProgressTips(progress);
}
});
}
});
private void setProgressTips(int progress) {
String progressTips = "processing";
if (progress >= MAX_PROGRESS) {
progressTips = "finish";
}
mProgressTips.setText(progressTips);
mHealthProgressBar.setProgress(progress);
}
Update data in real time and display the health check result.
Java:
mActivity.runOnUiThread(new Runnable() {
@Override
public void run() {
mHealthParamTable.removeAllViews();
TableRow heatRateTableRow = initTableRow(ARFace.HealthParameter.PARAMETER_HEART_RATE.toString(),
healthParams.getOrDefault(ARFace.HealthParameter.PARAMETER_HEART_RATE, 0.0f).toString());
mHealthParamTable.addView(heatRateTableRow);
TableRow breathRateTableRow = initTableRow(ARFace.HealthParameter.PARAMETER_BREATH_RATE.toString(),
healthParams.getOrDefault(ARFace.HealthParameter.PARAMETER_BREATH_RATE, 0.0f).toString());
mHealthParamTable.addView(breathRateTableRow);
}
});
References>> AR Engine official website
>> AR Engine Development Guide
>> Reddit to join developer discussions
>> GitHub to download the sample code
>> Stack Overflow to solve integration problems
Being creative is hard, but thinking of a fantastic idea is even more challenging. And once you've done that, the hardest part is expressing that idea in an attractive way.
This, I think, is the very reason why templates are gaining popularity in text, image, audio, and video editing, and more. Of all these templates, video templates are probably the most in demand by users. This is because a video is a lengthy creation which may also require high costs. And therefore, it is much more convenient to create a video using a template, rather than from scratch — which is particularly true for video editing amateurs.
The video template solution I have got for my app is a template capability of HMS Core Video Editor Kit. This capability comes preloaded with a library of templates that my users can use directly to quickly create a short video, make a vlog during their journey, create a product display video, generate a news video, and more.
On top of this, this capability comes with a platform where I can manage the templates easily, like this.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Template management platform — AppGallery ConnectTo be honest, one of the things that I really like about the capability is that it's easy to integrate, thanks to its straightforward code, as well as a whole set of APIs and relevant description on how to use them. Below is the process I followed to integrate the capability into my app.
Development ProcedurePreparations
Configure the app information.
Integrate the SDK.
Set up the obfuscation scripts.
Declare necessary permissions, including: device vibration permission, microphone use permission, storage read permission, and storage write permission.
Project ConfigurationSetting Authentication InformationSet the authentication information by using:
An access token. The setting is required only once during app initialization.
Code:
MediaApplication.getInstance().setAccessToken("your access token");
Or, an API key, which also needs to be set only once during app initialization.
Code:
MediaApplication.getInstance().setApiKey("your ApiKey");
Configuring a License IDSince this ID is used to manage the usage quotas of the service, the ID must be unique.
Code:
MediaApplication.getInstance().setLicenseId("License ID");
Initialize the runtime environment for HuaweiVideoEditor.
During project configuration, an object of HuaweiVideoEditor must be created first, and its runtime environment must be initialized. When exiting the project, the object shall be released.
1. Create a HuaweiVideoEditor object.
Code:
HuaweiVideoEditor editor = HuaweiVideoEditor.create(getApplicationContext());
2. Specify the preview area position. Such an area renders video images, which is implemented via SurfaceView created within the SDK. Before creating this area, specify its position first.
Code:
<LinearLayout
android:id="@+id/video_content_layout"
android:layout_width="0dp"
android:layout_height="0dp"
android:background="@color/video_edit_main_bg_color"
android:gravity="center"
android:orientation="vertical" />
// Specify a preview area.
LinearLayout mSdkPreviewContainer = view.findViewById(R.id.video_content_layout);
// Set the preview area layout.
editor.setDisplay(mSdkPreviewContainer);
3. Initialize the runtime environment. LicenseException will be thrown, if the license verification fails.
When a HuaweiVideoEditor object is created, no system resources have been used. Manually set the time when its runtime environment is initialized, and required threads and timers will be created in the SDK.
Code:
try {
editor.initEnvironment();
} catch (LicenseException error) {
SmartLog.e(TAG, "initEnvironment failed: " + error.getErrorMsg());
finish();
return;
}
Capability IntegrationIn this part, I use HVETemplateManager to obtain the on-cloud template list, and then provide the list to my app users.
Code:
// Obtain the template column list.
final HVEColumnInfo[] column = new HVEColumnInfo[1];
HVETemplateManager.getInstance().getColumnInfos(new HVETemplateManager.HVETemplateColumnsCallback() {
@Override
public void onSuccess(List<HVEColumnInfo> result) {
// Called when the list is successfully obtained.
column[0] = result.get(0);
}
@Override
public void onFail(int error) {
// Called when the list failed to be obtained.
}
});
// Obtain the list details.
final String[] templateIds = new String[1];
// size indicates the number of the to-be-requested on-cloud templates. The size value must be greater than 0. Offset indicates the offset of the to-be-requested on-cloud templates. The offset value must be greater than or equal to 0. true indicates to forcibly obtain the data of the on-cloud templates.
HVETemplateManager.getInstance().getTemplateInfos(column[0].getColumnId(), size, offset, true, new HVETemplateManager.HVETemplateInfosCallback() {
@Override
public void onSuccess(List<HVETemplateInfo> result, boolean hasMore) {
// Called when the list details are successfully obtained.
HVETemplateInfo templateInfo = result.get(0);
// Obtain the template ID.
templateIds[0] = templateInfo.getId();
}
@Override
public void onFail(int errorCode) {
// Called when the list details failed to be obtained.
}
});
// Obtain the template ID when the list details are obtained.
String templateId = templateIds[0];
// Obtain a template project.
final List<HVETemplateElement>[] editableElementList = new ArrayList[1];;
HVETemplateManager.getInstance().getTemplateProject(templateId, new HVETemplateManager.HVETemplateProjectCallback() {
@Override
public void onSuccess(List<HVETemplateElement> editableElements) {
// Direct to the material selection screen when the project is successfully obtained. Update editableElements with the paths of the selected local materials.
editableElementList[0] = editableElements;
}
@Override
public void onProgress(int progress) {
// Called when the progress of obtaining the project is received.
}
@Override
public void onFail(int errorCode) {
// Called when the project failed to be obtained.
}
});
// Prepare a template project.
HVETemplateManager.getInstance().prepareTemplateProject(templateId, new HVETemplateManager.HVETemplateProjectPrepareCallback() {
@Override
public void onSuccess() {
// Called when the preparation is successful. Create an instance of HuaweiVideoEditor, for operations like playback, preview, and export.
}
@Override
public void onProgress(int progress) {
// Called when the preparation progress is received.
}
@Override
public void onFail(int errorCode) {
// Called when the preparation failed.
}
});
// Create an instance of HuaweiVideoEditor.
// Such an instance will be used for operations like playback and export.
HuaweiVideoEditor editor = HuaweiVideoEditor.create(templateId, editableElementList[0]);
try {
editor.initEnvironment();
} catch (LicenseException e) {
SmartLog.e(TAG, "editor initEnvironment ERROR.");
}
Once you've completed this process, you'll have created an app just like in the demo displayed below.
ConclusionAn eye-catching video, for all the good it can bring, can be difficult to create. But with the help of video templates, users can create great-looking videos in even less time, so they can spend more time creating more videos.
This article illustrates a video template solution for mobile apps. The template capability offers various out-of-the-box preset templates that can be easily managed on a platform. And what's better is that the whole integration process is easy. So easy in fact even I could create a video app with templates.
ReferencesTips for Using Templates to Create Amazing Videos
Integrating the Template Capability