Tips for Developing a Screen Recorder - Huawei Developers

Let's face it. Sometimes it can be difficult for our app users to find a specific app function when our apps are loaded with all kinds of functions. Many of us tend to write up a guide detailing each function found in the app, but — honestly speaking — users don't really have the time or patience to read through long guides, and not all guides are user-friendly, either. Sometimes it's faster to play about with a function than it is to look it up and learn about it. But that creates the possibility that users are not using the functions of our app to its full potential.
Luckily, making a screen recording is a great way of showing users how functions work, step by step.
Just a few days ago, I decided to create some video tutorials of my own app, but first I needed to develop a screen recorder. One that looks like this.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
How the Screen Recorder Works​Tap START RECORDING on the home screen to start a recording. Then, switch to the screen that is going to be recorded. When the recording is under way, the demo app runs in the background so that the whole screen is visible for recording. To stop recording, simply swipe down on the screen and tap STOP in the notification center, or go back to the app and tap STOP RECORDING. It's as simple as that! The screen recording will be saved to a specified directory and displayed on the app's home screen.
To create such a lightweight screen recording tool, we just need to use the basic functions of the screen recorder SDK from HMS Core Video Editor Kit. This SDK is easy to integrate. Because of this, I believe that except for using it to develop an independent screen recording app, it is also ideal for equipping an app with the screen recording function. This can be really helpful for apps in gaming and online education, which enables users to record their screens without having to switch to another app.
I also discovered that this SDK actually allows a lot more than simply starting and stopping recording. The following are some examples.
The service allows its notification to be customized. For example, we can add a pause or resume button to the notification bar to let users pause and resume the recording at the touch of a button. Not only that, the duration of the recording can be displayed in the notification bar, so that users can check out how long a screen recording is in real time just by visiting the notification center.
The SDK also offers a range of other functions, for great flexibility. It supports several major resolutions (including 480p, 720p, and 1080p) which can be set according to different scenarios (such as the device model limitation), and it lets users manually choose where recordings will be saved.
Now, let's move on to the development part to see how the demo app was created.
Development Procedure​Necessary Preparations​step 1 Configure app information in AppGallery Connect.
Register as a developer.
Create an app.
Generate a signing certificate fingerprint.
Configure the signing certificate fingerprint.
Enable services for the app as needed.
step 2 Integrate the HMS Core SDK.
step 3 Configure obfuscation scripts.
step 4 Declare necessary permissions, including those allowing the screen recorder SDK to access the device microphone, write data into storage, read data from storage, close system dialogs, and access the foreground service.
Building the Screen Recording Function​step 1 Create an instance of HVERecordListener (which is the listener for events happening during screen recording) and override methods in the listener.
Code:
HVERecordListener mHVERecordListener = new HVERecordListener(){
@Override
public void onRecordStateChange(HVERecordState recordingStateHve) {
// Callback when the screen recording status changes.
}
@Override
public void onRecordProgress(int duration) {
// Callback when the screen recording progress is received.
}
@Override
public void onRecordError(HVEErrorCode err, String msg) {
// Callback when an error occurs during screen recording.
}
@Override
public void onRecordComplete(HVERecordFile fileHve) {
// Callback when screen recording is complete.
}
};
step 2 Initialize HVERecord by using the app context and the instance of HVERecordListener.
Code:
HVERecord.init(this, mHVERecordListener);
step 3 Create an HVERecordConfiguration.Builder instance to set up screen recording configurations. Note that this step is optional.
Code:
HVERecordConfiguration hveRecordConfiguration = new HVERecordConfiguration.Builder()
.setMicStatus(true)
.setOrientationMode(HVEOrientationMode.LANDSCAPE)
.setResolutionMode(HVEResolutionMode.RES_480P)
.setStorageFile(new File("/sdcard/DCIM/Camera"))
.build();
HVERecord.setConfigurations(hveRecordConfiguration);
step 4 Customize the screen recording notification.
Before this, we need to create an XML file that specifies the notification layout. This file includes IDs of components in the notification, like buttons. The code below illustrates how I used the XML file for my app, in which a button is assigned with the ID btn_1. Of course, the button count can be adjusted according to your own needs.
Code:
HVENotificationConfig notificationData = new HVENotificationConfig(R.layout.hms_scr_layout_custom_notification);
notificationData.addClickEvent(R.id.btn_1, () -> { HVERecord.stopRecord(); });
notificationData.setDurationViewId(R.id.duration);
notificationData.setCallingIntent(new Intent(this, SettingsActivity.class)
.addFlags(Intent.FLAG_ACTIVITY_CLEAR_TOP | Intent.FLAG_ACTIVITY_CLEAR_TASK));
HVERecord.setNotificationConfig(notificationData);
As you can see, in the code above, I initially passed the custom notification layout to the initialization method of HVENotificationConfig. Then, I used the addClickEvent method to create a tapping event. For this, I used the IDs of a button and textView, as well as the tapping event, which are specified in the XML file. Thirdly, I called setDurationViewId to set the ID of textView, to determine where the screen recording duration is displayed. After this, I called setCallingIntent to set the intent that is returned when the notification is tapped. In my app, this intent is used to open an Activity, which, as you know it, is a common intent use. And finally, I set up the notification configurations in the HVERecord class.
step 5 Start screen recording.
Code:
HVERecord.startRecord();
step 6 Stop screen recording.
Code:
HVERecord.stopRecord();
And just like that, I created a fully functional screen recorder.
Besides using it to make instructional videos for apps, a screen recorder can be a helpful companion for a range of other situations. For example, it can be used to record an online conference or lecture, and video chats with family and friends can also be recorded and saved.
I noticed that the screen recorder SDK is also capable of picking up external sounds and switching between landscape and portrait mode. This is ideal for gamers who want to show off their skills while recording a video with real-time commentary.
That pretty sums up my ideas of how a screen recording app can be used. So, what do you think? I look forward to reading your ideas in the comments section.
Conclusion​Screen recording is perfect for making video tutorials of app functions, showcasing gaming skills in videos, and recording online conferences or lectures. Not only is it useful for recording what's displayed on a screen, it's also able to record external sounds, meaning you can create an app that supports videos with commentary. The screen recorder SDK from Video Editor Kit is good for implementing the mentioned feature. Its streamlined integration process and flexibility (customizable notification and saving directory for recordings, for example) make it a handy tool for both creating an independent screen recording app and developing a screen recording function into an app.

wow this is awesome, some phones doesn't support a built-in screen recording, is it possible to create one too in different models?

PilyoDev said:
wow this is awesome, some phones doesn't support a built-in screen recording, is it possible to create one too in different models?
Click to expand...
Click to collapse
Hi, there! The screen recorder SDK of HMS Core Video Editor Kit supports both Huawei phones (EMUI 5.0 or later) and non-Huawei phones (Android 7.0 or later). Try it out → https://developer.huawei.com/consum...tegrate-srsdk-0000001334063037?ha_source=hms3

Related

How to Integrate ML Kit's Virtual Human Service

1. Introduction to Virtual Human
Virtual Human is a service that utilizes cutting-edge AI technologies, including image vision, emotion generation, voice cloning, and semantic understanding, and has a wide range of applications, spanning news broadcasting, financial customer service, and virtual gaming.
Application scenarios:
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
2. ML Kit's Virtual Human Service
ML Kit's Virtual Human service is backed by core Huawei AI technologies, such as image processing, text to speech, voice cloning, and semantic understanding, and provides innovative, cost-effective authoring modes for education, news, and multimedia production enterprises. Virtual Human service features a number of crucial advantages over other similar services, including the following:
Ultra-HD 4K cinematic effects
Supports large-screen displays. The details and textures of the entire body are rendered in the same definition.
Generates images that fit seamlessly with the real background, and achieve trackless fusion under HD resolution.
Generates detailed lip features, distinct lipstick reflection, and lifelike textures.
Produces clear and visible teeth, and true-to-life textures.
Hyper-real synthesis effects
True restoration of teeth (no painting involved), lips, and even lipstick reflections.
True restoration of facial features such as illumination, contrasts, shadows, and dimples.
Seamless connections between the generated texture for the mouth and the real texture.
Intricate animation effects that outperform those for 3D live anchors.
Comparing with services provided by other enterprises.
3. ML Kit's Virtual Human Video Display
As shown below, Virtual Human generates ultra-HD video effects, provides for clearer enunciation, and exercises better control over key details, such as lip features, lipstick reflections, actual pronunciation and illumination.
4. Integrating ML Kit's Virtual Human Service
4.1 Integration Process
4.1.1 Submitting the Text of the Video for Generation
Call the customized API for converting text into the virtual human video, and pass the required configurations (specified by parameter config) and text (specified by parameter data) to the backend for processing through the API. First, check the length of the passed text. The maximum length of the Chinese text is 1,000 characters, and that for the English text is 3,000 characters. Perform the non-null check on the passed configurations, then submit the text and configurations to convert the text into audio
4.1.2 Using the Data Provided by an Asynchronous Scheduled Task
Call the text-to-speech algorithm to convert text into the video, based on the data provided by the asynchronous scheduled task, and synthesize the video with the previously obtained audio.
4.1.3 Checking Whether the Text Has been Successfully Converted
Call the API for querying the results of converting text into the virtual human video, to check whether the text has been successfully converted. If the execution is complete, the video link will be returned.
4.1.4 Accessing the Videos via the Link
Access the generated video through the link returned by the API, to query the results of converting text into the virtual human video.
4.2 Main APIs for Integration
4.2.1 Customized API for Converting Text into the Virtual Human Video
URL: http://10.33.219.58:8888/v1/vup/text2vedio/submit
Request parameters
Main functions:
Input the customized API for converting text into the virtual human video. The API is asynchronous. Currently, Virtual Human can only complete the conversion using an offline mode, a process that takes time to complete. The conversion results can be queried via the API for querying the results of converting text into the virtual human video. If the submitted text has been synthesized, you can return and play the video directly.
Main logic:
Convert the text into audio based on the text and configurations to be synthesized, passed by the frontend. Execute multithreading asynchronously, generate the video that meets pronunciation requirements based on the text-to-speech algorithm, and then compound the video with audio to generate the virtual human video. If the submitted text has been synthesized, you can return and play the video directly.
4.2.2 API for Querying the Results of Converting Text into the Virtual Human Video
URL: http://10.33.219.58:8888/v1/vup/text2vedio/query
Request parameters
Main functions:
Query the conversion status in batches, based on the submitted text IDs.
Main logic:
Query the synthesis status of the video through textlds (the ID list of the synthesized text passed by the frontend), save the obtained status results to a set as the output parameter, and insert the parameter to the returned request. If the requested text has been synthesized, you can return and play the video directly.
4.2.3 API for Taking the Virtual Human Video Offline in Batches
URL: http://10.33.219.58:8888/v1/vup/text2vedio/offline
Request parameters
Main functions:
Bring the video offline in batches, based on the submitted text ID.
Main logic:
Change the status of the video corresponding to the ID in the array to offline through textlds (the ID array of the synthesized text transmitted by the frontend), and then delete the video. The offline video is not capable of being played.
4.3 Main Functions of ML Kit's Virtual Human
ML Kit's Virtual Human service has a myriad of powerful functions.
1. Dual language support: Virtual Human currently supports Chinese and English, and thus text in either Chinese or English can be used as audio data.
2. Multiple virtual anchors: The service supports up to four virtual anchors, one Chinese female voice, one English female voice, and two English male voices.
3. Picture-in-picture video: Picture-in-picture video play, in essence, small-window video playback, is supported as well. When playing a video in picture-in-picture mode, the video window moves in accordance with the rest of the screen. Users are able to view the text while playing the video, and can drag the video to any location on the screen for easier reading.
4. Adjustable speech speed, volume, and tone: The speech speed, volume, and tone can be adjusted at will, to meet a wide range of user needs.
5. Multi-background settings: The service allows you to choose from diverse backgrounds for virtual anchors. There are currently three built-in backgrounds provided: transparent, green-screen, and technological. You can also upload an image to apply a customized background.
6. Subtitles: Virtual Human is capable of automatically generating Chinese, English, and bilingual subtitles.
7. Multi-layout settings: You can change the position of the virtual anchors on the screen (left, right, or middle of the screen) by setting parameters. You can also determine the size of the virtual anchors and choose to place either their upper body or entire body in view. In addition, you are free to set a channel logo, its position on the screen, as well as the video to be played. This ensures that the picture-in-picture effect achieves a bona fide news broadcast experience.
Picture-in-picture effect:
5. Final Thoughts
As a developer, after using ML Kit's Virtual Human service to generate a video, I was shocked at its capabilities, especially the picture-in-picture capability, which helped me generate real news broadcast effects. It has got me wondering whether virtual humans will soon replace real anchors.
To learn more, please visit the official website:
Reference
Official website of Huawei Developers
Development Guide
HMS Core official community on Reddit
Demo and sample code
Discussions on Stack Overflow

How I Developed a Smile Filter for My App

{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
I recently read an article that explained how we as human beings are hardwired to enter the fight-or-flight mode when we realize that we are being watched. This feeling is especially strong when somebody else is trying to take a picture of us, which is why many of us find it difficult to smile in photos. This effect is so strong that we've all had the experience of looking at a photo right after it was taken and noticing straight away that the photo needs to be retaken because our smile wasn't wide enough or didn't look natural. So, the next time someone criticizes my smile in a photo, I'm just going to them, "It's not my fault. It's literally an evolutionary trait!"
Or, instead of making such an excuse, what about turning to technology for help? Actually, I have tried using some photo editor apps to modify my portrait photos, making my facial expression look nicer by, for example, removing my braces, whitening my teeth, and erasing my smile lines. However, maybe it's because of my rusty image editing skills, the modified images often turn out to be strange.
My lack of success with photo editing made me wonder: Wouldn't it be great if there was a function specially designed for people like me, who find it difficult to smile naturally in photos and who aren't good at photo editing, which could automatically give us picture-perfect smiles?
I then suddenly remembered that I had heard about an interesting function called smile filter that has been going viral on different apps and platforms. A smile filter is an app feature which can automatically add a natural-looking smile to a face detected in an image. I have tried it before and was really amazed by the result. In light of my sudden recall, I decided to create a demo app with a similar function, in order to figure out the principle behind it.
To provide my app with a smile filter, I chose to use the auto-smile capability provided by HMS Core Video Editor Kit. This capability automatically detects people in an image and then lightens up the detected faces with a smile (either closed- or open-mouth) that perfectly blends in with each person's facial structure. With the help of such a capability, a mobile app can create the perfect smile in seconds and save users from the hassle of having to use a professional image editing program.
Check the result out for yourselves:
Looks pretty natural, right? This is the result offered by my demo app integrated with the auto-smile capability. The original image looks like this:
Next, I will explain how I integrated the auto-smile capability into my app and share the relevant source code from my demo app.
Integration Procedure​Preparations​1, Configure necessary app information. This step requires you to register a developer account, create an app, generate a signing certificate fingerprint, configure the fingerprint, and enable required services.
2. Integrate the SDK of the kit.
3. Configure the obfuscation scripts.
4. Declare necessary permissions.
Project Configuration​1. Set the app authentication information. This can be done via an API key or an access token.
Using an API key: You only need to set the app authentication information once during app initialization.
Code:
MediaApplication.getInstance().setApiKey("your ApiKey");
Or, using an access token: You only need to set the app authentication information once during app initialization.
Code:
MediaApplication.getInstance().setAccessToken("your access token");
2. Set a License ID, which must be unique because it is used to manage the usage quotas of the service.
Code:
MediaApplication.getInstance().setLicenseId("License ID");
3. Initialize the runtime environment for the HuaweiVideoEditor object. Remember to release the HuaweiVideoEditor object when exiting the project.
Create a HuaweiVideoEditor object.
Code:
HuaweiVideoEditor editor = HuaweiVideoEditor.create(getApplicationContext());
Specify the preview area position. Such an area is used to render video images, which is implemented by SurfaceView created within the SDK. Before creating such an area, specify its position in the app first.
Code:
<LinearLayout
android:id="@+id/video_content_layout"
android:layout_width="0dp"
android:layout_height="0dp"
android:background="@color/video_edit_main_bg_color"
android:gravity="center"
android:orientation="vertical" />
// Specify the preview area position.
LinearLayout mSdkPreviewContainer = view.findViewById(R.id.video_content_layout);
// Specify the preview area layout.
editor.setDisplay(mSdkPreviewContainer);
Initialize the runtime environment. If license verification fails, LicenseException will be thrown.
After it is created, the HuaweiVideoEditor object will not occupy any system resources. You need to manually set when the runtime environment of the object will be initialized. Once you have done this, necessary threads and timers will be created within the SDK.
Code:
try {
editor.initEnvironment();
} catch (LicenseException error) {
SmartLog.e(TAG, "initEnvironment failed: " + error.getErrorMsg());
finish();
return;
}
Function Development​
Code:
// Apply the auto-smile effect. Currently, this effect only supports image assets.
imageAsset.addFaceSmileAIEffect(new HVEAIProcessCallback() {
@Override
public void onProgress(int progress) {
// Callback when the handling progress is received.
}
@Override
public void onSuccess() {
// Callback when the handling is successful.
}
@Override
public void onError(int errorCode, String errorMessage) {
// Callback when the handling failed.
}
});
// Stop applying the auto-smile effect.
imageAsset.interruptFaceSmile();
// Remove the auto-smile effect.
imageAsset.removeFaceSmileAIEffect();
And with that, I successfully integrated the auto-smile capability into my demo app, and now it can automatically add smiles to faces detected in the input image.
Conclusion​Research has demonstrated that it is normal for people to behave unnaturally when we are being photographed. Such unnaturalness becomes even more obvious when we try to smile. This explains why numerous social media apps and video/image editing apps have introduced smile filter functions, which allow users to easily and quickly add a naturally looking smile to faces in an image.
Among various solutions to such a function, HMS Core Video Editor Kit's auto-smile capability stands out by providing excellent, natural-looking results and featuring straightforward and quick integration.
What's better, the auto-smile capability can be used together with other capabilities from the same kit, to further enhance users' image editing experience. For example, when used in conjunction with the kit's AI color capability, you can add color to an old black-and-white photo and then use auto-smile to add smiles to the sullen expressions of the people in the photo. It's a great way to freshen up old and dreary photos from the past.
And that's just one way of using the auto-smile capability in conjunction with other capabilities. What ideas do you have? Looking forward to knowing your thoughts in the comments section.
References​How to Overcome Camera Shyness or Phobia
Introduction to Auto-Smile

How I Created a Smart Video Clip Extractor

{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Travel and life vlogs are popular among app users: Those videos are telling, covering all the most attractive parts in a journey or a day. To create such a video first requires great editing efforts to cut out the trivial and meaningless segments in the original video, which used to be a thing of video editing pros.
This is no longer the case. Now we have an array of intelligent mobile apps that can help us automatically extract highlights from a video, so we can focus more on spicing up the video by adding special effects, for example. I opted to use the highlight capability from HMS Core Video Editor Kit to create my own vlog editor.
How It Works​This capability assesses how appealing video frames are and then extracts the most suitable ones. To this end, it is said that the capability takes into consideration the video properties most concerned by users, a conclusion that is drawn from survey and experience assessment from users. On the basis of this, the highlight capability develops a comprehensive frame assessment scheme that covers various aspects. For example:
Aesthetics evaluation. This aspect is a data set built upon composition, lighting, color, and more, which is the essential part of the capability.
Tags and facial expressions. They represent the frames that are detected and likely to be extracted by the highlight capability, such as frames that contain people, animals, and laughter.
Frame quality and camera movement mode. The capability discards low-quality frames that are blurry, out-of-focus, overexposed, or shaky, to ensure such frames will not impact the quality of the finished video. Amazingly, despite all of these, the highlight capability is able to complete the extraction process in just 2 seconds.
See for yourself how the finished video by the highlight capability compares with the original video.
Backing Technology​The highlight capability stands out from the crowd by adopting models and a frame assessment scheme that are iteratively optimized. Technically and specifically speaking:
The capability introduces AMediaCodec for hardware decoding and Open Graphics Library (OpenGL) for rendering frames and automatically adjusting the frame dimensions according to the screen dimensions. The capability algorithm uses multiple neural network models. In this way, the capability checks the device model where it runs and then automatically chooses to run on NPU, CPU, or GPU. Consequently, the capability delivers a higher running performance.
To provide the extraction result more quickly, the highlight capability uses the two-stage algorithm of sparse sampling to dense sampling, checks how content distributed among numerous videos, and adopts the frame buffer. All these contribute to a higher efficiency of determining the most attractive video frames. To ensure high performance of the algorithm, the capability adopts the thread pool scheduling and producer-consumer model, to ensure that the video decoder and models can run at the same time.
During the sparse sampling stage, the capability decodes and processes some (up to 15) key frames in a video. The interval between the key frames is no less than 2 seconds. During the dense sampling stage, the algorithm picks out the best key frame and then extracts frames before and after to further analyze the highlighted part of the video.
The extraction result is closely related to the key frame position. The processing result of the highlight capability will not be ideal when the sampling points are not dense enough because, for example, the video does not have enough key frames or the duration is too long (greater than 1 minute). For the capability to deliver optimal performance, it recommends that the duration of the input video be less than 60 seconds.
Let's now move on to how this capability can be integrated.
Integration Process​Preparations​Make necessary preparations before moving on to the next part. Required steps include:
Configure the app information in AppGallery Connect.
Integrate the SDK of HMS Core.
Configure obfuscation scripts.
Declare necessary permissions.
Setting up the Video Editing Project​1. Configure the app authentication information by using either an access token or API key.
Method 1: Call setAccessToken to set an access token, which is required only once during app startup.
Code:
MediaApplication.getInstance().setAccessToken("your access token");
Method 2: Call setApiKey to set an API key, which is required only once during app startup.
Code:
MediaApplication.getInstance().setApiKey("your ApiKey");
2. Set a License ID.
This ID is used to manage the usage quotas of Video Editor Kit and must be unique.
Code:
MediaApplication.getInstance().setLicenseId("License ID");
Initialize the runtime environment of HuaweiVideoEditor.
When creating a video editing project, we first need to create an instance of HuaweiVideoEditor and initialize its runtime environment. When you exit the project, the instance shall be released.
Create an instance of HuaweiVideoEditor.
Code:
HuaweiVideoEditor editor = HuaweiVideoEditor.create(getApplicationContext());
Determine the layout of the preview area.
Such an area renders video images, and this is implemented by SurfaceView within the fundamental capability SDK. Before the area is created, we need to specify its layout.
Code:
<LinearLayout
android:id="@+id/video_content_layout"
android:layout_width="0dp"
android:layout_height="0dp"
android:background="@color/video_edit_main_bg_color"
android:gravity="center"
android:orientation="vertical" />
// Specify a preview area.
LinearLayout mSdkPreviewContainer = view.findViewById(R.id.video_content_layout);
// Design the layout of the area.
editor.setDisplay(mSdkPreviewContainer);
Initialize the runtime environment. If the license verification fails, LicenseException will be thrown.
After the HuaweiVideoEditor instance is created, it will not use any system resources, and we need to manually set the initialization time for the runtime environment. Then, the fundamental capability SDK will internally create necessary threads and timers.
Code:
try {
editor.initEnvironment();
} catch (LicenseException error) {
SmartLog.e(TAG, "initEnvironment failed: " + error.getErrorMsg());
finish();
return;
}
Integrating the Highlight Capability​
Code:
// Create an object that will be processed by the highlight capability.
HVEVideoSelection hveVideoSelection = new HVEVideoSelection();
// Initialize the engine of the highlight capability.
hveVideoSelection.initVideoSelectionEngine(new HVEAIInitialCallback() {
@Override
public void onProgress(int progress) {
// Callback when the initialization progress is received.
}
@Override
public void onSuccess() {
// Callback when the initialization is successful.
}
@Override
public void onError(int errorCode, String errorMessage) {
// Callback when the initialization failed.
}
});
// After the initialization is successful, extract the highlighted video. filePath indicates the video file path, and duration indicates the desired duration for the highlighted video.
hveVideoSelection.getHighLight(filePath, duration, new HVEVideoSelectionCallback() {
@Override
public void onResult(long start) {
// The highlighted video is successfully extracted.
}
});
// Release the highlight engine.
hveVideoSelection.releaseVideoSelectionEngine();
Conclusion​The vlog has been playing a vital part in this we-media era since its appearance. In the past, there were just a handful of people who could create a vlog, because the process of picking out the most interesting part from the original video could be so demanding.
Thanks to smart mobile app technology, even video editing amateurs can now create a vlog because much of the process can be completed automatically by an app with the function of highlighted video extraction.
The highlight capability from the Video Editor Kit is one such function. This capability introduces a set of features to deliver incredible results, such as AMediaCodec, OpenGL, neural networks, a two-stage algorithm (sparse sampling to dense sampling), and more. This capability can help create either a highlighted video extractor or build a highlighted video extraction feature in an app.

Create an HD Video Player with HDR Tech

{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
​What Is HDR and Why Does It Matter​Streaming technology has improved significantly, giving rise to higher and higher video resolutions from those at or below 480p (which are known as standard definition or SD for short) to those at or above 720p (high definition, or HD for short).
The video resolution is vital for all apps. A research that I recently came across backs this up: 62% of people are more likely to negatively perceive a brand that provides a poor-quality video experience, while 57% of people are less likely to share a poor-quality video. With this in mind, it's no wonder that there are so many emerging solutions to enhance video resolution.
One solution is HDR — high dynamic range. It is a post-processing method used in imaging and photography, which mimics what a human eye can see by giving more details to dark areas and improving the contrast. When used in a video player, HDR can deliver richer videos with a higher resolution.
Many HDR solutions, however, are let down by annoying restrictions. These can include a lack of unified technical specifications, high level of difficulty for implementing them, and a requirement for videos in ultra-high definition. I tried to look for a solution without such restrictions and luckily, I found one. That's the HDR Vivid SDK from HMS Core Video Kit. This solution is packed with image-processing features like the opto-electronic transfer function (OETF), tone mapping, and HDR2SDR. With these features, the SDK can equip a video player with richer colors, higher level of detail, and more.
I used the SDK together with the HDR Ability SDK (which can also be used independently) to try the latter's brightness adjustment feature, and found that they could deliver an even better HDR video playback experience. And on that note, I'd like to share how I used these two SDKs to create a video player.
Before Development​1. Configure the app information as needed in AppGallery Connect.
2. Integrate the HMS Core SDK.
For Android Studio, the SDK can be integrated via the Maven repository. Before the development procedure, the SDK needs to be integrated into the Android Studio project.
3. Configure the obfuscation scripts.
4. Add permissions, including those for accessing the Internet, for obtaining the network status, for accessing the Wi-Fi network, for writing data into the external storage, for reading data from the external storage, for reading device information, for checking whether a device is rooted, and obtaining the wake lock. (The last three permissions are optional.)
App Development​Preparations​1. Check whether the device is capable of decoding an HDR Vivid video. If the device has such a capability, the following function will return true.
Code:
public boolean isSupportDecode() {
// Check whether the device supports MediaCodec.
MediaCodecList mcList = new MediaCodecList(MediaCodecList.ALL_CODECS);
MediaCodecInfo[] mcInfos = mcList.getCodecInfos();
for (MediaCodecInfo mci : mcInfos) {
// Filter out the encoder.
if (mci.isEncoder()) {
continue;
}
String[] types = mci.getSupportedTypes();
String typesArr = Arrays.toString(types);
// Filter out the non-HEVC decoder.
if (!typesArr.contains("hevc")) {
continue;
}
for (String type : types) {
// Check whether 10-bit HEVC decoding is supported.
MediaCodecInfo.CodecCapabilities codecCapabilities = mci.getCapabilitiesForType(type);
for (MediaCodecInfo.CodecProfileLevel codecProfileLevel : codecCapabilities.profileLevels) {
if (codecProfileLevel.profile == HEVCProfileMain10
|| codecProfileLevel.profile == HEVCProfileMain10HDR10
|| codecProfileLevel.profile == HEVCProfileMain10HDR10Plus) {
// true means supported.
return true;
}
}
}
}
// false means unsupported.
return false;
}
2. Parse a video to obtain information about its resolution, OETF, color space, and color format. Then save the information in a custom variable. In the example below, the variable is named as VideoInfo.
Code:
public class VideoInfo {
private int width;
private int height;
private int tf;
private int colorSpace;
private int colorFormat;
private long durationUs;
}
3. Create a SurfaceView object that will be used by the SDK to process the rendered images.
Code:
// surface_view is defined in a layout file.
SurfaceView surfaceView = (SurfaceView) view.findViewById(R.id.surface_view);
4. Create a thread to parse video streams from a video.
Rendering and Transcoding a Video​1. Create and then initialize an instance of HdrVividRender.
Code:
HdrVividRender hdrVividRender = new HdrVividRender();
hdrVividRender.init();
2. Configure the OETF and resolution for the video source.
Code:
// Configure the OETF.
hdrVividRender.setTransFunc(2);
// Configure the resolution.
hdrVividRender.setInputVideoSize(3840, 2160);
When the SDK is used on an Android device, only the rendering mode for input is supported.
3. Configure the brightness for the output. This step is optional.
Code:
hdrVividRender.setBrightness(700);
4. Create a Surface object, which will serve as the input. This method is called when HdrVividRender works in rendering mode, and the created Surface object is passed as the inputSurface parameter of configure to the SDK.
Code:
Surface inputSurface = hdrVividRender.createInputSurface();
5. Configure the output parameters.
Set the dimensions of the rendered Surface object. This step is necessary in the rendering mode for output.
Code:
// surfaceView is the video playback window.
hdrVividRender.setOutputSurfaceSize(surfaceView.getWidth(), surfaceView.getHeight());
Set the color space for the buffered output video, which can be set in the transcoding mode for output. This step is optional. However, when no color space is set, BT.709 is used by default.
Code:
hdrVividRender.setColorSpace(HdrVividRender.COLORSPACE_P3);
Set the color format for the buffered output video, which can be set in the transcoding mode for output. This step is optional. However, when no color format is specified, R8G8B8A8 is used by default.
Code:
hdrVividRender.setColorFormat(HdrVividRender.COLORFORMAT_R8G8B8A8);
6. When the rendering mode is used as the output mode, the following APIs are required.
Code:
hdrVividRender.configure(inputSurface, new HdrVividRender.InputCallback() {
@Override
public int onGetDynamicMetaData(HdrVividRender hdrVividRender, long pts) {
// Set the static metadata, which needs to be obtained from the video source.
HdrVividRender.StaticMetaData lastStaticMetaData = new HdrVividRender.StaticMetaData();
hdrVividRender.setStaticMetaData(lastStaticMetaData);
// Set the dynamic metadata, which also needs to be obtained from the video source.
ByteBuffer dynamicMetaData = ByteBuffer.allocateDirect(10);
hdrVividRender.setDynamicMetaData(20000, dynamicMetaData);
return 0;
}
}, surfaceView.getHolder().getSurface(), null);
7. When the transcoding mode is used as the output mode, call the following APIs.
Code:
hdrVividRender.configure(inputSurface, new HdrVividRender.InputCallback() {
@Override
public int onGetDynamicMetaData(HdrVividRender hdrVividRender, long pts) {
// Set the static metadata, which needs to be obtained from the video source.
HdrVividRender.StaticMetaData lastStaticMetaData = new HdrVividRender.StaticMetaData();
hdrVividRender.setStaticMetaData(lastStaticMetaData);
// Set the dynamic metadata, which also needs to be obtained from the video source.
ByteBuffer dynamicMetaData = ByteBuffer.allocateDirect(10);
hdrVividRender.setDynamicMetaData(20000, dynamicMetaData);
return 0;
}
}, null, new HdrVividRender.OutputCallback() {
@Override
public void onOutputBufferAvailable(HdrVividRender hdrVividRender, ByteBuffer byteBuffer,
HdrVividRender.BufferInfo bufferInfo) {
// Process the buffered data.
}
});
new HdrVividRender.OutputCallback() is used for asynchronously processing the returned buffered data. If this method is not used, the read method can be used instead. For example:
Code:
hdrVividRender.read(new BufferInfo(), 10); // 10 is a timestamp, which is determined by your app.
8. Start the processing flow.
Code:
hdrVividRender.start();
9. Stop the processing flow.
Code:
hdrVividRender.stop();
10. Release the resources that have been occupied.
Code:
hdrVividRender.release();
hdrVividRender = null;
During the above steps, I noticed that when the dimensions of Surface change, setOutputSurfaceSize has to be called to re-configure the dimensions of the Surface output.
Besides, in the rendering mode for output, when WisePlayer is switched from the background to the foreground or vice versa, the Surface object will be destroyed and then re-created. In this case, there is a possibility that the HdrVividRender instance is not destroyed. If so, the setOutputSurface API needs to be called so that a new Surface output can be set.
Setting Up HDR Capabilities​HDR capabilities are provided in the class HdrAbility. It can be used to adjust brightness when the HDR Vivid SDK is rendering or transcoding an HDR Vivid video.
1. Initialize the function of brightness adjustment.
Code:
HdrAbility.init(getApplicationContext())
2. Enable the HDR feature on the device. Then, the maximum brightness of the device screen will increase.
Code:
HdrAbility.setHdrAbility(true);
3. Configure the alternative maximum brightness of white points in the output video image data.
Code:
HdrAbility.setBrightness(600);
4. Make the video layer highlighted.
Code:
HdrAbility.setHdrLayer(surfaceView, true);
5. Configure the feature of highlighting the subtitle layer or the bullet comment layer.
Code:
HdrAbility.setCaptionsLayer(captionView, 1.5f);
Summary​Video resolution is an important influencer of user experience for mobile apps. HDR is often used to post-process video, but it is held back by a number of restrictions, which are resolved by the HDR Vivid SDK from Video Kit.
This SDK is loaded with features for image processing such as the OETF, tone mapping, and HDR2SDR, so that it can mimic what human eyes can see to deliver immersive videos that can be enhanced even further with the help of the HDR Ability SDK from the same kit. The functionality and straightforward integration process of these SDKs make them ideal for implementing the HDR feature into a mobile app.

How to Develop a Portrait Retouching Function

Portrait Retouching Importance​Mobile phone camera technology is evolving — wide-angle lens, optical image stabilization, to name but a few. Thanks to this, video recording and mobile image editing apps are emerging one after another, utilizing technology to foster greater creativity.
Among these apps, live-streaming apps are growing with great momentum, thanks to an explosive number of streamers and viewers.
One function that a live-streaming app needs is portrait retouching. The reason is that though mobile phone camera parameters are already staggering, portraits captured by the camera can also be distorted for different reasons. For example, in a dim environment, a streamer's skin tone might appear dark, while factors such as the width of camera lens and shooting angle can make them look wide in videos. Issues like these can affect how viewers feel about a live video and how streamers feel about themselves, signaling the need for a portrait retouching function to address these issues.
I've developed a live-streaming demo app with such a function. Before I developed it, I identified two issues developing this function for a live-streaming app.
First, this function must be able to process video images in real time. A long duration between image input to output compromises interaction between a streamer and their viewers.
Secondly, this function requires a high level of face detection accuracy, to prevent the processed portrait from deformation, or retouched areas from appearing on unexpected parts.
To solve these challenges, I tested several available portrait retouching solutions and settled on the beauty capability from HMS Core Video Editor Kit. Let's see how the capability works to understand how it manages to address the challenges.
How the Capability Addresses the Challenges​This capability adopts the CPU+NPU+GPU heterogeneous parallel framework, which allows it to process video images in real time. The capability algorithm runs faster, but requires less power.
Specifically speaking, the beauty capability delivers a processing frequency of over 50 fps in a device-to-device manner. For a video that contains multiple faces, the capability can simultaneously process a maximum of two faces, whose areas are the biggest in the video. This takes as little as 10 milliseconds to complete.
The capability uses 855 dense facial landmarks so that it can accurately recognize a face, allowing the capability to adapt its effects to a face that moves too fast or at a big angle during live streaming.
To ensure an excellent retouching effect, the beauty capability adopts detailed face segmentation and neutral gray for softening skin. As a result, the final effect looks very authentic.
Not only that, the capability is equipped with multiple, configurable retouching parameters. This feature, I think, is considerate and makes the capability deliver an even better user experience — considering that it is impossible to have a portrait retouching panacea that can satisfy all users. Developers like me can provide these parameters (including those for skin softening, skin tone adjustment, face contour adjustment, eye size adjustment, and eye brightness adjustment) directly to users, rather than struggle to design the parameters by ourselves. This offers more time for fine-tuning portraits in video images.
Knowing these features of the capability, I believed that it could help me create a portrait retouching function for my demo app. So let's move on to see how I developed my app.
Demo Development​Preparations​Make sure the development environment is ready.
Configure app information in AppGallery Connect, including registering as a developer on the platform, creating an app, generating a signing certificate fingerprint, configuring the fingerprint, and enabling the kit.
Integrate the HMS Core SDK.
Configure obfuscation scripts.
Declare necessary permissions.
Capability Integration​1. Set up the app authentication information. Two methods are available, using an API key or access token respectively:
API key: Call the setApiKey method to set the key, which only needs to be done once during app initialization.
Code:
HVEAIApplication.getInstance().setApiKey("your ApiKey");
The API key is obtained from AppGallery Connect, which is generated during app registration on the platform.
It's worth noting that you do not need to hardcode the key in the app code, or store the key in the app's configuration file. The right way to handle this is to store it on cloud, and obtain it when the app is running.
Access token: Call the setAccessToken method to set the token. This is done only once during app initialization.
Code:
HVEAIApplication.getInstance().setAccessToken("your access token");
The access token is generated by an app itself. Specifically speaking, call the
https://oauth-login.cloud.huawei.com/oauth2/v3/token
API and then an app-level access token is obtained.
2. Integrate the beauty capability.
Code:
// Create an HVEAIBeauty instance.
HVEAIBeauty hveaiBeauty = new HVEAIBeauty();
// Initialize the engine of the capability.
hveaiBeauty.initEngine(new HVEAIInitialCallback() {
@Override
public void onProgress(int progress) {
// Callback when the initialization progress is received.
}
@Override
public void onSuccess() {
// Callback when engine initialization is successful.
}
@Override
public void onError(int errorCode, String errorMessage) {
// Callback when engine initialization failed.
}
});
// Initialize the runtime environment of the capability in OpenGL. The method is called in the rendering thread of OpenGL.
hveaiBeauty.prepare();
// Set textureWidth (width) and textureHeight (height) of the texture to which the capability is applied. This method is called in the rendering thread of OpenGL after initialization or texture change.
// resize is a parameter, indicating the width and height. The parameter value must be greater than 0.
hveaiBeauty.resize(textureWidth, textureHeight);
// Configure the parameters for skin softening, skin tone adjustment, face contour adjustment, eye size adjustment, and eye brightness adjustment. The value of each parameter ranges from 0 to 1.
HVEAIBeautyOptions options = new HVEAIBeautyOptions.Builder().setBigEye(1)
.setBlurDegree(1)
.setBrightEye(1)
.setThinFace(1)
.setWhiteDegree(1)
.build();
// Update the parameters, after engine initialization or parameter change.
hveaiBeauty.updateOptions(options);
// Apply the capability, by calling the method in the rendering thread of OpenGL for each frame. inputTextureId: ID of the input texture; outputTextureId: ID of the output texture.
// The ID of the input texture should correspond to a face that faces upward.
int outputTextureId = hveaiBeauty.process(inputTextureId);
// Release the engine.
hveaiBeauty.releaseEngine();
The development process ends here, so now we can check out how my demo works:
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Not to brag, but I do think the retouching result is ideal and natural: With all the effects added, the processed portrait does not appear deformed.
I've got my desired solution for creating a portrait retouching function. I believe this solution can also play an important role in an image editing app or any app that requires portrait retouching. I'm quite curious as to how you will use it. Now I'm off to find a solution that can "retouch" music instead of photos for a music player app, which can, for example, add more width to a song — Wish me luck!
Conclusion​The live-streaming app market is expanding rapidly, receiving various requirements from streamers and viewers. One of the most desired functions is portrait retouching, which is used to address the distorted portraits and unfavorable video watching experience.
Compared with other kinds of apps, a live-streaming app has two distinct requirements for the portrait retouching function, which are real-time processing of video images and accurate face detection. The beauty capability from HMS Core Video Editor Kit addresses them effectively, by using technologies such as the CPU+NPU+GPU heterogeneous parallel framework and 855 dense facial landmarks. The capability also offers several customizable parameters to enable different users to retouch their portraits as needed. On top of these, the capability can be easily integrated, helping develop an app requiring the portrait retouching feature.

Categories

Resources