Tips on Creating a Cutout Tool - Huawei Developers

{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
In photography, cutout is a function that is often used to editing images, such as removing the background. To achieve this function, a technique known as green screen is universally used, which is also called as chroma keying. This technique requires a green background to be added manually.
This, however, makes the green screen-dependent cutout a challenge to those new to video/image editing. The reason is that most images and videos do not come with a green background, and adding such a background is actually quite complex.
Luckily, a number of mobile apps on the market help with this, as they are able to automatically cut out the desired object for users to later edit. To create an app that is capable of doing this, I turned to the recently released object segmentation capability from HMS Core Video Editor Kit for help. This capability utilizes the AI algorithm, instead of the green screen, to intelligently separate an object from other parts of an image or a video, delivering an ideal segmentation result for removing the background and many other further editing operations.
This is what my demo has achieved with the help of the capability.
It is a perfect cutout, right? As you can see, the cut object comes with a smooth edge, without any unwanted parts appearing in the original video.
Before moving on to how I created this cutout tool with the help of object segmentation, let's see what lies behind the capability.
How It Works​The object segmentation capability adopts an interactive way for cutting out objects. A user first taps or draws a line on the object to be cut out, and then the interactive segmentation algorithm checks the track of the user's tap and intelligently identifies their intent. The capability then selects and cuts out the object. The object segmentation capability performs interactive segmentation on the first video frame to obtain the mask of the object to be cut out. The model supporting the capability traverses frames following the first frame by using the mask obtained from the first frame and applying it to subsequent frames, and then matches the mask with the object in them before cutting the object out.
The model assigns frames with different weights, according to the segmentation accuracy of each frame. It then blends the weighted segmentation result of the intermediate frame with the mask obtained from the first frame, in order to segment the desired object from other frames. Consequently, the capability manages to cut out an object as wholly as possible, delivering a higher segmentation accuracy.
What makes the capability better is that it has no restrictions on object types. As long as an object is distinctive to the other parts of the image or video and is against a simple background, the capability can cleanly cut out this object.
Now, let's check out how the capability can be integrated.
Integration Procedure​Making Preparations​There are necessary steps to do before the next part. The steps include:
Configure app information in AppGallery Connect.
Integrate the SDK of HMS Core.
Configure obfuscation scripts.
Apply for necessary permissions.
Setting Up the Video Editing Project​1. Configure the app authentication information. Available options include:
Call setAccessToken to set an access token, which is required only once during app startup.
Code:
MediaApplication.getInstance().setAccessToken("your access token");
Or, call setApiKey to set an API key, which is required only once during app startup.
Code:
MediaApplication.getInstance().setApiKey("your ApiKey");
2. Set a License ID.
Because this ID is used to manage the usage quotas of the mentioned service, the ID must be unique.
Code:
MediaApplication.getInstance().setLicenseId("License ID");
1) Initialize the runtime environment for HuaweiVideoEditor.
When creating a video editing project, we first need to create an instance of HuaweiVideoEditor and initialize its runtime environment. When you exit the project, the instance shall be released.
Create a HuaweiVideoEditor object.
Code:
HuaweiVideoEditor editor = HuaweiVideoEditor.create(getApplicationContext());
Determine the layout of the preview area.
Such an area renders video images, and this is implemented by SurfaceView within the fundamental capability SDK. Before the area is created, we need to specify its layout.
Code:
<LinearLayout
android:id="@+id/video_content_layout"
android:layout_width="0dp"
android:layout_height="0dp"
android:background="@color/video_edit_main_bg_color"
android:gravity="center"
android:orientation="vertical" />
// Specify a preview area.
LinearLayout mSdkPreviewContainer = view.findViewById(R.id.video_content_layout);
// Design the layout for the area.
editor.setDisplay(mSdkPreviewContainer);
Initialize the runtime environment. If the license verification fails, LicenseException will be thrown.
After the HuaweiVideoEditor instance is created, it will not use any system resources, and we need to manually set the initialization time for the runtime environment. Then, the fundamental capability SDK will internally create necessary threads and timers.
Code:
try {
editor.initEnvironment();
} catch (LicenseException error) {
SmartLog.e(TAG, "initEnvironment failed: " + error.getErrorMsg());
finish();
return;
}
Integrating Object Segmentation​
Code:
// Initialize the engine of object segmentation.
videoAsset.initSegmentationEngine(new HVEAIInitialCallback() {
@Override
public void onProgress(int progress) {
// Initialization progress.
}
@Override
public void onSuccess() {
// Callback when the initialization is successful.
}
@Override
public void onError(int errorCode, String errorMessage) {
// Callback when the initialization failed.
}
});
// After the initialization is successful, segment a specified object and then return the segmentation result.
// bitmap: video frame containing the object to be segmented; timeStamp: timestamp of the video frame on the timeline; points: set of coordinates determined according to the video frame, and the upper left vertex of the video frame is the coordinate origin. It is recommended that the coordinate count be greater than or equal to two. All of the coordinates must be within the object to be segmented. The object is determined according to the track of coordinates.
int result = videoAsset.selectSegmentationObject(bitmap, timeStamp, points);
// After the handling is successful, apply the object segmentation effect.
videoAsset.addSegmentationEffect(new HVEAIProcessCallback() {
@Override
public void onProgress(int progress) {
// Progress of object segmentation.
}
@Override
public void onSuccess() {
// The object segmentation effect is successfully added.
}
@Override
public void onError(int errorCode, String errorMessage) {
// The object segmentation effect failed to be added.
}
});
// Stop applying the object segmentation effect.
videoAsset.interruptSegmentation();
// Remove the object segmentation effect.
videoAsset.removeSegmentationEffect();
// Release the engine of object segmentation.
videoAsset.releaseSegmentationEngine();
And this concludes the integration process. A cutout function ideal for an image/video editing app was just created.
I just came up with a bunch of fields where object segmentation can help, like live commerce, online education, e-conference, and more.
In live commerce, the capability helps replace the live stream background with product details, letting viewers conveniently learn about the product while watching a live stream.
In online education and e-conference, the capability lets users switch the video background with an icon, or an image of a classroom or meeting room. This makes online lessons and meetings feel more professional.
The capability is also ideal for video editing apps. Take my demo app for example. I used it to add myself to a vlog that my friend created, which made me feel like I was traveling with her.
I think the capability can also be used together with other video editing functions, to realize effects like copying an object, deleting an object, or even adjusting the timeline of an object. I'm sure you've also got some great ideas for using this capability. Let me know in the comments section.
Conclusion​Cutting out objects used to be a thing of people with editing experience, a process that requires the use of a green screen.
Luckily, things have changed thanks to the cutout function found in many mobile apps. It has become a basic function in mobile apps that support video/image editing and is essential for some advanced functions like background removal.
Object segmentation from Video Editor Kit is a straightforward way of implementing the cutout feature into your app. This capability leverages an elaborate AI algorithm and depends on the interactive segmentation method, delivering an ideal and highly accurate object cutout result.