Build a Background Eraser app with Huawei ML Kit Image Segmentation - Huawei Developers

Image segmentation is a widely used term in image processing and computer vision world. It is the process of partitioning a digital image into multiple segments. The expected output from an image that we applied image segmentation on is labeling of each pixel into subgroups that we defined. By applying image segmentation, we get a more meaningful image, we get an image that each pixel of which is represented by categories such as human body, sky, plant, food etc.
Image segmentation can be used in many different scenarios. It can be used in photography apps to change the background or apply some effect only on plants or human body etc. It can also be used to determine cancerous cells in a microscope image or get land usage information in a satellite image or to determine the amount of herbicides that needed to be sprayed in a field according to crop density.
Huawei ML Kit’s Image Segmentation service segments same elements (such as human body, plant, and sky) from an image. The elements supported include human body, sky, plant, food, cat, dog, flower, water, sand, building, mountain, and others. By the way, Huawei ML Kit works on all Android phones with ARM architecture and as it is a device-side capability it is free.
ML Kit Image Segmentation allows developers two types of segmentation: human body and multiclass segmentation. We can apply image segmentation on static images and video streams if we select human body type. But we can only apply segmentation for static images in multiclass segmentation.
In multiclass image segmentation the return value of the process is the coordinate array of each element. For instance, if an image consists of human body, sky, plant, and cat, the return value is the coordinate array of the four elements. After this point our app can apply different effects on elements. For example, we can shange the blue sky with a red one.
The return values of human body segmentation include the coordinate array of the human body, human body image with a transparent background, and gray-scale image with a white human body and black background. Our app can further process the elements based on the return values, such as replacing the video background and cutting out the human body from an image.
Today, we are going to build Background Eraser app in Kotlin, on Android Studio. We are going to use human body segmentation function of ML Kit. In this example we are going to learn how to integrate HMS ML Kit into our project first and then we will see how to apply segmentation on images simply.
In the application we will have three imageViews. One for background image, one for selected image (supposed to contain human body) and one for processed image. We will distract human body/ bodies in the selected image and change its background with our selected background. It will be really simple, just to see how to apply this function in an application. I won’t bother you much with details. After we develop this simple application, you can go ahead and add various capabilities to your app.
Let’s start building our demo application step by step from scratch!
1. Firstly, let’s create our project on Android Studio. I named my project as Background Eraser. It’s totally up to you. We can create a project by selecting Empty Activity option and then follow the steps described in this page to create and sign our project in App Gallery Connect.
2. Secondly, In HUAWEI Developer AppGallery Connect, go to Develop > Manage APIs. Make sure ML Kit is activated.
3. Now we have integrated Huawei Mobile Services (HMS) into our project. Now let’s follow the documentation on developer.huawei.com and find the packages to add to our project. In the website click Developer / HMS Core/ AI / ML Kit. Here you will find introductory information to services, references, SDKs to download and others. Under ML Kit tab follow Android / Getting Started / Integrating HMS Core SDK / Adding Build Dependencies / Integrating the Image Segmentation SDK. We can follow the guide here to add image segmentation capability to our project. As we are not going to use multiclass segmentation in this project we only add base SDK and human body segmentation package. We have also one meta-data tag to be added into our AndroidManifest.xml file. After the integration your app-level build.gradle file will look like this.
Code:
apply plugin: 'com.android.application'
apply plugin: 'kotlin-android'
apply plugin: 'kotlin-android-extensions'
apply plugin:'com.huawei.agconnect'
android {
compileSdkVersion 30
buildToolsVersion "30.0.1"
defaultConfig {
applicationId "com.demo.backgrounderaser"
minSdkVersion 23
targetSdkVersion 30
versionCode 1
versionName "1.0"
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
}
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
}
}
}
dependencies {
implementation fileTree(dir: "libs", include: ["*.jar"])
implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
implementation 'androidx.core:core-ktx:1.3.1'
implementation 'androidx.appcompat:appcompat:1.2.0'
implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
testImplementation 'junit:junit:4.12'
androidTestImplementation 'androidx.test.ext:junit:1.1.1'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
//AGC Core
implementation'com.huawei.agconnect:agconnect-core:1.3.1.300'
//Image Segmentation Base SDK
implementation 'com.huawei.hms:ml-computer-vision-segmentation:2.0.2.300'
//Image Segmentation Human Body Model
implementation 'com.huawei.hms:ml-computer-vision-image-segmentation-body-model:2.0.2.300'
}
And your project-level build.gradle file will look like this.
Code:
buildscript {
ext.kotlin_version = "1.4.0"
repositories {
google()
jcenter()
maven {url 'http://developer.huawei.com/repo/'}
}
dependencies {
classpath "com.android.tools.build:gradle:4.0.1"
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
classpath 'com.huawei.agconnect:agcp:1.3.1.300'
}
}
allprojects {
repositories {
google()
jcenter()
maven {url 'http://developer.huawei.com/repo/'}
}
}
task clean(type: Delete) {
delete rootProject.buildDir
}
Don’t forget to add the following meta-data tags in your AndroidManifest.xml. This is for automatic update of the machine learning model.
Code:
<manifest … >
<application
… >
<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value= "imgseg"/>
</application>
</manifest>
4. We can select applying human body segmentation on static images or video streams. In this project we are going to see how to do this on static images. To apply segmentation on images firstly we need to create our analyzer. setExact() method is to determine if we apply fine segmentation or not. I chose true here. As analyzer type we chose BODY_SEG here. The second option being IMAGE_SEG for multiclass segmentation. setScene() method is for determining result type. Here we chose FOREGROUND_ONLY, meaning a human body image with a transparent background and an original image for segmentation will be returned. Here is how to create it.
Code:
private lateinit var mAnalyzer: MLImageSegmentationAnalyzer
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
createAnalyzer()
}
private fun createAnalyzer(): MLImageSegmentationAnalyzer {
val analyzerSetting = MLImageSegmentationSetting.Factory()
.setExact(true)
.setAnalyzerType(MLImageSegmentationSetting.BODY_SEG)
.setScene(MLImageSegmentationScene.FOREGROUND_ONLY)
.create()
return MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer(analyzerSetting).also {
mAnalyzer = it
}
}
5. Create a simple layout. It should contain three imageView and two butons. One imageView for background image, one imageView for image with human body to apply segmentation, one imageView for displaying processed image. One button for selecting background image and the other button for selecting image to be processed. Here is my example, you can reach the same result with a different design, too.
Code:
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
xmlns:app="http://schemas.android.com/apk/res-auto"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
<androidx.constraintlayout.widget.Guideline
android:id="@+id/guidelineTopHorizontal"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
app:layout_constraintGuide_percent="0.3"
android:orientation="horizontal" />
<androidx.constraintlayout.widget.Guideline
android:id="@+id/guidelineBottomHorizontal"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
app:layout_constraintGuide_percent="0.9"
android:orientation="horizontal" />
<androidx.constraintlayout.widget.Guideline
android:id="@+id/guidelineCenterVertical"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
app:layout_constraintGuide_percent="0.5"
android:orientation="vertical" />
<ImageView
android:id="@+id/ivBackgroundFill"
android:layout_width="0dp"
android:layout_height="0dp"
android:padding="@dimen/margin_m"
android:contentDescription="@string/image_to_fill_background"
app:layout_constraintBottom_toBottomOf="@id/guidelineTopHorizontal"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintEnd_toEndOf="@id/guidelineCenterVertical"
app:layout_constraintTop_toTopOf="parent" />
<ImageView
android:id="@+id/ivSelectedBitmap"
android:layout_width="0dp"
android:layout_height="0dp"
android:padding="@dimen/margin_m"
android:contentDescription="@string/image_to_fill_background"
app:layout_constraintBottom_toBottomOf="@id/guidelineTopHorizontal"
app:layout_constraintStart_toStartOf="@id/guidelineCenterVertical"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintTop_toTopOf="parent" />
<ImageView
android:id="@+id/ivProcessedBitmap"
android:layout_width="0dp"
android:layout_height="0dp"
android:padding="@dimen/margin_m"
android:contentDescription="@string/image_to_fill_background"
app:layout_constraintBottom_toBottomOf="@id/guidelineBottomHorizontal"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintTop_toTopOf="@id/guidelineTopHorizontal" />
<Button
android:id="@+id/buttonSelectBackground"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="@string/select_background"
android:contentDescription="@string/image_to_fill_background"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintStart_toStartOf="parent"
app:layout_constraintEnd_toEndOf="@id/guidelineCenterVertical"
app:layout_constraintTop_toTopOf="@id/guidelineBottomHorizontal" />
<Button
android:id="@+id/buttonPickImage"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:text="@string/pick_image"
android:contentDescription="@string/image_to_fill_background"
app:layout_constraintBottom_toBottomOf="parent"
app:layout_constraintStart_toStartOf="@id/guidelineCenterVertical"
app:layout_constraintEnd_toEndOf="parent"
app:layout_constraintTop_toTopOf="@id/guidelineBottomHorizontal" />
</androidx.constraintlayout.widget.ConstraintLayout>
6. In order to get our background image and source image we can use intents. OnClickListeners of our buttons will help us get images via intents from device storage as shown below. In onActivityResult we will appoint bitmaps we get to corresponding variables: mBackgroundFill and mSelectedBitmap. Here WRITE_EXTERNAL_STORAGE permission is requested to automatically update Huawei ML Kit’s image segmentation model.
Code:
class MainActivity : AppCompatActivity() {
companion object {
private const val IMAGE_REQUEST_CODE = 58
private const val BACKGROUND_REQUEST_CODE = 32
}
private lateinit var mAnalyzer: MLImageSegmentationAnalyzer
private var mBackgroundFill: Bitmap? = null
private var mSelectedBitmap: Bitmap? = null
private var mProcessedBitmap: Bitmap? = null
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
if(ContextCompat.checkSelfPermission(this, Manifest.permission.WRITE_EXTERNAL_STORAGE) == PackageManager.PERMISSION_GRANTED)
init()
else
ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.WRITE_EXTERNAL_STORAGE), 0)
}
override fun onRequestPermissionsResult(requestCode: Int, permissions: Array<out String>, grantResults: IntArray) {
super.onRequestPermissionsResult(requestCode, permissions, grantResults)
if (requestCode == 0 && grantResults.isNotEmpty() && grantResults[0] == PackageManager.PERMISSION_GRANTED)
init()
}
private fun init() {
initView()
createAnalyzer()
}
private fun initView() {
buttonPickImage.setOnClickListener { getImage(IMAGE_REQUEST_CODE) }
buttonSelectBackground.setOnClickListener { getImage(BACKGROUND_REQUEST_CODE) }
}
private fun getImage(requestCode: Int) {
Intent(Intent.ACTION_GET_CONTENT).also {
it.type = "image/*"
startActivityForResult(it, requestCode)
}
}
override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
super.onActivityResult(requestCode, resultCode, data)
if (resultCode == Activity.RESULT_OK) {
data?.data?.also {
when(requestCode) {
IMAGE_REQUEST_CODE -> {
mSelectedBitmap = MediaStore.Images.Media.getBitmap(contentResolver, it)
if (mSelectedBitmap != null) {
ivSelectedBitmap.setImageBitmap(mSelectedBitmap)
}
}
BACKGROUND_REQUEST_CODE -> {
mBackgroundFill = MediaStore.Images.Media.getBitmap(contentResolver, it)
if (mBackgroundFill != null) {
ivBackgroundFill.setImageBitmap(mBackgroundFill)
}
}
}
}
}
}
}
Code:
7. Now let’s create our method for analyzing images. It is very simple. It takes a bitmap as parameter, creates an MLFrame object out of it and asynchronously analyzes this MLFrame. It has OnSuccess and OnFailure callbacks. We will try to add the selected background if the analyze result is successful. Please remember that we chose our analyzer’s result type as FOREGROUND_ONLY. So, expect it to return us the original image together with human body image with transparent background.
Code:
private fun analyse(bitmap: Bitmap) {
val mlFrame = MLFrame.fromBitmap(bitmap)
mAnalyzer.asyncAnalyseFrame(mlFrame)
.addOnSuccessListener {
addSelectedBackground(it)
}
.addOnFailureListener {
Log.e(TAG, "analyse -> asyncAnalyseFrame: ", it)
}
}
8. To add human body part onto our background image, we should make sure we have a background image first, then we should obtain a mutable bitmap out of our background to work upon, then we should resize this mutable bitmap according to our selected bitmap to make our image more realistic, then we should create our canvas from our mutable bitmap, then we draw human body part on our canvas and lastly we can use our processed image. Here is our addSelectedBackground() method.
Code:
private fun addSelectedBackground(mlImageSegmentation: MLImageSegmentation) {
if (mBackgroundFill == null) {
Toast.makeText(applicationContext, "Please select a background image!", Toast.LENGTH_SHORT).show()
} else {
var mutableBitmap = if (mBackgroundFill!!.isMutable) {
mBackgroundFill
} else {
mBackgroundFill!!.copy(Bitmap.Config.ARGB_8888, true)
}
if (mutableBitmap != null) {
/*
* If background image size is different than our selected image,
* we change our background image's size according to selected image.
*/
if (mutableBitmap.width != mlImageSegmentation.original.width ||
mutableBitmap.height != mlImageSegmentation.original.height) {
mutableBitmap = Bitmap.createScaledBitmap(
mutableBitmap,
mlImageSegmentation.original.width,
mlImageSegmentation.original.height,
false)
}
val canvas = mutableBitmap?.let { Canvas(it) }
canvas?.drawBitmap(mlImageSegmentation.foreground, 0F, 0F, null)
mProcessedBitmap = mutableBitmap
ivProcessedBitmap.setImageBitmap(mProcessedBitmap)
}
}
}
9. Here is the whole of MainActivity. You can find this project on Github, too.
Code:
class MainActivity : AppCompatActivity() {
companion object {
private const val TAG = "BE_MainActivity"
private const val IMAGE_REQUEST_CODE = 58
private const val BACKGROUND_REQUEST_CODE = 32
}
private lateinit var mAnalyzer: MLImageSegmentationAnalyzer
private var mBackgroundFill: Bitmap? = null
private var mSelectedBitmap: Bitmap? = null
private var mProcessedBitmap: Bitmap? = null
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
if(ContextCompat.checkSelfPermission(this, Manifest.permission.WRITE_EXTERNAL_STORAGE) == PackageManager.PERMISSION_GRANTED)
init()
else
ActivityCompat.requestPermissions(this, arrayOf(Manifest.permission.WRITE_EXTERNAL_STORAGE), 0)
}
override fun onRequestPermissionsResult(requestCode: Int, permissions: Array<out String>, grantResults: IntArray) {
super.onRequestPermissionsResult(requestCode, permissions, grantResults)
if (requestCode == 0 && grantResults.isNotEmpty() && grantResults[0] == PackageManager.PERMISSION_GRANTED)
init()
}
private fun init() {
initView()
createAnalyzer()
}
private fun initView() {
buttonPickImage.setOnClickListener { getImage(IMAGE_REQUEST_CODE) }
buttonSelectBackground.setOnClickListener { getImage(BACKGROUND_REQUEST_CODE) }
}
private fun createAnalyzer(): MLImageSegmentationAnalyzer {
val analyzerSetting = MLImageSegmentationSetting.Factory()
.setExact(true)
.setAnalyzerType(MLImageSegmentationSetting.BODY_SEG)
.setScene(MLImageSegmentationScene.FOREGROUND_ONLY)
.create()
return MLAnalyzerFactory.getInstance().getImageSegmentationAnalyzer(analyzerSetting).also {
mAnalyzer = it
}
}
private fun analyse(bitmap: Bitmap) {
val mlFrame = MLFrame.fromBitmap(bitmap)
mAnalyzer.asyncAnalyseFrame(mlFrame)
.addOnSuccessListener {
addSelectedBackground(it)
}
.addOnFailureListener {
Log.e(TAG, "analyse -> asyncAnalyseFrame: ", it)
}
}
private fun addSelectedBackground(mlImageSegmentation: MLImageSegmentation) {
if (mBackgroundFill == null) {
Toast.makeText(applicationContext, "Please select a background image!", Toast.LENGTH_SHORT).show()
} else {
var mutableBitmap = if (mBackgroundFill!!.isMutable) {
mBackgroundFill
} else {
mBackgroundFill!!.copy(Bitmap.Config.ARGB_8888, true)
}
if (mutableBitmap != null) {
/*
* If background image size is different than our selected image,
* we change our background image's size according to selected image.
*/
if (mutableBitmap.width != mlImageSegmentation.original.width ||
mutableBitmap.height != mlImageSegmentation.original.height) {
mutableBitmap = Bitmap.createScaledBitmap(
mutableBitmap,
mlImageSegmentation.original.width,
mlImageSegmentation.original.height,
false)
}
val canvas = mutableBitmap?.let { Canvas(it) }
canvas?.drawBitmap(mlImageSegmentation.foreground, 0F, 0F, null)
mProcessedBitmap = mutableBitmap
ivProcessedBitmap.setImageBitmap(mProcessedBitmap)
}
}
}
private fun getImage(requestCode: Int) {
Intent(Intent.ACTION_GET_CONTENT).also {
it.type = "image/*"
startActivityForResult(it, requestCode)
}
}
override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent?) {
super.onActivityResult(requestCode, resultCode, data)
if (resultCode == Activity.RESULT_OK) {
data?.data?.also {
when(requestCode) {
IMAGE_REQUEST_CODE -> {
mSelectedBitmap = MediaStore.Images.Media.getBitmap(contentResolver, it)
if (mSelectedBitmap != null) {
ivSelectedBitmap.setImageBitmap(mSelectedBitmap)
analyse(mSelectedBitmap!!)
}
}
BACKGROUND_REQUEST_CODE -> {
mBackgroundFill = MediaStore.Images.Media.getBitmap(contentResolver, it)
if (mBackgroundFill != null) {
ivBackgroundFill.setImageBitmap(mBackgroundFill)
mSelectedBitmap?.let { analyse(it) }
}
}
}
}
}
}
}
10. Well done! We have finished all the steps and created our project.
11. We have created a simple Background Eraser app. We can get human body related pixels out of our image and apply different backgrounds to it. This project is to show you the basics. You can go further, create much better projects and come up with various ideas to apply image segmentation to. I hope you enjoyed this article and created your project easily. If you have any questions please ask.