Build a Face Detection App with Huawei ML Kit - Huawei Developers

{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Hi all,
In the era of powerful mobile devices we store thousands of photos, have video calls, shop, manage our bank accounts and perform many other tasks in the palm of our hands. We can also manage to take photos and videos or have video calls with our cameras integrated on our mobile phones. But, it would be inefficient to only use these cameras we carry by ourselves all day to take raw photos and videos. They can perform more, much more.
Face detection services are used by many applications in different industries. It is mainly used for security and entertainment purposes. For example, it can be used by a taxi app to identify its taxi’s customer, it can be used by a smart home app to identify guests’ faces and announce to the host who the person ringing the bell at the door is, it can be used to draw moustache on faces in an entertainment app or it can be used to detect if a drivers eyes are open or not and warn our driver if his/her eyes closed.
In contrary to how many different areas face detection can be used and how important tasks this service performs, it is really easy to implement a face detection app with the help of Huawei ML Kit. As it’s a device side capability that works on all Android devices with ARM architecture, it is completely free, faster and more secure than other services. The face detection service can detect the shapes and features of your user’s face, including their facial expression, age, gender, and wearing.
With face detection service you can detect up to 855 face contour points to locate face coordinates including face contour, eyebrows, eyes, nose, mouth, and ears, and identify the pitch, yaw, and roll angles of a face. You can detect seven facial features including the possibility of opening the left eye, possibility of opening the right eye, possibility of wearing glasses, gender possibility, possibility of wearing a hat, possibility of wearing a beard, and age. In addition to there, you can also detect facial expressions, namely, smiling, neutral, anger, disgust, fear, sadness, and surprise.
Let’s start to build our demo application step by step from scratch!
1.Firstly, let’s create our project on Android Studio. We can create a project selecting Empty Activity option and then follow the steps described in this post to create and sign our project in App Gallery Connect. You can follow this guide.
2. Secondly, In HUAWEI Developer AppGallery Connect, go to Develop > Manage APIs. Make sure ML Kit is activated.
3. Now we have integrated Huawei Mobile Services (HMS) into our project. Now let’s follow the documentation on developer.huawei.com and find the packages to add to our project. In the website click Developer > HMS Core > AI > ML Kit. Here you will find introductory information to services, references, SDKs to download and others. Under ML Kit tab follow Android > Getting Started > Integrating HMS Core SDK > Adding Build Dependencies > Integrating the Face Detection SDK. We can follow the guide here to add face detection capability to our project. To later go round and learn more about this service I added all of the three packages shown here. You can only choose the base SDK or select packages according to your needs. After the integration your app-level build.gradle file will look like this.
Code:
apply plugin: 'com.android.application'
apply plugin: 'kotlin-android'
apply plugin: 'kotlin-android-extensions'
apply plugin: 'com.huawei.agconnect'
android {
compileSdkVersion 29
buildToolsVersion "30.0.1"
defaultConfig {
applicationId "com.demo.faceapp"
minSdkVersion 21
targetSdkVersion 29
versionCode 1
versionName "1.0"
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
}
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
}
}
}
dependencies {
implementation fileTree(dir: "libs", include: ["*.jar"])
implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
implementation 'androidx.core:core-ktx:1.3.1'
implementation 'androidx.appcompat:appcompat:1.2.0'
implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
testImplementation 'junit:junit:4.12'
androidTestImplementation 'androidx.test.ext:junit:1.1.1'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
// Import the base SDK.
implementation 'com.huawei.hms:ml-computer-vision-face:2.0.1.300'
// Import the contour and key point detection model package.
implementation 'com.huawei.hms:ml-computer-vision-face-shape-point-model:2.0.1.300'
// Import the facial expression detection model package.
implementation 'com.huawei.hms:ml-computer-vision-face-emotion-model:2.0.1.300'
// Import the facial feature detection model package.
implementation 'com.huawei.hms:ml-computer-vision-face-feature-model:2.0.1.300'
}
And your project-level build.gradle file will look like this.
Code:
buildscript {
ext.kotlin_version = "1.3.72"
repositories {
google()
jcenter()
maven { url 'https://developer.huawei.com/repo/' }
}
dependencies {
classpath "com.android.tools.build:gradle:4.0.1"
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
classpath 'com.huawei.agconnect:agcp:1.3.1.300'
}
}
allprojects {
repositories {
google()
jcenter()
maven { url 'https://developer.huawei.com/repo/' }
}
}
task clean(type: Delete) {
delete rootProject.buildDir
}
Don’t forget to add the following meta-data tags in your AndroidManifest.xml. This is for automatic update of the machine learning model.
Code:
<?xml version="1.0" encoding="utf-8"?>
<manifest ...
<application ...
</application>
<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value= "face"/>
</manifest>
4. Now we can select detecting faces on a static image or on a camera stream. Let’s choose detecting faces on a camera stream for this example. Firstly, let’s create our analyzer. Its type is MLFaceAnalyzer. It is responsible for analyzing the faces detected. Here is the sample implementation. We can also use our MLFaceAnalyzer with default settings to make it simple.
Code:
private lateinit var mAnalyzer: MLFaceAnalyzer
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
mAnalyzer = createAnalyzer()
}
private fun createAnalyzer(): MLFaceAnalyzer {
val settings = MLFaceAnalyzerSetting.Factory()
.allowTracing()
.setFeatureType(MLFaceAnalyzerSetting.TYPE_FEATURES)
.setShapeType(MLFaceAnalyzerSetting.TYPE_SHAPES)
.setMinFaceProportion(.5F)
.setKeyPointType(MLFaceAnalyzerSetting.TYPE_KEYPOINTS)
.create()
return MLAnalyzerFactory.getInstance().getFaceAnalyzer(settings)
}
5. Create a simple layout. Use two surfaceViews, one above the other, for camera frames and for our overlay. Because, later we will draw some shapes on our overlay view. Here is a sample layout.
Code:
<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:layout_width="match_parent"
android:layout_height="match_parent"
tools:context=".MainActivity">
<SurfaceView
android:id="@+id/surfaceViewCamera"
android:layout_width="match_parent"
android:layout_height="match_parent" />
<SurfaceView
android:id="@+id/surfaceViewOverlay"
android:layout_width="match_parent"
android:layout_height="match_parent" />
</androidx.constraintlayout.widget.ConstraintLayout>
6. We should prepare our views. We need two surfaceHolders in our application. We are going to make surfaceHolderOverlay transparent, because we want to see our camera frames. Later we are going to add a callback to our surfaceHolderCamera to know when it’s created, changed and destroyed. Let’s create them.
Code:
private lateinit var mAnalyzer: MLFaceAnalyzer
private var surfaceHolderCamera: SurfaceHolder? = null
private var surfaceHolderOverlay: SurfaceHolder? = null
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
mAnalyzer = createAnalyzer()
prepareViews()
}
private fun prepareViews() {
surfaceHolderCamera = surfaceViewCamera.holder
surfaceHolderOverlay = surfaceViewOverlay.holder
surfaceHolderOverlay?.setFormat(PixelFormat.TRANSPARENT)
surfaceHolderCamera?.addCallback(surfaceHolderCallback)
}
private val surfaceHolderCallback = object : SurfaceHolder.Callback {
override fun surfaceChanged(holder: SurfaceHolder?, format: Int, width: Int, height: Int) {
}
override fun surfaceDestroyed(holder: SurfaceHolder?) {
}
override fun surfaceCreated(holder: SurfaceHolder?) {
}
}
7. Now we can create our LensEngine. It is a magic class that handles camera frames for us. You can set different settings to your LensEngine. Here is how you can create it simply. As you can see in the example, the order of width and height passed to LensEngine changes according to orientation. We can create our LensEngine inside surfaceChanged method of surfaceHolderCallback and release it inside surfaceDestroyed. Here is an example of creating and running LensEngine. LensEngine needs a surfaceHolder or a surfaceTexture to run on.
Code:
private lateinit var mAnalyzer: MLFaceAnalyzer
private lateinit var mLensEngine: LensEngine
private var surfaceHolderCamera: SurfaceHolder? = null
private var surfaceHolderOverlay: SurfaceHolder? = null
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
mAnalyzer = createAnalyzer()
prepareViews()
}
private fun prepareViews() {
surfaceHolderCamera = surfaceViewCamera.holder
surfaceHolderOverlay = surfaceViewOverlay.holder
surfaceHolderOverlay?.setFormat(PixelFormat.TRANSPARENT)
surfaceHolderCamera?.addCallback(surfaceHolderCallback)
}
private val surfaceHolderCallback = object : SurfaceHolder.Callback {
override fun surfaceChanged(holder: SurfaceHolder?, format: Int, width: Int, height: Int) {
mLensEngine = createLensEngine(width, height)
mLensEngine.run(holder)
}
override fun surfaceDestroyed(holder: SurfaceHolder?) {
mLensEngine.release()
}
override fun surfaceCreated(holder: SurfaceHolder?) {
}
}
private fun createLensEngine(width: Int, height: Int): LensEngine {
val lensEngineCreator = LensEngine.Creator(this, mAnalyzer)
.applyFps(20F)
.setLensType(LensEngine.FRONT_LENS)
.enableAutomaticFocus(true)
return if (resources.configuration.orientation == Configuration.ORIENTATION_PORTRAIT) {
lensEngineCreator.let {
it.applyDisplayDimension(height, width)
it.create()
}
} else {
lensEngineCreator.let {
it.applyDisplayDimension(width, height)
it.create()
}
}
}
8. We also need somewhere to receive detected results and interact with them. For this purpose we create our FaceAnalyzerTransactor class, you can name it as you wish. It should implement MLAnalyzer.MLTransactor<MLFace> interface. We are going to set an overlay which is of type SurfaceHolder, get our canvas from this overlay and draw some shapes on the canvas. We have the required data about the detected face inside transactResult method. Here is the sample implementation of the whole of our FaceAnalyzerTransactor class.
Code:
class FaceAnalyzerTransactor : MLAnalyzer.MLTransactor<MLFace> {
private var mOverlay: SurfaceHolder? = null
fun setOverlay(surfaceHolder: SurfaceHolder) {
mOverlay = surfaceHolder
}
override fun transactResult(result: MLAnalyzer.Result<MLFace>?) {
draw(result?.analyseList)
}
private fun draw(faces: SparseArray<MLFace>?) {
val canvas = mOverlay?.lockCanvas()
if (canvas != null && faces != null) {
//Clear the canvas
canvas.drawColor(0, PorterDuff.Mode.CLEAR)
for (face in faces.valueIterator()) {
//Draw all 855 points of the face. If Front Lens is selected, change x points side.
for (point in face.allPoints) {
val x = mOverlay?.surfaceFrame?.right?.minus(point.x)
if (x != null) {
Paint().also {
it.color = Color.YELLOW
it.style = Paint.Style.FILL
it.strokeWidth = 16F
canvas.drawPoint(x, point.y, it)
}
}
}
//Prepare a string to show if the user smiles or not and draw a text on the canvas.
val smilingString = if (face.emotions.smilingProbability > 0.5) "SMILING" else "NOT SMILING"
Paint().also {
it.color = Color.RED
it.textSize = 60F
it.textAlign = Paint.Align.CENTER
canvas.drawText(smilingString, face.border.exactCenterX(), face.border.exactCenterY(), it)
}
}
mOverlay?.unlockCanvasAndPost(canvas)
}
}
override fun destroy() {
}
}
9. Create a FaceAnalyzerTransactor instance in MainActivity and use it as shown below.
Code:
private lateinit var mAnalyzer: MLFaceAnalyzer
private lateinit var mLensEngine: LensEngine
private lateinit var mFaceAnalyzerTransactor: FaceAnalyzerTransactor
private var surfaceHolderCamera: SurfaceHolder? = null
private var surfaceHolderOverlay: SurfaceHolder? = null
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
init()
}
private fun init() {
mAnalyzer = createAnalyzer()
mFaceAnalyzerTransactor = FaceAnalyzerTransactor()
mAnalyzer.setTransactor(mFaceAnalyzerTransactor)
prepareViews()
}
Also, don’t forget to set the overlay of our transactor. We can do this inside surfaceChanged method like this.
Code:
private val surfaceHolderCallback = object : SurfaceHolder.Callback {
override fun surfaceChanged(holder: SurfaceHolder?, format: Int, width: Int, height: Int) {
mLensEngine = createLensEngine(width, height)
surfaceHolderOverlay?.let { mFaceAnalyzerTransactor.setOverlay(it) }
mLensEngine.run(holder)
}
override fun surfaceDestroyed(holder: SurfaceHolder?) {
mLensEngine.release()
}
override fun surfaceCreated(holder: SurfaceHolder?) {
}
}
10. We are almost done! Don’t forget to ask for permissions from our users. We need CAMERA and WRITE_EXTERNAL_STORAGE permissions. WRITE_EXTERNAL_STORAGE permission is for automatically updating the machine learning model. Add these permissions to your AndroidManifest.xml and ask from user to grant them at runtime. Here is a simple example.
Code:
private val requiredPermissions = arrayOf(Manifest.permission.CAMERA, Manifest.permission.WRITE_EXTERNAL_STORAGE)
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)
if (hasPermissions(requiredPermissions))
init()
else
ActivityCompat.requestPermissions(this, requiredPermissions, 0)
}
private fun hasPermissions(permissions: Array<String>) = permissions.all {
ContextCompat.checkSelfPermission(this, it) == PackageManager.PERMISSION_GRANTED
}
override fun onRequestPermissionsResult(requestCode: Int, permissions: Array<out String>, grantResults: IntArray) {
super.onRequestPermissionsResult(requestCode, permissions, grantResults)
if (requestCode == 0 && grantResults.isNotEmpty() && hasPermissions(requiredPermissions))
init()
}
11. Well done! We have finished all the steps and created our project. Now we can test it. Here are some examples.
12. We have created a simple FaceApp which detects faces, face features and emotions. You can produce countless number of types of face detection apps, it is up to your imagination. ML Kit empowers your apps with the power of AI. If you have any questions, please ask through the link below. You can also find this project on Github.
Related Links
Thanks to Oğuzhan Demirci for this article.
Original post: https://medium.com/huawei-developers/build-a-face-detection-app-with-huawei-ml-kit-32caec06484

---

Nice and useful article
Does face detection depend on specific hardware devices?