Learn Form Recognition using ML kit | JAVA - Huawei Developers

Introduction
Huawei ML kit allows to improve our lives in countless ways, streamling complex interactions with better solution such as Text detection,Face detection,Product visual search,Voice detection,Image related etc ..!
Form recognition service can recognize the information from FORM it will return table content such as table count, rows, columns, cellcoordinate, textInfo, etc..!
Use case
This service will help you in daily basis, for example after collecting a large number of data we can use this service to recognize and convert the content into electronic documents.
Suggestions
1. Forms such as questionnaires can be recognized.
2. Currently images containing multiple forms cannot be recognized.
3. Shooting Angle: The horizontal tilt angle is less than 5 degrees.
4. Form Integrity: No missing corners and no bent or segment lines
5. Form Content: Only printed content can recognized, images, hand written content, seals and watermarks in the form cannot be recognized.
6. Image Specification: Image ratio should be less than or equal 3:1, resolution must be greater than 960 x 960 px.
ML Kit Configuration.
1. Login into AppGallery Connect, select MlKitSample in My Project list.
2. Enable Ml Kit, Choose My Projects > Project settings > Manage APIs
Development Process
Create Application in Android Studio.
App level gradle dependencies.
Code:
apply plugin: 'com.android.application'
apply plugin: 'com.huawei.agconnect'
Gradle dependencies
Code:
implementation 'com.huawei.hms:ml-computer-vision-formrecognition:2.0.4.300'
implementation 'com.huawei.hms:ml-computer-vision-formrecognition-model:2.0.4.300'
Root level gradle dependencies
Code:
maven {url 'https://developer.huawei.com/repo/'}
classpath 'com.huawei.agconnect:agcp:1.3.1.300'
Add the below permissions in Android Manifest file
Code:
<manifest xlmns:android...>
...
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<application ...
</manifest>
Add the following meta data in Manifest file its automatically install machine learning model.
Code:
<meta-data
android:name="com.huawei.hms.ml.DEPENDENCY"
android:value="fr" />
1. Create Instance for MLFormRecognitionAnalyzerSetting in onCreate.
Code:
MLFormRecognitionAnalyzerSetting setting = new MLFormRecognitionAnalyzerSetting.Factory().create();
2. Create instance for MLFormRecognitionAnalyzer in onCreate.
Code:
MLFormRecognitionAnalyzerFactory analyzer = MLFormRecognitionAnalyzerFactory.getInstance().getFormRecognitionAnalyzer(setting);
3. Check Runtime permissions.
4. FormRecogActivity is the responsible for load forms from local storage and capture the live forms using FormRecognigation service and we can extract the form cells content.
Code:
public class FormRecogActivity extends AppCompatActivity {
private MLFormRecognitionAnalyzerSetting setting;
private MLFormRecognitionAnalyzer analyzer;
private ImageView mImageView;
private TextView text, textTotal;
private MLFrame mlFrame;
private Uri imageUri;
private Bitmap bitmap;
private int camRequestCode = 100;
private int storageRequestCode = 200;
private int sum = 0;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_form_recog);
mImageView = findViewById(R.id.image);
text = findViewById(R.id.text);
textTotal = findViewById(R.id.text_total);
setting = new MLFormRecognitionAnalyzerSetting.Factory().create();
analyzer = MLFormRecognitionAnalyzerFactory.getInstance().getFormRecognitionAnalyzer(setting);
}
public void onLoadImage(View view) {
Intent intent = new Intent(Intent.ACTION_PICK, MediaStore.Images.Media.EXTERNAL_CONTENT_URI);
startActivityForResult(intent, storageRequestCode);
}
public void onClikCam(View view) {
if (checkSelfPermission(Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED) {
requestPermissions(new String[]{Manifest.permission.CAMERA}, camRequestCode);
} else {
Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
startActivityForResult(intent, camRequestCode);
}
}
@Override
public void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {
super.onRequestPermissionsResult(requestCode, permissions, grantResults);
if (requestCode == camRequestCode) {
if (grantResults[0] == PackageManager.PERMISSION_GRANTED) {
Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
startActivityForResult(intent, camRequestCode);
} else {
Toast.makeText(this, "Camera permission denied", Toast.LENGTH_LONG).show();
}
}
if (requestCode == storageRequestCode) {
if (grantResults[0] == PackageManager.PERMISSION_GRANTED) {
Intent intent = new Intent(
Intent.ACTION_PICK,
MediaStore.Images.Media.EXTERNAL_CONTENT_URI
);
startActivityForResult(intent, storageRequestCode);
} else {
Toast.makeText(this, "Storage permission denied", Toast.LENGTH_LONG).show();
}
}
}
@Override
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
super.onActivityResult(requestCode, resultCode, data);
if (resultCode == RESULT_OK && requestCode == storageRequestCode) {
imageUri = data.getData();
try {
bitmap = MediaStore.Images.Media.getBitmap(this.getContentResolver(), imageUri);
mImageView.setImageBitmap(bitmap);
callFormService();
} catch (IOException e) {
e.printStackTrace();
}
} else if (resultCode == RESULT_OK && requestCode == camRequestCode) {
bitmap = (Bitmap) data.getExtras().get("data");
mImageView.setImageBitmap(bitmap);
callFormService();
}
}
private void callFormService() {
mlFrame = MLFrame.fromBitmap(bitmap);
analyzer = MLFormRecognitionAnalyzerFactory.getInstance().getFormRecognitionAnalyzer();
Task<JsonObject> task = analyzer.asyncAnalyseFrame(mlFrame);
task.addOnSuccessListener(new OnSuccessListener<JsonObject>() {
@Override
public void onSuccess(JsonObject jsonObject) {
if (jsonObject != null && jsonObject.get("retCode").getAsInt() == MLFormRecognitionConstant.SUCCESS) {
Gson gson = new Gson();
String result = jsonObject.toString();
MLFormRecognitionTablesAttribute mlObject = gson.fromJson(result, MLFormRecognitionTablesAttribute.class);
ArrayList<MLFormRecognitionTablesAttribute.TablesContent.TableAttribute> tableAttributeArrayList = mlObject.getTablesContent().getTableAttributes();
ArrayList<MLFormRecognitionTablesAttribute.TablesContent.TableAttribute.TableCellAttribute> tableCellAttributes = tableAttributeArrayList.get(0).getTableCellAttributes();
for (MLFormRecognitionTablesAttribute.TablesContent.TableAttribute.TableCellAttribute attribute : tableCellAttributes) {
String info = attribute.textInfo;
text.setText(text.getText().toString() + "\n" + info);
}
Toast.makeText(FormRecogActivity.this, "Successfully Form Recognized", Toast.LENGTH_LONG).show();
Log.d("TAG", "result: " + result);
} else if (jsonObject != null && jsonObject.get("retCode").getAsInt() == MLFormRecognitionConstant.FAILED) {
Toast.makeText(FormRecogActivity.this, "Form Recognition Convertion Failed", Toast.LENGTH_LONG).show();
}
textTotal.setText("Total Cart Value : "+ sum +" Rs ");
}
}).addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(Exception e) {
Toast.makeText(FormRecogActivity.this, "Form Recognition API Failed", Toast.LENGTH_LONG).show();
}
});
}
}
5.This Xml class for creating the UI.
Code:
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:background="#2196F3"
android:orientation="vertical">
<ImageView
android:id="@+id/image"
android:layout_width="match_parent"
android:layout_height="250dp"
android:layout_marginTop="?actionBarSize" />
<Button
android:id="@+id/btn_storage"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:layout_margin="20dp"
android:onClick="onLoadImage"
android:background="#F44336"
android:textColor="@color/upsdk_white"
android:text="Load Image from storage" />
<Button
android:id="@+id/btn_capture"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:layout_marginLeft="20dp"
android:layout_marginRight="20dp"
android:onClick="onClikCam"
android:background="#F44336"
android:textColor="@color/upsdk_white"
android:text="Capture Image" />
<TextView
android:id="@+id/text_total"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:layout_marginLeft="20dp"
android:textSize="20sp"
android:textColor="@color/upsdk_white"
android:textStyle="bold" />
<ScrollView
android:layout_marginTop="8dp"
android:layout_width="match_parent"
android:layout_height="wrap_content">
<TextView
android:id="@+id/text"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:layout_marginLeft="20dp"
android:textSize="20sp"
android:textColor="@color/upsdk_white"
android:textStyle="bold" />
</ScrollView>
</LinearLayout>
Result
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Tips & Tricks
1. Enable ML service API in AppGallery Connect.
2. Capture single form only, form service not supporting multiple forms.
3. The resolution must greater than 960 x 960px.
4. Currently this service not supporting handwritten text information.
Conclusion
This article will help you to get textinfo from form, it will extract the individual cells data with coordinates. This service will help you in daily basis.
Thank you for reading and if you have enjoyed this article I would suggest you implement this and provide your experience.
Reference
ML Kit – Form Recognition
Refer the URL