BackgroundIt's now possible to carry a mobile recording studio in your pocket, thanks to a range of apps on the market that allow music enthusiasts to sing and record themselves anytime and anywhere.
However, you'll often find that nasty background noise creeps into recordings. That's where HMS Core Audio Editor Kit comes into the mix, which, when integrated into an app, will cancel out background noise. Let's see how to integrate it to develop a noise reduction function.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Development ProcessMaking PreparationsComplete these prerequisites.
Configuring the Project1. Set the app authentication information via an access token or API key.
Call setAccessToken during app initialization to set an access token. This needs setting only once.
Code:
HAEApplication.getInstance().setAccessToken("your access token");
Or, call setApiKey to set an API key during app initialization. This needs to be set only once.
Code:
HAEApplication.getInstance().setApiKey("your ApiKey");
2. Call the file API for the noise reduction capability. Before this, the callback for the file API must have been created.
Code:
private ChangeSoundCallback callBack = new ChangeSoundCallback() {
@Override
public void onSuccess(String outAudioPath) {
// Callback when the processing is successful.
}
@Override
public void onProgress(int progress) {
// Callback when the processing progress is received.
}
@Override
public void onFail(int errorCode) {
// Callback when the processing failed.
}
@Override
public void onCancel() {
// Callback when the processing is canceled.
}
};
3. Call applyAudioFile for noise reduction.
Code:
// Reduce noise.
HAENoiseReductionFile haeNoiseReductionFile = new HAENoiseReductionFile();
// API calling.
haeNoiseReductionFile.applyAudioFile(inAudioPath, outAudioDir, outAudioName, callBack);
// Cancel the noise reduction task.
haeNoiseReductionFile.cancel();
And the function is now created.
This function is ideal for audio/video editing, karaoke, live streaming, instant messaging, and for holding online conferences, as it helps mute steady state noise and loud sounds captured from one or two microphones, to make a person's voice sound crystal clear. How would you use this function? Share your ideas in the comments section.
ReferencesTypes of Noise
How to Implement Noise Reduction?
Related
Playback Process
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
1. Create a WisePlayer instance.
A WisePlayerFactory instance is returned when the WisePlayer initialization is successful. Your app will use this instance to call createWisePlayer() to create WisePlayer.
Code:
WisePlayer player = factory.createWisePlayer();
2. Set parameters such as the video URL.
a. Set listeners. Your app will react based on listener callbacks. For more information about the listeners, please refer to Interface Summary.
Code:
player.setErrorListener(errorListener);
player.setEventListener(eventListener);
player.setResolutionUpdatedListener(resolutionUpdatedListener);
player.setReadyListener(readyListener);
player.setLoadingListener(loadingListener);
player.setPlayEndListener(playEndListener);
player.setSeekEndListener(seekEndListener);
b. Set required parameters, such as playback type, bookmark, and repeat mode.
Code:
player.setVideoType(0);
player.setBookmark(10000);
player.setCycleMode(1);
c. Specify the address from which WisePlayer retrieves video content to play.
WisePlayer provides multiple APIs for setting the video URL, which your app can call as needed.
Code:
// Method 1: Set a single URL for a video.
player.setPlayUrl("http://192.168.0.100:8080/a.mp4");
// Method 2: Set multiple URLs (one primary, the other standby) for a video.
// player.setPlayUrl(new String[] {"http://192.168.1.100:8080/b.mp4", "http://192.168.1.101:8080/b.mp4"});
d. Set a view to display the video.
Currently, WisePlayer supports SurfaceView and TextureView. Make sure that a valid view is available; otherwise, the playback will fail.
Code:
player.setView(surfaceView);
3. Call ready to start loading data.
Code:
player.ready();
4. Call start in the onReady event notification to start playback.
After receiving the onReady event notification, your app calls the start method of WisePlayer to start playback. Then your app can use the getDuration() and getCurrentTime() APIs to obtain the video duration and playback progress for displaying the progress bar. WisePlayer parameters can be obtained only after the onReady event notification is received.
Code:
player.start();
a. Register ResolutionUpdatedListener, using which WisePlayer detects video resolution changes.
Code:
public void onResolutionUpdated(WisePlayer wp, int width, int height) {
// Your app can adjust the view size based on the width and height of the new resolution.
}
b. Drag the slider on the progress bar.
Code:
// Drag the slider to the "3600000 milliseconds" position.
player.seek(3600000);
c. Pause the playback and resume it.
To resume the playback, call the start method.
Code:
player.pause();
player.start();
d. Suspend WisePlayer.
If your app uses SurfaceView to display the video, it will suspend WisePlayer upon receiving a surfaceDestroyed event. If your app uses TextureView to display the video, it will suspend WisePlayer upon receiving an onSurfaceTextureDestroyed event.
Code:
@Override
public void surfaceDestroyed(SurfaceHolder holder) {
if (player != null) {
player.suspend();
}
}
e. Resume WisePlayer.
If your app uses SurfaceView to display the video, it will call the view set beforehand and resume WisePlayer upon receiving a surfaceCreated event. If your app uses TextureView to display the video, it will call the view set beforehand and resume WisePlayer upon receiving an onSurfaceTextureAvailable event.
Code:
@Override
public void surfaceCreated(SurfaceHolder holder) {
if (player != null) {
player.setView(surfaceView);
// To resume WisePlayer when you bring your app to the foreground, call the resume API. You can determine whether the playback automatically starts after your app is brought to the foreground by passing a parameter.
player.resume(-1);
}
}
f. Change the view size.
Your app must notify WisePlayer of the view size change. Otherwise, the playback is affected. If your app uses SurfaceView to display the video, the setSurfaceChange API will be called when a surfaceChanged event is received. If your app uses TextureView to display the video, the setSurfaceChange API will be called when an onSurfaceTextureSizeChanged event is received.
Code:
@Override
public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
if (player != null) {
player.setSurfaceChange();
}
}
5. Call release to stop playback and release the WisePlayer instance.
a. Stop the playback.
Code:
player.stop();
b.Release WisePlayer.
To stop playing, your app must call the following APIs:
Code:
// Set the WisePlayer listener to null.
player.setErrorListener(null);
player.setEventListener(null);
player.setResolutionUpdatedListener(null);
player.setReadyListener(null);
player.setLoadingListener(null);
player.setPlayEndListener(null);
player.setSeekEndListener(null);
// Release WisePlayer and set the WisePlayer object to null.
player.release();
player = null;
Simple and useful, thank you
Text to speech (TTS) is highly sought after by audio/video editors, thanks to its ability to automatically turn text into naturally sounding speech, as a low cost alternative to human dubbing. It can be used on all kinds of video, regardless of whether the video is long or short.
I recently stumbled upon the AI dubbing capability of HMS Core Audio Editor Kit, which does just that. It is able to turn input text into speech with just a tap, and comes loaded with a selection of smooth, naturally-sounding male and female timbres.
This is ideal for developing apps that involve e-books, creating audio content, and editing audio/video. Below describes how I integrated this capability.
Making PreparationsComplete all necessary preparations by following the official guide.
Configuring the Project1. Set the app authentication information
The information can be set via an API key or access token (recommended).
Use setAccessToken to set an access token during app initialization.
Java:
HAEApplication.getInstance().setAccessToken("your access token");
Or, use setApiKey to set an API key during app initialization. The API key needs to be set only once.
Java:
HAEApplication.getInstance().setApiKey("your ApiKey");
2. Initialize the runtime environment
Initialize HuaweiAudioEditor, and create a timeline and necessary lanes.
Java:
// Create a HuaweiAudioEditor instance.
HuaweiAudioEditor mEditor = HuaweiAudioEditor.create(mContext);
// Initialize the runtime environment of HuaweiAudioEditor.
mEditor.initEnvironment();
// Create a timeline.
HAETimeLine mTimeLine = mEditor.getTimeLine();
// Create a lane.
HAEAudioLane audioLane = mTimeLine.appendAudioLane();
Import audio.
Java:
// Add an audio asset to the end of the lane.
HAEAudioAsset audioAsset = audioLane.appendAudioAsset("/sdcard/download/test.mp3", mTimeLine.getCurrentTime());
3. Integrate AI dubbing.
Call HAEAiDubbingEngine to implement AI dubbing.
Java:
// Configure the AI dubbing engine.
HAEAiDubbingConfig haeAiDubbingConfig = new HAEAiDubbingConfig()
// Set the volume.
.setVolume(volumeVal)
// Set the speech speed.
.setSpeed(speedVal)
// Set the speaker.
.setType(defaultSpeakerType);
// Create a callback for an AI dubbing task.
HAEAiDubbingCallback callback = new HAEAiDubbingCallback() {
@Override
public void onError(String taskId, HAEAiDubbingError err) {
// Callback when an error occurs.
}
@Override
public void onWarn(String taskId, HAEAiDubbingWarn warn) {}
@Override
public void onRangeStart(String taskId, int start, int end) {}
@Override
public void onAudioAvailable(String taskId, HAEAiDubbingAudioInfo haeAiDubbingAudioFragment, int i, Pair<Integer, Integer> pair, Bundle bundle) {
// Start receiving and then saving the file.
}
@Override
public void onEvent(String taskId, int eventID, Bundle bundle) {
// Synthesis is complete.
if (eventID == HAEAiDubbingConstants.EVENT_SYNTHESIS_COMPLETE) {
// The AI dubbing task has been complete. That is, the synthesized audio data is completely processed.
}
}
@Override
public void onSpeakerUpdate(List<HAEAiDubbingSpeaker> speakerList, List<String> lanList,
List<String> lanDescList) { }
};
// AI dubbing engine.
HAEAiDubbingEngine mHAEAiDubbingEngine = new HAEAiDubbingEngine(haeAiDubbingConfig);
// Set the listener for the playback process of an AI dubbing task.
mHAEAiDubbingEngine.setAiDubbingCallback(callback);
// Convert text to speech and play the speech. In the method, text indicates the text to be converted to speech, and mode indicates the mode for playing the converted audio.
String taskId = mHAEAiDubbingEngine.speak(text, mode);
// Pause playback.
mHAEAiDubbingEngine.pause();
// Resume playback.
mHAEAiDubbingEngine.resume();
// Stop AI dubbing.
mHAEAiDubbingEngine.stop();
ResultIn the demo below, I successfully implement the AI dubbing function in app. Now, I can converts text into emotionally expressive speech, with default and custom timbres.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
To learn more, please visit:
>> Audio Editor Kit official website
>> Audio Editor Kit Development Guide
>> Reddit to join developer discussions
>> GitHub to download the sample code
>> Stack Overflow to solve integration problems
Follow our official account for the latest HMS Core-related news and updates.
It's so rewarding to set the model you've created into motion. If only there were an easy way to do this… well, actually there is!
I had long sought out this kind of solution, and then voila! I got my hands on motion capture, a capability from HMS Core 3D Modeling Kit, which comes with technologies like human body detection, model acceleration, and model compression, as well as a monocular human pose estimation algorithm from the deep learning perspective.
Crucially, this capability does NOT require advanced devices — a mobile phone with an RGB camera is good enough on its own. The camera captures 3D data from 24 key skeletal points on the body, which the capability uses to seamlessly animate a model.
What makes the motion capture capability even better is its straightforward integration process, which I'd like to share with you.
Application ScenariosMotion capture is ideal for 3D content creation for gaming, film & TV, and healthcare, among other similar fields. It can be used to animate characters and create videos for user generated content (UGC) games, animate virtual streamers in real time, and provide injury rehab, to cite just a few examples.
Integration ProcessPreparationsRefer to the official instructions to complete all necessary preparations.
Configuring the Project
Before developing the app, there are a few more things you'll need to do: Configure app information in AppGallery Connect; make sure that the Maven repository address of the 3D Modeling SDK has been configured in the project, and that the SDK has been integrated.
1. Create a motion capture engine
Java:
// Set necessary parameters as needed.
Modeling3dMotionCaptureEngineSetting setting = new Modeling3dMotionCaptureEngineSetting.Factory()
// Set the detection mode.
// Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON_QUATERNION: skeleton point quaternions of a human pose.
// Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON: skeleton point coordinates of a human pose.
.setAnalyzeType(Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON_QUATERNION
| Modeling3dMotionCaptureEngineSetting.TYPE_3DSKELETON)
.create();
Modeling3dMotionCaptureEngine engine = Modeling3dMotionCaptureEngineFactory.getInstance().getMotionCaptureEngine(setting);
Modeling3dFrame encapsulates video frame or static image data sourced from a camera, as well as related data processing logic.
Customize the logic for processing the input video frames, to convert them to the Modeling3dFrame object for detection. The video frame format can be NV21.
Use android.graphics.Bitmap to convert the input image to the Modeling3dFrame object for detection. The image format can be JPG, JPEG, or PNG.
Java:
// Create a Modeling3dFrame object using a bitmap.
Modeling3dFrame frame = Modeling3dFrame.fromBitmap(bitmap);
// Create a Modeling3dFrame object using a video frame.
Modeling3dFrame.Property property = new Modeling3dFrame.Property.Creator().setFormatType(ImageFormat.NV21)
// Set the frame width.
.setWidth(width)
// Set the frame height.
.setHeight(height)
// Set the video frame rotation angle.
.setQuadrant(quadant)
// Set the video frame number.
.setItemIdentity(framIndex)
.create();
Modeling3dFrame frame = Modeling3dFrame.fromByteBuffer(byteBuffer, property);
2. Call the asynchronous or synchronous API for motion detection.
Sample code for calling the asynchronous API asyncAnalyseFrame:
Java:
Task<List<Modeling3dMotionCaptureSkeleton>> task = engine.asyncAnalyseFrame(frame);
task.addOnSuccessListener(new OnSuccessListener<List<Modeling3dMotionCaptureSkeleton>>() {
@Override
public void onSuccess(List<Modeling3dMotionCaptureSkeleton> results) {
// Detection success.
}
}).addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(Exception e) {
// Detection failure.
}
});
Sample code for calling the synchronous API analyseFrame
Java:
SparseArray<Modeling3dMotionCaptureSkeleton> sparseArray = engine.analyseFrame(frame);
for (int i = 0; i < sparseArray.size(); i++) {
// Process the detection result.
};
3. Stop the motion capture engine to release detection resources, once the detection is complete
Java:
try {
if (engine != null) {
engine.stop();
}
} catch (IOException e) {
// Handle exceptions.
}
Result
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
To learn more, please visit:
>> 3D Modeling Kit official website
>> 3D Modeling Kit Development Guide
>> Reddit to join developer discussions
>> GitHub to download the sample code
>> Stack Overflow to solve integration problems
Follow our official account for the latest HMS Core-related news and updates.
Research has shown that our voice is often an indicator of our personalities, and this is why we're so fascinated with changing our voice to make it sound more fun and uplifting in, for example, videos and live streams.
As a mobile developer, I have implemented the voice changing function into my own app, which you can try out in my demo. This function allows users to mask their voice using seven preset voices: seasoned, cute, male, female, monster, cartoon, and robots.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
I don't want to brag, but I myself still find this function amazing. Let's move on to how it's developed.
Making PreparationsEnsure you have completed these steps first.
Configuring the ProjectSet the app authentication information.
This can be set via an API key or access token.
Call setAccessToken during app initialization to set an access token. This only needs to be set once.
Java:
HAEApplication.getInstance().setAccessToken("your access token");
Or, use setApiKey to set an API key during app initialization. This only needs to be set once.
Code:
HAEApplication.getInstance().setApiKey("your ApiKey");
Calling the File APICall the file API for the voice changer function, which is necessary for calling back the file API.
Java:
private ChangeSoundCallback callBack = new ChangeSoundCallback() {
@Override
public void onSuccess(String outAudioPath) {
// Callback when the processing is successful.
}
@Override
public void onProgress(int progress) {
// Callback when the processing progress is received.
}
@Override
public void onFail(int errorCode) {
// Callback when the processing fails.
}
@Override
public void onCancel() {
// Callback when the processing is canceled.
}
};
Implementing the Voice Changer CapabilityCall applyAudioFile to change the voice.
Java:
// Change the voice.
HAEChangeVoiceFile haeChangeVoiceFile = new HAEChangeVoiceFile();
ChangeVoiceOption changeVoiceOption = new ChangeVoiceOption();
changeVoiceOption.setSpeakerSex(ChangeVoiceOption.SpeakerSex.MALE);
changeVoiceOption.setVoiceType(ChangeVoiceOption.VoiceType.CUTE);
haeChangeVoiceFile.changeVoiceOption(changeVoiceOption);
// Call the API.
haeChangeVoiceFile.applyAudioFile(inAudioPath, outAudioDir, outAudioName, callBack);
// Cancel the task of changing the voice.
haeChangeVoiceFile.cancel();
And now it's implemented. Easy, right? I hope this article can help you develop your own voice changer, and feel free to leave a comment if you want to learn more about my development journey.
ReferencesVoice Changer
Efficient records management is more relevant now than ever. In our digital age, huge growth of information — audio, video, and more — must be handled in a limited time. This makes a real-time transcription function essential, because it is useful in many scenarios.
In audio or video conferencing, this function records meeting minutes that I can refer to later, which is more convenient than writing them all by myself. I've seen my kids struggling to take notes during their online courses, so I know this process can be so much easier with the help of the transcription function. In short, it removed the job of writing down everything the teacher says, allowing the kids to focus on the lecture itself and easily review the content again later. Also, the live captions provide viewers with real-time subtitles, for a better watching experience.
As a coder, I am a believer in "Actions speak louder than words". That's why I developed a real-time transcription function, with the help of a real-time transcription capability from ML Kit, like this.
Demo
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
This function transcribes up to 5 hours of speech into Chinese, English (or both), and French languages in real time. In addition, the output text is punctuated and contains timestamps.
This function has some requirements: the support for French is dependent on the mobile phone model, whereas Chinese and English are available on all mobile phone models. Also, the function requires Internet connection.
Okay, let's move on to the point of this article: How I developed this real-time transcription function.
Development Procedure1. Make necessary preparations. This is described in detail in the References section.
2. Create and then configure a speech recognizer.
Code:
MLSpeechRealTimeTranscriptionConfig config = new MLSpeechRealTimeTranscriptionConfig.Factory()
// Set the language, which can be Chinese, English, both Chinese and English, or French.
.setLanguage(MLSpeechRealTimeTranscriptionConstants.LAN_ZH_CN)
// Punctuate the text recognized from the speech.
.enablePunctuation(true)
// Set the sentence offset.
.enableSentenceTimeOffset(true)
// Set the word offset.
.enableWordTimeOffset(true)
.create();
MLSpeechRealTimeTranscription mSpeechRecognizer = MLSpeechRealTimeTranscription.getInstance();
3. Create a callback for the speech recognition result listener.
Code:
// Use the callback to implement the MLSpeechRealTimeTranscriptionListener API and methods in the API.
Protected class SpeechRecognitionListener implements MLSpeechRealTimeTranscriptionListener{
@Override
public void onStartListening() {
// The recorder starts to receive speech.
}
@Override
public void onStartingOfSpeech() {
// The speech recognizer detects the user speaking.
}
@Override
public void onVoiceDataReceived(byte[] data, float energy, Bundle bundle) {
// Return the original PCM stream and audio power to the user. The API does not run in the main thread, and the return result is processed in a sub-thread.
}
@Override
public void onRecognizingResults(Bundle partialResults) {
// Receive recognized text from MLSpeechRealTimeTranscription.
}
@Override
public void onError(int error, String errorMessage) {
// Callback when an error occurs during recognition.
}
@Override
public void onState(int state,Bundle params) {
// Notify the app of the recognizer status change.
}
}
4. Bind the speech recognizer.
Code:
mSpeechRecognizer.setRealTimeTranscriptionListener(new SpeechRecognitionListener());
5. Call startRecognizing to begin speech recognition.
Code:
mSpeechRecognizer.startRecognizing(config);
6. Stop recognition and release resources occupied by the recognizer when the recognition is complete.
Code:
if (mSpeechRecognizer!= null) {
mSpeechRecognizer.destroy();
}
ReferencesAudio Transcription: What It Is, What It Is Not, and Why It's in High Demand
Configuring Necessary Information During Preparation
Adding a Plug-In and the Maven Repository Address, and Configuring the Building Dependencies