Principles Behind HUAWEI Prediction How We Trained Models for the Service - Huawei Developers

HUAWEI Prediction utilizes machine learning, based on user behavior and attributes reported by HUAWEI Analytics Kit, to predict target audiences with next-level precision. The service can help you with carrying out and optimizing operations. For example, it can work with A/B Testing to evaluate how effective your promotions have been and it can also join hands with Remote Configuration to configure dedicated plans for specific audiences through Remote Configuration. This is likely to result in dramatically improved user retention and conversion.
{
"lightbox_close": "Close",
"lightbox_next": "Next",
"lightbox_previous": "Previous",
"lightbox_error": "The requested content cannot be loaded. Please try again later.",
"lightbox_start_slideshow": "Start slideshow",
"lightbox_stop_slideshow": "Stop slideshow",
"lightbox_full_screen": "Full screen",
"lightbox_thumbnails": "Thumbnails",
"lightbox_download": "Download",
"lightbox_share": "Share",
"lightbox_zoom": "Zoom",
"lightbox_new_window": "New window",
"lightbox_toggle_sidebar": "Toggle sidebar"
}
Integrating the Analytics SDK into your app enables the Prediction service to run preset tasks for predicting lost, paying, and returning users. On the details page of a specific prediction task, you'll find audiences with high, medium, and low probabilities of triggering a specific event, with meticulous profiling. For example, an audience with a high churn probability will include users who are very likely to quit using the app over the next 7 days. The characteristics of these users are displayed on cards, which makes it easy for you to pursue targeted operations.
The following figures give you a sense of how the prediction task list and details page look in practice.
* Data in these figures is for reference only.
Ø How we built these prediction models
First of all, we made it clear what our goal was to make predictions, so the type of data we collect reflects this. We then cleansed and sampled the collected data based on user characteristics to obtain a data set. This data set was divided into a 20% validation set and an 80% training set; multiple rounds of offline experiments were then conducted to determine the features and most suitable parameters for forming models. The generated models were later trained online to perform prediction tasks.
This process is outlined in detail below:
Ø Feature and model selection and optimization
Feature exploration
At the early stage of the project, we made sure to analyze user attributes, behavior, and requirements, in order to determine the business-relevant variables, such as user active days over the last 7 days and app use durations, through which we built a feature table.
After the features were identified, we chose a method that best suited our service and optimized parameters by performing multiple rounds of experiments. Common tree boosting methods that can be found across the industry include XGBoost, random forests, and Gradient Boost Decision Tree (GBDT). We trained our data set using these methods, and found that random forests perform best. Then the bagging method was adopted to improve models' fitting and generalization capabilities.
In addition to parameter optimization, the sampling ratio was also considered, especially for the payment prediction scenario, in which the proportion between positive samples and negative samples was large (about 1:100). For such cases, the accuracy and recall indicators should both be ensured. Then we adjusted the ratio of positive samples to negative samples to 1.5:1 during model training for payment prediction, in order to boost the recall of the model.
Hyperparameter and feature determination
Unnecessary features in a model can undermine the efficacy of predictions made by the model, or slow down model training. During experiments at this early stage, features were sorted by weight, and the top features were selected. In the model that would actually come to be, these features and relevant hyperparameters were configured.
Even after a model is applied for prediction, the data still needs to be observed and analyzed to supplement necessary features. In later iterations, we added a range of features, including the event and trend features, bringing the feature count over 400.
Automatic hyperparameter search
Model training involving full features can be quite time-consuming, and fail to produce the optimal output. In addition, the optimal hyperparameters and features may vary depending on the app. Therefore, the training should be performed by app.
To address this issue, we applied the automatic hyperparameter search function to search for optimal parameters in the configured parameter space. Matched parameters are stored in a Hive table.
The following figures show the modeling procedure and relevant external support.
Ø Research emphasis
We will continue optimizing our models, by researching the following:
l Neural network
As the number of features continues to grow (400+ currently), and user behaviors become too complex to mine common rules, our prediction models will need to be enhanced to ensure that predictions remain accurate. This will require that we introduce neural networks with strong expressive power, in addition to decision trees to train models based on behavioral features.
l Federated learning
Currently, data is isolated between apps and tenants. Horizontal federated learning can be used to train models across apps and tenants on a collaborative basis.
l Time series feature
A typical app user's device will report hundreds of events (among 1,000+ event types) and access nearly 100 pages within the app on a weekly basis. These times series can be used to build both short- and long-term user behavioral features, with the goal of improving prediction accuracy across a wide range of scenarios. Page access user behavioral data can be valuable for research, as such data bear characteristics of time series data.
l Feature mining and processing
The feature set is still being expanded. We will explore additional relevant features, such as the average app use interval, device attributes, download sources, and locations. In addition, we will also undertake such measures as discretization, normalization, square and square root operations, Cartesian product calculation, and Cartesian product calculation for multiple data sets, to build subsequent features that are based on existing features.
For more on HUAWEI Prediction, visit>>
For more details, you can go to:
l Our official website
l Our Development Documentation page, to find the documents you need
l Reddit to join our developer discussion
l GitHub to download demos and sample codes
l Stack Overflow to solve any integration problems