Visual modeling is an interface-based, drag-and-drop data processing and modeling feature provided by this product. This capability is commonly referred to as ETL (Extract-Transform-Load) and can also be called data modeling. As an intermediate step between data sources and visual presentation, visual modeling enables users to efficiently process initial datasets through drag-and-drop, easy-to-use, and intelligent operations before performing visual queries and dashboard creation. After data is cleaned, transformed, and loaded, the resulting datasets are more suitable for business analysis. At the same time, this module also supports model training and algorithmic prediction scenarios.
*Visual modeling is not available in the Personal Edition and Standard Edition

Create a visual modeling task --> Select a data connection --> Add data operators --> Configure operator relationships --> Configure operators --> Execute operators --> Output nodes --> Set the storage method for the output node. For a complete learning course, see the documentationVolcano Engine Intelligent Platform (VeDI) Learning Center.

Visual modeling makes the data development process product-based, tool-based, and visual. Users only need to have basic data knowledge and do not require assistance from data developers. Complex data processing can be achieved in just a few drag-and-drop steps, making the process easy to use and highly efficient. These results are achieved based on the product's core features. The core functional modules of visual modeling include the following:
(1) Input & output datasets
Supports extracting data from data sources, adding data processing nodes via drag-and-drop, and outputting the processed data to the target source. Supports outputting to built-in data sources: datasets stored in Hive, ClickHouse, and ByteHouse. The overall process involves extracting data from data sources, adding data processing nodes via drag-and-drop, and outputting the processed data to the target source to form the output dataset.
(2) Data processing and cleaning
Data cleaning typically refers to the process of identifying and correcting recognizable errors in data files, including checking data consistency, handling invalid values, missing values, and more. Our visual modeling supports: field configuration, row filtering, adding calculated columns, aggregation, joining, merging, transposing rows and columns, and more.
(3)AI model construction
Visual modeling encapsulates the capabilities of more than 30 common AI operators. To complete model training, you only need to understand the purpose of the algorithm and configure the input and training objectives of the algorithm operator in a configurable manner. Based on other configured data content, you can quickly obtain prediction results.
These include three categories: feature engineering, machine learning, and natural language processing.
Taking feature engineering as an example, here is a brief introduction: Feature engineering typically refers to the process of transforming raw data into features that better express the essence of the problem, so that applying these features to prediction models can improve the model's prediction accuracy for unseen data. After completing data input, you can use feature engineering operators to further process and refine the input data.
(4)Task configuration and management
Viewing, operating, running management, status statistics, and management configuration of visual modeling tasks. In the visual modeling task overview section, users can view an overview of all visual modeling tasks within the project, including task status statistics and the task list. In the details section of a visual modeling task, you can view detailed configuration information for the task, including basic information, processing workflow, input and output table data preview, and operation records.

4.1 What you see is what you get: visually complete the data processing workflow #

To obtain the number of orders and order amounts by date and city granularity, and to retrieve city data for the daily Top 10 consumption amounts.

4.2 Rapid integration of multiple tables for easy multi-source data association and computation #

During data processing, multiple data sources often need to be combined. For example, there may be two large order data tables and a customer attribute information table. Profit is calculated based on the billing amount and cost amount, and then the Top 100 user orders with the highest profit contribution are selected.

4.3 No-code configuration of AI data mining capabilities #

Visual modeling encapsulates more than 30 types of common AI operator capabilities. Simply understand the purpose of each algorithm and configure its inputs and training targets to complete model training, and quickly obtain prediction results based on other configured data content.

• Feature engineering operators (13)

• Machine learning operators (22)

• Natural language processing operators (3)

• AI operator parameter configuration

• AI model training results

4.4 Customer intent mining and prediction #

For example, to mine customer intent based on existing high-intent customer samples:
(1) Drag in sample data and all data as data inputs
(2) Drag in a classification algorithm, such as the XGB algorithm, for model training
(3) Drag in a prediction operator to build the relationship between the model and all data for prediction
(4) Combine actual data and prediction results to output a dataset, thereby analyzing the intent distribution of all user data

4.5 Building a repurchase prediction model #

In e-commerce scenarios, it is necessary to build a user repurchase model based on existing data. During model construction, after data cleaning and format conversion, a gradient boosting tree is used to build the prediction model. At this point, the repurchase model workflow can be constructed through visual modeling as follows:

(1) Merge rows: Merge the output data tables of n operators (rectangles in the diagram) into a single master data table according to consistent headers. If there are no additions, deletions, or new attributes in the user sales data, no changes are needed here.
(2) Missing value replacement: When there are null values in attribute columns, this will affect subsequent model calculations. Use the missing value replacement operator to replace null values with a specified default value. If the user's sales data does not add, delete, or modify attributes, no changes are needed here.
(3) One-hot encoding: Attribute columns of text type cannot be directly used for model training and need to be one-hot encoded into numerical vectors, for example:
This content cannot be displayed outside of the Feishu document for now.
(4) Gradient boosting tree: Responsible for fitting the training data and outputting a model that can be used for prediction (parameters not labeled in the diagram do not need to be modified by maintenance personnel):

(5) Aggregate_1: Remove duplicates from the prediction data and select the maximum probability.
(6) Field extraction: Extract the necessary label and probability values for output.