In the visual modeling module of Intelligent Data Insight, the output operator category supports real-time stream splitting output operators. With the stream splitting operator, users can directly process a single real-time stream into multiple output streams. During the splitting process, rule configurations are used, and real-time streams that meet the conditions are routed into the corresponding sub-streams for output. The stream splitting operator can be directly connected to data cleansing or output operators, or it can run independently. When running independently, it outputs to a built-in Topic for system consumption. A single real-time stream can be split into multiple streams based on event behavior or application appid. Only one Flink task is initiated during this process, making it an operator capability that efficiently utilizes computing resources.

2.1 Add stream splitting output operator #

In a visual modeling task, click the add operator button to see "Stream splitting output" in the output operator category, then click to select it. The following functional notes apply:
(1) Input restrictions: read nodes and data cleansing nodes.
(2) Output restrictions: as a final operator, downstream operators cannot be connected.

2.2 Configure stream splitting output operator #

With the stream splitting operator, a single real-time stream can be directly processed into multiple output streams. During the splitting process, rule configurations are used, and real-time streams that meet the conditions are routed into the corresponding sub-streams for output. The output is provided as datasets, and you can set the lifecycle of the datasets as needed to avoid duplicate data storage.
After selecting and adding the stream splitting output operator, you can configure each stream split in detail, and add or delete stream split configurations. Caution: There is a limit on the number of stream splits—currently, up to 10 splits are supported. The specific configuration is as follows:
(1) Create a new output dataset. You can also edit dimension metrics.
(2) Configure filter criteria: Build AND/OR relationships using field conditions to output data that meets the criteria. Click "AND" or "OR" in the logic tree to switch between AND/OR relationships.
(3) You can click Add branch to continue creating and configuring.
(4) Click the delete button in the upper right corner to delete.
Notes
(1) Branch configuration restrictions: Once added, it cannot be left empty. If it is left empty, you will be prompted with "There are empty configuration items" when saving.
(2) Output dataset:
a. Naming rules: Only English letters, numbers, and underscores are allowed. Spaces and other special characters are not permitted. The maximum length is 50 characters, and names must be unique.
b. Add toggle option (or reuse offline toggle): Supports editing kafka topic/CK database name and table name. After enabling, you can customize the database name, table name, and topic name.
c. When disabled: It is recommended to add a prefix to topics generated by branching to facilitate their identification.