Data output refers to the stage of outputting and saving data during the process of creating a visual modeling task. After completing data input and data processing, the task results need to be saved as a dataset to facilitate subsequent visual queries and presentation. There are three types of output operators: Output, Output Real-Time Label, and Split Output. This article will introduce the first type, the standard output operator.

2.1 Output method selection #

In the visual modeling editing interface, click the Add Operator button, and in the output type section, select the first "Output" option.

In the "Output Dataset" section at the bottom of the task interface, click to create a new dataset or write into an existing dataset for modeling tasks where each node has already been processed.

• To create a new dataset, you need to enter the dataset name and dataset description.

The output node in visual modeling supports field correction. When there is a compatibility issue between the field format of the upstream node and the target storage, you can click One-Click Correction. After clicking, a [Field Settings] operator will be added before the current node and the correction will be applied automatically.

• Data storage selection: The system supports two types of storage: Hive and ClickHouse. You can select as needed.
  • Hive: Partition selection is required.

• ClickHouse: You need to select the cluster, sorting key, sharding field, sampling field, lifecycle, and partition. (For detailed function explanations, click the question mark to view.)

2.2 Dependency configuration #

This product provides users with flexible task scheduling dependency configuration functionality. Scheduling dependencies usually refer to the upstream and downstream dependencies between periodic scheduling nodes during data synchronization and data development processes. By configuring the dependencies between nodes, tasks can be scheduled to run upstream and downstream nodes in order. That is, after the upstream node runs successfully, the downstream node will start, ensuring timely production of valid business data.
This product's dependency configuration provides two main methods: automatic detection of whether upstream data has been produced and custom dependency configuration. This section provides a detailed introduction to both methods. Users can establish dependencies on data sources, so that when a data source is updated or run, the current visual modeling task is triggered to execute.

• Configuration methods: There are two types—system automatic configuration and custom configuration.
• Note: When selecting the third-party API configuration method, you must first configure the API in the Task Management module under System Management, and then complete the dependency configuration for task scheduling in the dependency configuration modules of Visual Modeling and Datasets.

System automatic data source partition detection tasks #

• Usage restrictions: Only supports some partitioned data sources (Hive/MaxCompute/LAS) and partitioned tables. ***LAS in the SaaS environment currently only supports built-in data connections and does not support custom connections
• Special statement: To avoid inaccurate dependencies caused by incomplete data production during partition detection, please complete data production before building partitions.

(1) System automatic configuration
For data sources such as Hive, MaxCompute, and LAS, the system can automatically obtain and display the dependency information configured upstream.
Note: After the system wakes up the Sensor task at the scheduled time, it checks every 5 minutes whether the specified partition table of the data source has been generated. If the partition is generated, the current task will be executed immediately. If the partition is not produced for more than 50 hours, the Sensor task will be marked as failed.

Sensor definition: Sensor tasks mainly refer to scheduled tasks generated by the system to detect partitions, not the current task. Based on user settings, the system will automatically detect and probe whether the required partition table for the task exists in the user's data source. If it exists, it means that the preparation of upstream data is complete and downstream data can start running.

Based on the data source type of the dragged-in node table and the project space, if the system matches a dependency rule, this interface will automatically recommend a “system auto-configuration” dependency.

(2) Custom configuration
This product also allows users to modify dependency offsets and add dependencies through custom configuration. The specific steps are as follows:
Step 1: Click Custom configuration, then click “Add dependency” to open the configuration interface. In this interface, you can click Variable description to view prompts related to dynamic time, or enter a table name to search and view.

Step 2: Click “Create system custom dependency” to open the detailed configuration window. At this point, you need to fill in the content shown in the figure below:

Step 3: After clicking OK, the custom dependency configuration is successfully completed.

2.2.2 Third-party API dependencies #

If there is already a scheduling platform for data development or data processing tasks within the enterprise, and it can be configured according to this product’s API specifications, integration and linkage with this product can be achieved to establish task status dependencies for the scheduling platform.

• Special note: All data sources supported for extraction by this product also support building API dependencies.

(1) Create an API task dependency
In System Management - Task Management, create a new API task dependency. See Task Management.
(2) System auto-configuration
For data sources covered by API dependency rules created by users, the system can automatically obtain and display the upstream configured dependency information.
Note: After the system wakes up the Sensor task at the scheduled time, it checks every 5 minutes whether the specified partition table of the data source has been generated. If it has been generated, the current task will be executed immediately. If the partition has not been produced for more than 50 hours, the Sensor task will be marked as failed.
Based on the data source type of the dragged-in node table and the project space, if the system matches a dependency rule, this interface will automatically recommend a “system auto-configuration” dependency.

(3) Custom configuration
This product also allows users to modify dependency offsets and add dependencies through custom configuration. The specific steps are as follows:
Step 1: Click Custom Configuration, then click "Add Dependency" to open the configuration interface. In this interface, you can click Variable Description to view information related to dynamic time, or enter a table name to search for and view.

Step 2: Click "Create System Custom Dependency" to open the specific configuration window. At this point, you need to fill in the information shown in the following figure:

Step 3: After clicking OK, the custom dependency configuration has been successfully configured.

2.2.3 Other scenarios #

Scenario 1
If there is already a scheduling platform for data development or data processing tasks within the enterprise, users can also choose to manually run tasks based on the status of relevant tasks in their own scheduling platform.
Special note: Users need to specify the task ID, and if the original task is a manually executed task, users can trigger manual execution through OpenAPI.

• First, set the task update method to manual execution.

• Second, find the task ID and project ID (App ID). The task URL contains the task ID and project ID.

Project refers to the project space of this product.

• Finally, users can run the task by manually invoking the interface.

Scenario 2
If the user's data development or data task management scheduling platform cannot be configured in this product's task management settings through a standard API, it is necessary to develop and configure the interface through an intermediate layer. After that, this product can adapt to build task dependencies. For this scenario, we recommend that enterprises adopt the ISV-developed service model. For specific solutions, please consult BytePlus staff.

2.3 Advanced configuration #

For datasets from Hive data sources, you need to select the execution queue and the execution priority within the queue. Select a queue. The corresponding HIVE query will be executed on the specified queue before importing the dataset. The queue selection affects the duration of dataset synchronization, but does not impact visualization query results.
Runtime parameters: Supports setting runtime parameters for dataset synchronization as needed to ensure successful synchronization or optimize performance. For details, see: Extraction Parameters

2.4 Monitoring configuration #

Prerequisites: Notification methods require completing (集成相关,不发布)Office Platform Integration prerequisite configuration before they can be set.
Alarm conditions: Supports adding monitoring alarms to promptly notify relevant users when dataset synchronization encounters exceptions. Multiple alarm rules can be configured. The system provides three types of alarm conditions.

• Failure alarm: A message notification is triggered if dataset synchronization fails.
• Timeout alarm: Supports selecting three types of timeout definitions.

• Result anomaly alarm: Monitors the number of data rows and triggers an alarm when the row count is abnormal. Commonly used in the following scenarios:
  • To prevent synchronization exceptions or upstream data anomalies that result in no synchronized data: Set an alarm when the number of data rows equals 0.
  • Upstream data may be updated multiple times. To prevent incomplete synchronized data: Set an alarm when the number of data rows is less than the expected row count. For example, if the expected dataset synchronizes 100,000 rows daily and less than 80,000 rows may indicate incomplete data, set an alarm when the number of data rows is less than 80,000.
  • Percentage difference alarm: To monitor changes in the number of data rows in a timely manner, set the percentage difference compared to the previous day, the same period last week, the same period last month, or the average of the previous seven days. For example, if the percentage difference (compared to the previous day) is set to 10%, an alarm is triggered when the number of data rows changes by more than 10% from the previous day.

Notification methods: Supports email, Lark, WebEx, DingTalk, WeCom, email groups, Lark groups, and generic IM alarms.

• Generic IM is available in version V2.91.0 and above. Generic IM refers to the webhook method. When selecting this method, webhook must be configured in advance. For detailed instructions, see （暂不发布）Configure Subscription Push Method—Webhook.