For extracted datasets, the product allows users to configure synchronization frequency and time, dependency settings, synchronization monitoring, performance settings, and more. This article describes how to configure synchronization settings.

The product supports multiple forms of data connections, and allows you to set whether the data in the product and the original data source have an extraction or direct connection relationship. For extracted datasets, you can adjust synchronization frequency and time, dependency settings, synchronization monitoring, performance settings, and more.

The specific features are described as follows:

• Only extracted datasets have synchronization settings. The concept of synchronization does not apply to direct connection datasets. For the differences between extraction and direct connection, see Supported connection methods.
• Some configurations can only be set during dataset creation and cannot be modified after the dataset is created, such as dependency settings and performance settings.

In monitoring configuration, to set notification methods such as email or Feishu, you must first complete the prerequisite configuration for (集成相关,不发布)Office Platform Integration.

The trading calendar and multiple synchronization features in synchronization frequency are currently supported only in self-hosted deployments of version V2.83.0 and above. To enable this feature, contact the BytePlus team.

Synchronization frequency #

Supports configuration of synchronization type, synchronization frequency, write partition time, data lifecycle, and more.

The relevant parameter configuration instructions are as follows:

Dependency settings #

This product provides users with flexible task scheduling dependency configuration capabilities.
Scheduling dependencies usually refer to the upstream and downstream dependencies between periodic scheduling nodes during data synchronization and data development processes. Tasks are scheduled in an orderly manner according to node dependencies, meaning that downstream nodes are only triggered to run after upstream nodes have completed successfully, ensuring the timely output of valid business data.
The dependency configuration of this product provides two main types: automatic detection of whether upstream data has been produced and third-party API dependencies. For each type, the configuration method can be either automatic system configuration or custom configuration. Users can establish dependencies on data sources so that when a data source is updated or run, synchronization of the corresponding dataset is triggered.

The system automatically constructs partition detection tasks for data sources. #

• Usage limitation: Only certain partitioned data sources (Hive, MaxCompute, LAS) and partitioned tables are supported.
• Important: To avoid inaccurate dependencies caused by incomplete data output during partition detection, complete data output before constructing partitions.

Method 1: Automatic system configuration
For data sources in Hive, MaxCompute, and LAS, the system can automatically obtain and display the dependency information configured for upstream nodes.
Note: After the system wakes up the Sensor task at the scheduled time, it checks every 5 minutes whether the specified partitioned table of the data source has been generated. If the partition has been generated, the current task will be executed immediately. If the partition is not produced for more than 50 hours, the Sensor task will be marked as failed.

Definition of Sensor: Sensor tasks mainly refer to scheduled tasks generated by the system to detect partitions, not the current task. For user-configured scenarios, the system will automatically detect and check whether the required partitioned table exists in the user's data source. If it exists, this means the preparation of upstream data is complete and downstream data can start running.

The system will automatically discover Sensor tasks for dependencies that have already been created for the current table and recommend them for selection. If there are no recommendations, the list will be empty, and users can choose custom configuration.

Method 2: Custom configuration
This product also supports users in modifying dependency offsets and adding dependencies through custom configuration. The specific steps are as follows:

1. Click Custom Configuration, then click "Add Dependency" to open the configuration interface. In this interface, you can click Variable Description to view prompts related to dynamic time, or enter a table name to search and view.
   
2. Click "Create System Custom Dependency" to open the specific configuration window. At this point, you need to fill in the content shown in the following figure:
   
3. After clicking Confirm, the custom dependency configuration is successful.

Third-party API dependency #

If there is already a scheduling platform for relevant data development or data processing tasks within the enterprise, and it can be configured according to this product's API specifications, then integration and linkage with this product can be achieved to establish the task status dependency relationships for the scheduling platform.

• Special statement: All data sources that can be extracted by this product also support building API dependencies.

1. Create API task dependency
   Create an API task dependency in System Management - Task Management. For details, please refer to (SaaS有对应版本文档,不发布)Overview of System Management – SaaS Edition.

2. Dependency configuration
   Method 1: System automatic configuration
   For the data sources covered by the API dependency rules created by the user, the system can automatically obtain the upstream configured dependency information and display it.
   Note: After the system wakes up the Sensor task at the scheduled time, it checks every 5 minutes whether the specified partition table of the data source has been generated. If generated, the current task will be executed immediately. If the partition has not been produced for more than 50 hours, the Sensor task will be marked as failed.

   tip

   Sensor definition: At the user-defined time, the system automatically detects whether the required partitioned table for the task exists in the user's data source. If it exists, this means that the upstream data preparation is complete and the downstream data can start running.
   A Sensor task refers to a scheduled task generated by the system script to detect partitions, not the current task.

   The system will automatically detect Sensor tasks that have already been created as dependencies for the current table and recommend them for selection. If there are no recommendations, the list will be empty, and the user can choose custom configuration.
   
   Method 2: Custom configuration
   This product also supports users in modifying dependency offsets and adding dependencies through custom configuration. The specific steps are as follows:
   First, click Custom configuration, then click "Add dependency" to open the configuration interface. In this interface, you can click "Variable description" to view tips related to dynamic time, or enter a table name to search and view.
   
   Next, click "Create system custom dependency" to open the specific configuration window. At this point, you need to fill in the information shown in the figure below:

   

   As shown above, a custom dependency configuration has been set for the data table. The following are explanations of related terms:

   Term	Description	Example
   Data source type	The data source type of the dependency table, including Hive, MaxCompute, and LAS.	Hive indicates that a data source of the Hive data type was selected based on data requirements.
   Database name	The name of the database corresponding to the dependency table.	database_1 indicates that the database named database_1 in the Hive data source was selected.
   Table name	The name of the data table corresponding to the dependency table.	dim.user_basic_info_1d_a (user information table), indicating that the table dim.user_basic_info_1d_a under this database was selected (here, 'user information table' is used as an example).
   Table type	Divided into partitioned and non-partitioned types.	For example, for an incremental table, selecting non-partitioned means selecting all historical data in the data table. Selecting partitioned means selecting all data within the specified partition range.
   Partition settings	Partition settings are a method of dividing data into different subsets according to specific rules. The drop-down fields for partition settings include all fields in the dependency table you selected. You can select the partition field based on the field types in your table. After selecting the partition field, choose the corresponding partition.	Taking the above diagram as an example, the partition of the dependency table in this example is p_date (representing a time partition). According to business requirements, we set p_date='20250317' (the partition for March 17, 2025), which means viewing all data within this partition.
   Dependency type	Data source partition probe dependency, API dependency	By default, data source partition probe dependency is selected. If the system is configured with API dependency, you can choose either data source partition probe dependency or API dependency as needed.

   Finally, after clicking OK, the custom dependency configuration is successfully completed.

Other scenarios #

Scenario 1
If a scheduling platform for data development or data processing tasks already exists within the enterprise, users can also choose to manually run tasks based on the status of tasks in their own scheduling platform.
Special note: The user needs to specify the task ID, and if the original task is set to manual execution, the user can trigger manual execution through OpenAPI.

• First, set the task synchronization type to manual synchronization.
  
• Next, find the task ID and project ID (Appid). The task URL contains both the task ID and project ID (the project refers to the project space of this product).
  
• Finally, the user can trigger the task to run by manually calling the interface.

Scenario 2
If the user's data development or data task management scheduling platform cannot be configured in this product's task management via the standard API method, it is necessary to develop and configure the interface through an intermediate layer, and then have this product adapt to build task dependencies. For this scenario, we recommend that enterprises adopt the ISV development service approach. For specific solutions, please consult BytePlus staff.

Monitoring configuration #

Alarm conditions**:** Supports adding monitoring alarms to promptly notify relevant users when data set synchronization exceptions occur. Multiple alarm rules can be configured.
The system provides three types of alarm conditions

• Failure alarm: A notification will be triggered if dataset synchronization fails.
  
• Timeout alarm: Four types of timeout definitions are supported.
  
  • Absolute time (Beijing Time deadline): An alarm will be triggered if synchronization is not completed by the specified time.
  • Synchronization duration exceeded: An alarm will be triggered if dataset synchronization exceeds the specified duration. For example, if the alarm is set for a duration of 2 hours and synchronization starts at 7:00, an alarm will be triggered if synchronization is not completed by 9:00.
  • Exceeding the scheduled synchronization time: Applicable to scheduled synchronization. After setting the synchronization time in the synchronization frequency, if synchronization has not started or is not completed after the specified timeout, an alarm will be triggered. For example, if the synchronization time is set to 7:00 and the timeout is set to "not started within 1 hour after the scheduled synchronization time," an alarm will be triggered if the dataset has not started synchronization by 8:00 (in unsynchronized/not ready status).
  • Synchronization duration fluctuation: An alarm will be triggered if the synchronization duration of the dataset exceeds the specified duration.
• Result anomaly alarm: Monitors the number of data rows and triggers an alarm when the row count is abnormal.
  
  • To prevent synchronization anomalies or upstream data anomalies that result in no data being synchronized: An alarm can be set when the number of data rows equals 0.
  • Upstream data may be updated multiple times. To prevent incomplete synchronization: An alarm can be set when the number of data rows is less than the expected row count. For example, if the expected dataset synchronizes 100,000 rows daily and less than 80,000 rows are synchronized, it may indicate incomplete data. In this case, an alarm can be set when the number of data rows is less than 80,000.
  • Abnormal fluctuation rate alarm: The fluctuation rate can be set, and an alarm will be triggered when the number of data rows fluctuates beyond the set value compared to the previous day or within 7 days.
    

Notification methods: Supports email, Feishu, DingTalk, WeCom, email groups, Feishu groups, Wehook, and general IM.

Performance settings #

Performance settings may affect dataset synchronization and query performance. Performance settings cannot be modified after the dataset is saved.

Sort key:
Set the fields most commonly used as filter conditions as sort keys to make queries faster. Multiple fields can be set as sort keys. The first field has the greatest effect, with the influence of subsequent fields decreasing in order. It is recommended not to exceed three fields.
Shard key:
Used to ensure that data is evenly distributed across all storage nodes. If the daily data volume does not exceed 20 million rows, retain the system default settings. Only one field can be set as the shard key. You can select the field most commonly used as a dimension (set this if the field has more than 100 distinct values), or the field most commonly used as a deduplication count metric (set this if the field has more than 10,000 distinct values, such as user_id, device_id, item_id, and so on).

For usage scenarios of sort keys and partition fields, click the button above to view.

Sampling field:
During queries, random sampling can be performed based on the sampling field to improve query speed. This must be specified when creating the table. Only int, float, and string field types are supported.
The system defaults to the first primary key field. This is optional.
Sampling ratio:
By default, 100% of the data is sampled based on the sampling field. The smaller the ratio, the less data is sampled, and the faster the visualization query speed.
The system default is 100%. The ratio can be adjusted during actual visualization queries.
Partition:
Specifies the partition of the dataset. The default is p_date.
Query consistency guarantee:
When this feature is enabled, the results of multiple queries on the dataset are consistent, which can prevent extreme cases such as discrepancies between primary and secondary data. However, query performance will decrease accordingly.

Advanced settings #

Queue and priority:
This feature only supports extracted datasets. It affects extraction speed but does not affect query speed.
No changes are required for the default settings.
Retry count and retry interval for failures:
When dataset synchronization fails, the system will automatically initiate one retry by default. You can customize the number of retries and the retry interval for dataset synchronization failures to improve the success rate of synchronization.

Steps for configuring existing datasets:
On the dataset "Synchronization Status" page, click Synchronization Configuration.

Edit advanced settings.

Adjust the number of reruns and the rerun interval in the pop-up sidebar.

Synchronize multiple days:
To refresh the synchronized data for each day multiple times, configure synchronization for multiple days. Supports refreshing data for specified N days ago (multiple selections allowed) or the most recent N days. This feature needs to be used together with Custom SQL.
Runtime parameters:

Supports configuring runtime parameters for dataset synchronization as needed to ensure successful synchronization or optimize synchronization performance. For details, see Extraction Parameters.

When the dataset is in the saved preview state, click Dataset - Synchronization Status to view the historical synchronization status of the dataset below, as well as the historical data synchronization list, runtime view, logs, and synchronization operation buttons.

When the dataset is in the saved preview state, click Dataset -- Synchronization Status to view the historical synchronization details of the dataset below.

• The upper left area allows quick filtering of historical data synchronization results by business date, synchronization status, or when the result row count is 0.
  

• In the upper right are the "Synchronization Configuration" and "Data Synchronization" buttons.

  • Click the "Synchronization Configuration" button to display the synchronization configuration information. Click the pencil-shaped edit icon to the right of each feature to enter the settings for that feature.
    
  • Click the "Data Synchronization" button to open the synchronization settings pop-up. Select the required business date start and end times in the selection range to immediately synchronize the data within that business date range in bulk.
    
    Select "After enabling synchronization, the synchronization status list displays this time range" to show the execution information for this period in the list. As shown below, selecting the time period from 2-1 to 2-3 and enabling the toggle will display the following synchronization information in the synchronization status.
    
  • In the data backtracking settings, you can choose to execute according to dependency settings or ignore dependencies and execute backtracking directly.
    

• The list at the bottom displays all synchronized and updated data, with columns for the business date of data synchronization, synchronization status, scheduled time, synchronization start time, synchronization end time, execution duration, number of data rows, and operations (run view, logs, synchronize).

  • Run view: After clicking, a pop-up appears in the lower right corner, displaying the execution records of the dataset as a view.

    

  • Logs: After clicking, a new page opens, listing the execution logs for the dataset.
    

  • Synchronize: Clicking synchronize opens the backtracking settings. Click confirm to immediately synchronize the data for the specified business date range.