Data operations and maintenance consists of five modules: task management, data connections, queue configuration, storage management, and synchronization configuration. This article introduces the specific operational procedures for each module in data operations and maintenance configuration.

The data operations and maintenance configuration module is provided in system management. Administrators can systematically manage various aspects, including task management, data connections, queue configuration, storage management, and synchronization configuration:

• Task management: In task management, administrators can establish dependencies on the execution status of third-party platform tasks by creating HTTP APIs, ensuring the timely generation of valid business data.
• Data connections: In data connections, administrators can configure different types of data connections for different projects. For example, in different projects, users may be allowed to select different types of data connections; administrators and non-administrators may also have access to different data connection types.
• Queue configuration: Configure different queues for scheduled tasks and data backtracking tasks to reduce the impact of data backtracking on routine tasks.
• Storage management: View resource overview information and configure storage policies, including enabling resource control switches, configuring project resource limits, and configuring actions for exceeding resource limits.
• Synchronization configuration: Set upper limits for lifecycle and backtracking time.

• Enterprises aim to use third-party APIs to configure dependencies and orchestrate the orderly scheduling of upstream and downstream task nodes based on these dependencies. That is, downstream nodes will only start running after upstream nodes have successfully completed.
• In some projects, project administrators are allowed to use all types of data connections, while certain data connection types are not available to other users.
• Managers within the company often need to review long-term data, while planning or operations personnel frequently compare data from different stages, requiring frequent retrospective analysis of datasets.
• The goal is to avoid blocking runtime resources and unnecessary overhead by adjusting the upper limits of the lifecycle and retrospection time.

Task management #

This product provides users with flexible configuration capabilities for task scheduling dependencies. Scheduling dependencies typically refer to the upstream and downstream dependencies between periodic scheduling nodes during data synchronization or data development processes. By orchestrating task execution for upstream and downstream nodes based on their dependencies, downstream nodes will only start running after upstream nodes have successfully completed, ensuring the timely production of valid business data. The dependency configuration of this product offers two main approaches: automatic detection of upstream data output and custom dependency configuration.
If a company already has a scheduling platform for data development or data processing tasks that can be configured according to this product's API specifications, it can be integrated and linked with this product to establish task status dependencies for that scheduling platform. If the company chooses to configure dependencies using a third-party API, users must first perform API configuration in the task management module under system management（that is, the content introduced in this section）, and then complete dependency configuration for task scheduling in the visual modeling and dataset dependency configuration modules.

The specific operation process is as follows:

1. Complete API configuration
   Click "New API Task Dependency" to enter the API configuration form. Depending on the interfaces of the data source, multiple API dependencies can be configured.
   1. Prerequisite operations
      
      Please complete the following required information:
      • Data source type: Specifies the data source for which the API dependency is effective. The dependency interface can be called when using the selected data source.
      • Dependency name: Used to understand the dependency name for system integration.
      • Scope: Controls the project space for API dependencies. You can select all projects or a single project from the dropdown list.
      • Dependency description: Descriptive information used to understand or fill in the dependency configuration.
   2. API configuration

Configuration 1: URL
  The Key and Value in Params will be automatically appended to the custom URL. The description section contains a sample URL.
  Configuration 2: Params
  * Request parameter description:
     The Key and Value in Params will be automatically appended to the custom URL. The Key name can be customized, and the Value parameter format remains unchanged. The system can automatically pass in specific task parameters based on the parameter content. You can use them as needed.
  * System parameter Params:
     
     | | | \
     |Parameters |Remarks |
     |---|---|
     | | | \
     |${dataSourceType} |Data source type, such as Hive-jdbc, MaxCompute |
     | | | \
     |${dbName} |The database name of the data source in the task |
     | | | \
     |${tableName} |The table name of the data source in the task |
     | | | \
     |${taskTime} |Business date, that is, the specific date when the task is executed. In most cases, the business date is consistent with the partition date. |
     | | | \
     |${partition} |The partition value of the data source in the task |
     | | | \
     |${batchNum} |The task batch number, which is defined by both the business date (taskTime) and execution frequency. For daily granularity, the batch number is always 1. For hourly granularity, 1 to 24 batch numbers are generated sequentially according to the execution order. |

  * Custom parameters
     **Note: When adding a custom parameter, please enter this parameter when configuring the dependency.**
     If the user's data scheduling platform cannot locate a specific instance based on the system fields provided by this product, you can choose to add a custom parameter field, such as defining a task ID. Please enter the specific task ID or name when configuring the dependency so that the dependency interface can locate the target parameter field.

3. Successful interface return structure
   When the API interface call is successful, please define the structure of a successful return: the JSON structure returns {"status":"success"} or {'status':0}.
4. Testing and saving

Test button
Note: Click Test. You can only save the API dependency after a successful connectivity test. This test button only tests the interface call and does not check the content. You can enter any content for testing.

• Clicking checks whether all configuration items are complete. If not, the form will be highlighted to prompt you to complete them.
• The test button is available when highlighted. Click to open the test dialog for testing.
• Different test dialogs and input methods are displayed based on the dependency method. The parameter format for the test date is fixed.
• After passing the test, a success icon is displayed. If information is changed, you need to click Test again.
  Save button
• If Test is not clicked or the test fails, the Save button is unavailable.
• Test passed: The Save button works, and after saving, the page returns to the view page.
  Cancel button
  Click Cancel, and a secondary confirmation dialog appears: After cancellation, configuration or modification information will not be saved. Confirm cancellation.

3. Subsequent operations
   After completing the API configuration for your own scheduling system, you can select the configured dependency interface in the dependency configuration of modules such as visual modeling and datasets. The specific steps are as follows:
   • When performing data output in visual modeling, you can configure dependencies. For details, see 2.2 Dependency configuration.
   • You can configure dependencies in the synchronization settings of the dataset. For details, see Dependency settings.
     

Data connection #

Type configuration #

To enhance enterprise data security and achieve rigorous and flexible data management, the product supports differentiated configuration for data connection types. For example, project administrators are allowed to use all data connection types, while some data connection types are not available to other users.
Steps:

1. In the left operation bar, select Data Connection and click Edit.

2. Select the scope of effect as All Projects or Distinguish by Project. If Distinguish by Project is selected, you can create a custom rule and select specific projects.
3. Select "Differentiated Rule" as "Yes", choose the rule type, and click the edit icon.

4. Select the data connection types to be displayed, click OK, and complete the configuration.

System configuration - Meego data connection configuration #

Meego data connections require unified configuration of plugin credentials. Only after successful verification does the current environment support the Meego data connection type. Before connecting to the Meego data source, please configure it in System Settings.

Steps:
Open Data Connection - System Configuration, click Edit, enter the Plugin ID and Plugin Password, and click Test Connection. After the test is successful, click Save to connect to the Meego data source.

How to obtain the plugin ID and secret
After enabling the Lark office platform in DataWind, the user needs to create and publish a plugin in Meego, then obtain the ID and secret on the plugin information page.

1. Enable the Lark office platform
   • Go to the System Configuration page and enable the Lark office platform.
   • The user needs to bind their Lark account.
   • Add a sending channel
2. Create a Meego plugin
   • Go to Meego, click Space Configuration -> Plugin Management -> Add Plugin, then on the page, click Create Plugin in the upper right corner.
   • Click Add Plugin on the page, enter the plugin name, and click OK.
   • After creation, go to the plugin configuration page and first fill in the basic configuration information.
   • Go to Permission Management and obtain the permissions listed below.

3. Publish the plugin
   • For the plugin usage scope, select internal enterprise use. For the installable space scope, select all spaces.
   • Return to Space Configuration -> Plugin Management and add the plugin you just created.
   • After successful publication, return to the plugin basic information page, copy the Plugin ID and Plugin Secret to obtain the plugin ID and secret.

System configuration - third-party data connections #

Third-party data connections must be enabled via a unified switch. After enabling, the current environment supports third-party data connection types such as WeChat Official Account, Ocean Engine Qianchuan, and Calendar. If the third-party data connection switch is not enabled in the system settings, users will not be able to detect system support for third-party data connections on any project's new data connection page. (Available starting from self-hosted deployment v2.74.0 and above)
Operation steps: Open Data Connection - System Configuration - Third-Party Data Connections, click "Enable" and save.

Queue configuration #

In System Management, click the System Configuration button, then select the Queue Configuration module in the left sidebar to manage queues. For a complete introduction, please refer to Queue Configuration.

Storage management #

When creating a new project, the system administrator can allocate specified storage resources to each project according to the settings. In the storage management interface, you can view resource overview information and configure storage policies, including the switch for enabling resource control, project resource limit configuration, and resource over-limit configuration. In addition, you can view and search usage details and more. The total amount of storage resources is generated or changed during deployment or upgrade. To make adjustments, contact system operations and maintenance.

Administrators can set the default upper limit for project resources. Administrators can set unified project resource limits for all projects in Storage Policy - Enable Resource Control, or set limits for a specific project in Usage Details. When project resources exceed the limit, creating new extraction tasks will be prohibited.

Synchronization configuration #

System administrators can set the upper limits for data lifecycle and rollback time in Synchronization Configuration to proactively constrain user behavior, reduce resource waste caused by human error, and avoid blocking of operating resources and unnecessary overhead.