This document introduces the Open API for dataset synchronization tasks. You will learn how users can use the API to view and update dataset synchronization task settings, submit dataset synchronization task backtracking, and terminate dataset synchronization task instances.
Caution: Before using the APIs described in this document, you must complete the prerequisites of integrating JWT-Token and applying for a Token. Please first review the invocation method. For details, see Invocation Method.

2.1 View dataset synchronization task settings #

• Permission requirements

  Resource	Permission
  Dataset	read

• API description: View the synchronization task settings of a dataset, including synchronization mode, performance parameters, and more.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncSettings

Request example

curl --location --request GET '{{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncSettings' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'

Request parameters

Response parameters

Response example

{
    "code": "aeolus/ok",
    "data": {
        "backtrackingConf": {
            "dateRange": {
                "endDate": "2022-12-13",
                "startDate": "2022-12-13"
            },
            "enable": 1
        },
        "doradoPriority": "normal",
        "frequency": "daily",
        "monitorConf": {
            "alarmRules": [
                {
                    "failedAlarmItems": [
                        {
                            "item": "retry_failed"
                        }
                    ],
                    "id": 6***,
                    "name": "auto_aeolus_rule_{{dataSetId}}_16***********_dev",
                    "normalNoticeConf": [
                        {
                            "dutySchedules": [],
                            "noticeChannel": "lark",
                            "users": [
                                "userEmailPrefix"
                            ]
                        }
                    ],
                    "resultAlarmItems": [],
                    "timeoutAlarmItems": []
                }
            ]
        },
        "paramsConfList": [
            {
                "name": "tqs.query.auto.retry.enable",
                "value": "true"
            }
        ],
        "performanceSettings": {
            "chQueryParams": {},
            "clusterName": "cnch_alpha_lq",
            "cnchVwDefault": "vw_default",
            "cnchVwWrite": "vw_write",
            "driverName": "click_house",
            "partitionKey": [
                "p_date"
            ],
            "primaryKeyList": [],
            "sampleKey": null,
            "sampleRate": 1.0,
            "shardKey": null
        },
        "retryInterval": 5,
        "retryNum": 1,
        "scheduleDay": "0",
        "scheduleTime": "00:00",
        "scheduleTimeType": 0,
        "slaDaytime": null,
        "slaStatus": 0,
        "syncType": 1,
        "ttl": 7,
        "upstreamSettings": {
            "dynamicPartitionConfList": [],
            "dynamicPartitionMode": false,
            "partitionConfList": [],
            "uniqueIndexList": []
        },
        "writePartition": 0,
        "yarnName": "root.default_hrtech_da_pm"
    },
    "msg": "成功"
}

2.2 Update dataset synchronization task settings #

• Permission requirements

  Resource	Permission
  Dataset	write

• API description: The content updated by this API is equivalent to modifying the synchronization configuration information—specifically, the synchronization frequency—on the dataset details page frontend.
  This API performs a full update. All configuration information must be provided in the request.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncSettings

Request example

curl --location --request PUT '{{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncSettings' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}' \
--data-raw '{
    "dataTableConf": {
        "ttl": 8
    },
    "syncConf": {
        "syncType": 1,
        "scheduleConf": {
            "frequency": "daily",
            "scheduleDay": "0",
            "scheduleTime": "00:00"
        }
    }
}'

Request parameters

Response example

{
    "code": "aeolus/ok",
    "data": "{{dataSetId}}",
    "msg": "成功"
}

2.3 Update dataset synchronization task settings-advanced settings #

• Permission requirements

  Resource	Permission
  Dataset	write

• API description: Update the advanced settings in the dataset synchronization settings, corresponding to the advanced configuration in the dataset frontend synchronization configuration information.
  This API performs a full update. All configuration information must be provided in the request.

Request description

Request URL: {{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncSettings/advanced

Request example

curl --location --request PUT '{{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncSettings/advanced' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}' \
--data-raw '{
    "yarnName": "root.default_hrtech_da_pm",
    "doradoPriority": "normal",
    "retryNum": 3,
    "retryInterval": 5
}'

Request parameters

Response example

{
    "code": "aeolus/ok",
    "data": null,
    "msg": "成功"
}

2.4 View dataset synchronization task status information #

• Permission requirements

  Resource	Permission
  Dataset	read

• API description: View the list of dataset synchronization tasks, including business date, synchronization status, synchronization time, and more.

Request description

• Request URL:

{{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncJob

Request example

curl --location --request GET '{{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncJob?startDate=2022-12-13&endDate=2022-12-15&nodeId' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'

Request parameters

Response parameters

{
    "code": "aeolus/ok",
    "data": {
        "instanceList": [
            {
                "bizTime": "2022-12-15 00:00:00",
                "bizTimePage": "2022-12-15",
                "diagPageUrl": "https://data.bytedanc*.net/dorado/instance-detail/diagnosis?project=cn_756&id=108301911&schedule=2022-12-15%2000%3A00%3A00&pid=756&status=2&instanceId=1324365220&groupName=cn",
                "instanceDurationTime": null,
                "instanceId": 1324365220,
                "instanceSign": "8332ea1af3a3d960**************",
                "logPageUrl": "https://data.bytedanc*.net/dorado/instance-detail/log?project=cn_7**&id=1********&schedule=2022-12-15%2000%3A00%3A00&pid=7**&instanceId=1324365220&groupName=cn",
                "scheduledStartTime": "2022-12-16 00:00:00",
                "syncEndTime": null,
                "syncStartTime": null,
                "syncStatus": 2,
                "syncStatusDescCode": 3,
                "tableSize": "-",
                "taskId": 9178
            },
            {
                "bizTime": "2022-12-14 00:00:00",
                "bizTimePage": "2022-12-14",
                "diagPageUrl": "https://data.bytedanc*.net/dorado/instance-detail/diagnosis?project=cn_756&id=108******&schedule=2022-12-14%2000%3A00%3A00&pid=756&status=4&instanceId=132*******&groupName=cn",
                "instanceDurationTime": 501.0,
                "instanceId": 1322531180,
                "instanceSign": "b1285**************",
                "logPageUrl": "https://data.bytedanc*.net/dorado/instance-detail/log?project=cn_756&id=108******&schedule=2022-12-14%2000%3A00%3A00&pid=756&instanceId=132*******&groupName=cn",
                "scheduledStartTime": "2022-12-15 00:00:00",
                "syncEndTime": "2022-12-15 00:33:29",
                "syncStartTime": "2022-12-15 00:25:08",
                "syncStatus": 4,
                "syncStatusDescCode": 30,
                "tableSize": "56",
                "taskId": 9***
            },
            {
                "bizTime": "2022-12-13 00:00:00",
                "bizTimePage": "2022-12-13",
                "diagPageUrl": "https://data.bytedanc*.net/dorado/instance-detail/diagnosis?project=cn_756&id=108301911&schedule=2022-12-13%2000%3A00%3A00&pid=7**&status=4&instanceId=132*******&groupName=cn",
                "instanceDurationTime": 1197.0,
                "instanceId": 1322222925,
                "instanceSign": "af3a3d960**************",
                "logPageUrl": "https://data.bytedanc*.net/dorado/instance-detail/log?project=cn_756&id=108*******&schedule=2022-12-13%2000%3A00%3A00&pid=7**&instanceId=132*******&groupName=cn",
                "scheduledStartTime": "2022-12-14 00:00:00",
                "syncEndTime": "2022-12-14 12:38:57",
                "syncStartTime": "2022-12-14 12:19:00",
                "syncStatus": 4,
                "syncStatusDescCode": 30,
                "tableSize": "56",
                "taskId": 9178
            }
        ],
        "total": 3
    },
    "msg": "成功"
}

• Remarks

  • Definition of example values for dataset task instance statuses.

    syncStatus values	Instance status
    1	Not ready
    2	Waiting for execution
    3	In progress
    4	Execution successful
    5	Execution failed
    6	Terminated

2.5 Submit dataset synchronization task backtracking #

• Permission requirements

  Resource	Permission
  Dataset	write

• Interface description and notes: Batch submit synchronization task instances to perform data backtracking.
  Caution: The selectable range for the backtracking date and time period must be less than or equal to the dataset's data lifecycle (ttl).

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncJob

Request example

curl --location --request POST '{{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/syncJob' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}' \
--data-raw '{
    "nodeId": "",
    "startDate": "2022-12-07",
    "endDate": "2022-12-11",
    "checkMinMax": true,
    "skipCheck": true,
    "isSpecifyQueue": true,
    "queueName": "root.oryx_aeolus_vip",
    "maxParallelism": 10
}'

Request parameters

Response example

{
    "code": "aeolus/ok",
    "data": "ok",
    "msg": "成功"
}

2.6 Terminate dataset synchronization task instances #

• Permission requirements

  Resource	Permission
  Dataset	write

• Interface description and notes: Batch terminate task instances. Available in version 2.60.0.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/stopSyncJob

Request example

curl --location --request POST '{{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/stopSyncJob' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}' \
--data-raw '{
    "instanceList": [
        {
            "id": 226*******, 
            "taskTime": "2023-12-25 00:00:00",
            "sign": ""
        },
        {
            "id": 226*******, 
            "taskTime": "2023-12-24 00:00:00"
        }
     ],
    "nodeId": ""
}'

Request parameters

Response example

{
    "code": "aeolus/ok",
    "data": "ok",
    "msg": "成功"
}

2.7 View details of dataset synchronization task instance #

• Permission requirements

• Interface description and notes: View detailed information of a task instance based on dataset ID and instance ID. Available in version 2.64.0

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/instanceInfo

Request example

curl --location --request POST '{{domain}}/aeolus/api/v4/open/dataset/{{dataSetId}}/instanceInfo' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}' \
--data-raw '{"instance_id_list": [94]}'

Request parameters

Response parameters

Response example

{
    "code": "aeolus/ok",
    "data": [
        "id": 94,
        "scheduleTaskId": 3,
        "subTaskId": 2,
        "taskId": 1,
        "appExternalId": "application_17***********_****",
        "createTime": "2024-02-23 23:00:00",
        "taskTime": "2024-02-23 20:00:00",
        "startTime": "2024-02-24 00:01:01",
        "endTime": "2024-02-24 00:02:01",
        "scheduleTime": "2024-02-24 00:00:00",
        "status": "SUCCEEDED",
        "triggerType": "BACKFILL"
    ]
    "msg": "成功"
}