This document introduces the "Open API for project datasets", which mainly includes the project dataset tag and folder API, project dataset API, and project cluster configuration API. You will learn that users can use the API to retrieve, create, and update dataset information and folder information under a project, as well as retrieve, add, and delete users in the private cluster write user list.
Caution: Before using the APIs described in this document, you must complete the prerequisites of integrating JWT-Token and applying for a Token. Please first review the invocation method. For details, see Invocation Method.

2.1 Retrieve dataset tag information under a project #

• Permission requirements

  Resource	Permission
  Project	read

• API description: Retrieve information about dataset tags already created under a project.
  

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetTag

Request example

curl --location --request GET '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetTag' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'

Request parameters

Response parameters

Response example

{
    "code": "aeolus/ok",
    "data": [
        {
            "appId": 7******,
            "colour": "purple",
            "ctime": "2022-12-13 16:48",
            "dataSetInfos": [
                {
                    "id": 19****,
                    "name": "v4专用测试数据集"
                },
                {
                    "id": {{dataSetId}},
                    "name": "测试open-api专用数据集1"
                }
            ],
            "id": 153,
            "mtime": "2022-12-13 16:48",
            "name": "test用",
            "reportEnable": 0,
            "status": 0
        }
    ],
    "msg": "成功"
}

2.2 Create a dataset tag under a project #

• Permission requirements

  Resource	Permission
  Project	admin

• API description: Create a dataset tag under a project.
  

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetTag

Request example

curl --location --request POST '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetTag' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}' \
--data-raw '{
    "name": "test用",
    "colour": "purple",
    "reportEnable": 0
}'

Request parameters

Response parameters

• Response description: $.data.id is the ID of the newly created dataset tag

  {
      "code": "aeolus/ok",
      "data": {
          "id": 154
      },
      "msg": "成功"
  }
  

2.3 Update dataset tag information under a project #

• Permission requirements

  Resource	Permission

  

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetTag/{{tagId}}

Request example

curl --location --request PUT '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetTag/{{tagId}}' \
--header 'Content-Type: application/json' \
--header 'app-id: 77*****' \
--header 'Authorization: Bearer {{jwtToken}}' \
--data-raw '{
    "colour": "purple",
    "name": "12",
    "reportEnable": 0
}'

Request parameters

Response example

{
    "code": "aeolus/ok",
    "data": null,
    "msg": "成功"
}

2.4 Delete a dataset tag under a project #

• Permission requirements

  Resource	Permission
  Project	admin

• API description: Delete dataset tag information under a project.
  

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetTag/{{tagId}}

Request example

curl --location --request DELETE '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetTag/{{tagId}}' \
--header 'Content-Type: application/json' \
--header 'app-id: 7777947' \
--header 'Authorization: Bearer {{jwtToken}}'

Request parameters

Response parameters

{
    "code": "aeolus/ok",
    "data": null,
    "msg": "成功"
}

3.1 Retrieve dataset folder information under a project #

• Permission requirements

  Resource	Permission
  Project	read

• API description: Delete dataset tag information under a project.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetFolder

Request example

curl --location --request GET '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetFolder' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'

Request parameters

Response parameters

Response example

{
    "code": "aeolus/ok",
    "data": [
        {
            "appId": *******,
            "ctime": "2022-04-02 11:52",
            "deleteTime": null,
            "descr": "",
            "id": 2320,
            "mtime": "2022-04-02 11:52",
            "name": "test1",
            "ownerEmailPrefix": "userEmailPrefix",
            "parentId": 0,
            "status": 0
        },
        {
            "appId": *******,
            "childList": [
                {
                    "appId": *******,
                    "childList": [
                        {
                            "appId": *******,
                            "childList": [
                                {
                                    "appId":*******,
                                    "ctime": "2022-12-14 12:14",
                                    "deleteTime": null,
                                    "descr": "",
                                    "id": 3075,
                                    "mtime": "2022-12-14 12:14",
                                    "name": "子文件夹-层级3",
                                    "ownerEmailPrefix": "userEmailPrefix",
                                    "parentId": 2706,
                                    "status": 0
                                }
                            ],
                            "ctime": "2022-09-29 16:30",
                            "deleteTime": null,
                            "descr": "",
                            "id": 2706,
                            "mtime": "2022-12-14 12:13",
                            "name": "子文件夹-层级2",
                            "ownerEmailPrefix": "userEmailPrefix",
                            "parentId": 2704,
                            "status": 0
                        }
                    ],
                    "ctime": "2022-09-29 16:29",
                    "deleteTime": null,
                    "descr": "111",
                    "id": 2704,
                    "mtime": "2022-12-14 12:13",
                    "name": "子文件夹-层级1",
                    "ownerEmailPrefix": "userEmailPrefix",
                    "parentId": 2321,
                    "status": 0
                }
            ],
            "ctime": "2022-04-02 11:53",
            "deleteTime": null,
            "descr": "",
            "id": 2321,
            "mtime": "2022-05-04 11:24",
            "name": "模板数据集",
            "ownerEmailPrefix": "userEmailPrefix",
            "parentId": 0,
            "status": 0
        },
        {
            "appId": *******,
            "ctime": "2022-12-14 12:14",
            "deleteTime": null,
            "descr": "",
            "id": 3076,
            "mtime": "2022-12-14 12:14",
            "name": "测试文件夹1",
            "ownerEmailPrefix": "userEmailPrefix",
            "parentId": 0,
            "status": 0
        }
    ],
    "msg": "成功"
}

3.2 Create a dataset folder under a project #

• Permission requirements

  Resource	Permission
  Parent folder	write
  Project (when parentId=0)	write

  To create a folder in the root directory, project write permission is required.

• API description: Create a folder in the root directory or under a specific folder.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetFolder

Request example

curl --location --request POST '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetFolder' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}' \
--data-raw '{
    "name": "子文件夹1111",
    "descr": "666",
    "confidentiality": "L3",
    "isPersonalDataRelated": true,
    "dataRegion": [
        "CN"，
        "US"
    ],
    "parentId": 0
}'

Request parameters

Response example

• Response description: folderId is the ID of the new folder.

  {
      "code": "aeolus/ok",
      "data": {
          "folderId": 3079
      },
      "msg": "成功"
  }
  

3.3 Update dataset folder information in the project #

• Permission requirements

  Resource	Permission
  Dataset folder	write

• API description and notes

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetFolder/{{dataSetFolderId}}

Request example

curl --location --request PUT '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetFolder/{{dataSetFolderId}}' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}' \
--data-raw '{
    "name": "测试文件夹1",
    "descr": "111",
    "isPersonalDataRelated": true,
    "dataRegion": [
        "CN"，
        "US"
    ],
    "confidentiality": "L2"
}'

Request parameters

Response example

• Response description: Returning folderId indicates that the folder information was updated successfully.

  {
      "code": "aeolus/ok",
      "data": {
          "folderId": 3079
      },
      "msg": "成功"
  }
  

4.1 Retrieve the list of datasets in the project #

• Permission requirements

  Resource	Permission
  Dataset folder	read

• API description and notes: Retrieve the list of information for all current datasets in the project.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetList

Request example

curl --location --request GET '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetList?needSyncInfo=0&needUpstreamInfo=0&dataSetIdList=186583,186600' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'

Request parameters

Response parameters

{
    "code": "aeolus/ok",
    "data": [
        {
            "ctime": "2022-12-14 12:17:54",
            "elapsedTimeForLastSuccessTaskTime": 501,
            "frequency": "daily",
            "hivePartitionSpecified": false,
            "hiveTableUsed": false,
            "hsqlList": [],
            "hsqlUsed": false,
            "id": {{dataSetId}},
            "lastSuccessTaskTime": "2022-12-14 00:00:00",
            "lastSyncTime": "2022-12-15 18:19:19",
            "mtime": "2022-12-15 19:37:31",
            "name": "测试open-api专用数据集1",
            "owner": "userEmailPrefix",
            "rowsForLastSuccessTaskTime": 56,
            "scheduleDay": "0",
            "scheduleTime": "00:00",
            "status": 0,
            "syncType": 2,
            "ttl": 8,
            "type": {
                "code": 22,
                "name": "prep 数据集"
            },
            "upstreamInfo": [
                {
                    "dbName": "aeolus_excel_upload_aeolus_toy",
                    "tableName": "table_4f0cc551_9dfa_4915_a38f_737c716591a8"
                }
            ],
            "url": "https://aeolus-release.fedev.bytedanc*.net/aeolus#/dataManage/detail/{{dataSetId}}?appId=7777947"
        }
    ],
    "msg": "成功"
}

4.2 Retrieve summary information for datasets in the project #

• Minimum supported version 2.58.1.

• Permission requirements

  Resource	Permission
  Dataset	read

• API description and notes: Retrieve the list of information for all current datasets in the project.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetSimpleList

Request example

curl --location --request GET '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSetSimpleList?page=1&perPage=2&kw=测试' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'

Request parameters

Response parameters

Response example

{
    "code": "aeolus/ok",
    "data": {
        "dataSetList": [
            {
                "appId": 1,
                "belong": 1,
                "ctime": "2023-10-16 19:51",
                "id": 1,
                "mtime": "2023-10-16 19:51",
                "name": "data_set_name",
                "ownerEmailPrefix": "user",
                "status": 0,
                "type": 22
            }
        ],
        "totalSize": 1
    },
    "msg": "成功"
}

4.3 Retrieve dataset information in the project where the upstream is a specified data source #

• First supported version 2.51.

• Permission requirements

  Resource	Permission
  Project	read

• API description and notes: By specifying dataSourceType (data source type), dbName (database name of the data source, not applicable for some data sources), and tableName (table name of the data source), you can query the basic information of all datasets that use this data source as their upstream.
  Note: The current query scope does not include datasets of custom SQL type.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSourceLineage

Request example

curl --location --request POST '{{domain}}/aeolus/api/v4/open/app/{{appId}}/dataSourceLineage' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'
--data '{
    "dataSourceType": "click_house",
    "dbName": "aeolus_excel_upload_aeolus_toy",
    "tableName": "table_35b15b69_2ea5_****_****_*********",
    "authOnly": false
}'

• Request parameters

  Parameter	Type	Required	Sample value	Description
  appId	integer	Yes	123	Project ID
  dataSourceType	string	Yes	hive click_house	Data source type
  dbName	string	No	aeolus_excel_upload_aeolus_toy	Data source - database name
  tableName	string	No	table_35b15b69_2ea5___*********	Data source - table name
  authOnly	boolean	No	false	Whether to return only the dataset information for the current user (with read-only or higher permission). Default value is false.

Response parameters

• Response description: For the response fields under dataSetList, see the detailed description.
  [API - View basic dataset information]

  Parameter	Type	Description
  dataSetList	list	Result content
  dataSourceType	string	dataSourceType used for the query
  dbName	string	dbName used for the query
  tableName	string	tableName used for the query
  listSize	integer	Number of dataset records found

  Response example

  {
      "code": "aeolus/ok",
      "data": {
          "dataSetList": [
              {
                  "appId": "*******",
                  "appName": "hsm",
                  "demoUrl": "",
                  "descr": null,
                  "detailUrl": "https://aeolus-integration.bytedanc*.net/aeolus#/dataManage/detail/193016?appId=7778352",
                  "folderName": [
                      "根目录"
                  ],
                  "id": ******,
                  "name": "good_hive_data_set",
                  "ownerEmailPrefix": "husimin.xcl",
                  "status": 0,
                  "type": 2
              },
              {
                  "appId": "*******",
                  "appName": "hsm",
                  "demoUrl": "",
                  "descr": null,
                  "detailUrl": "https://aeolus-integration.bytedanc*.net/aeolus#/dataManage/detail/196003?appId=7778352",
                  "folderName": [
                      "原始内容保留",
                      "根目录"
                  ],
                  "id": 196003,
                  "name": "测试tableSchema权限",
                  "ownerEmailPrefix": "husimin.xcl",
                  "status": 0,
                  "type": 2
              }
          ],
          "dataSourceType": "hive",
          "dbName": null,
          "listSize": 2,
          "tableName": "hsm_own_metrics"
      },
      "msg": "成功"
  }
  

5.1 Retrieve the list of users with write access for a private cluster #

• Permission requirements: The user must be the owner or manager of the private cluster, and a project administrator.
• API description and notes: Retrieve the list of eligible users for a cluster in the project.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/clusterWriteUserList

Request example

curl --location --request GET '{{domain}}/aeolus/api/v4/open/app/{{appId}}/clusterWriteUserList?clusterId=1' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'

Request parameters

Response parameters

Response example

{
    "code": "aeolus/ok",
    "data": [
        "luowenlei",
        "chenweichen.six"
    ],
    "msg": "成功"
}

5.2 Add users with write access to a private cluster #

• Permission requirements: The user must be the owner or person in charge of the private cluster, and must also be a project administrator.
• Interface description and notes: Add certain users to the admission user list of a specific cluster under the project.
• Note: If the cluster's permissions are already available to the entire project, this operation will not change the current state of project-wide availability. The permissions for the added users will exist redundantly and will take effect only after the project-wide permissions are removed. If an error occurs during batch writing, you can retry the write operation. The interface returns idempotent results upon success.

Request description

Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/clusterWriteUserList

Request example

curl --location --request POST '{{domain}}/aeolus/api/v4/open/app/{{appId}}/clusterWriteUserList' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'\
--data-raw '{
    "userList": ["******.****"],
    "clusterId": 1
}'

Request parameters

Response example

• $.data.id is the cluster ID.

{
    "code": "aeolus/ok",
    "data": {
        "id": 1
    },
    "msg": "成功"
}

5.3 Delete users from the private cluster write access list #

• Permission requirements: The user must be the owner or person in charge of the private cluster, and must also be a project administrator.
• Interface description and notes: Remove certain users from the admission user list of a specific cluster under the project.
• Note: If an error occurs during batch deletion, you can retry the write operation. The interface does not guarantee idempotent results upon success.

Request description

• Request URL: {{domain}}/aeolus/api/v4/open/app/{{appId}}/clusterWriteUserList

Request example

curl --location --request DELETE '{{domain}}/aeolus/api/v4/open/app/{{appId}}/clusterWriteUserList' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{jwtToken}}'\
--data-raw '{
    "userList": ["*******.****"],
    "clusterId": 1
}'

Request parameters

Response example

• $.data.id is the cluster ID.

{
    "code": "aeolus/ok",
    "data": {
        "id": 1
    },
    "msg": "成功"
}