Data connections support integration with Tencent Cloud COS.

This section describes two methods for creating a Tencent Cloud COS data connection.

2.1 Create from data connection #

(1) Enter Volcano Engine, click to enter a specific project, click Data Preparation, find Data Connection in the drop-down list, and create a new data connection.

(2) On the page, select Tencent Cloud COS.

(3) Fill in the required basic information, test the connection, and click Save after the connection is successful.

Data connection parameter description:

• Username: Corresponds to accessId
• Password: Corresponds to accessKey
• Region: The geographical region of the Tencent Cloud hosting data center
• Bucket name: The bucket name
• File format: ORC
• Table type: Optional partitioned table (divided into yyyMMdd and yyyy-MM-dd formats) or non-partitioned table
• File directory: The path corresponding to the table. Note: Do not start with / when entering this value.

(4) After confirming that the basic information of the data connection is correct, the data connection is complete.
(5) You can use this connection to create a dataset: Click Data Preparation --> Dataset, and click the "New" button in the upper left corner to create a new dataset.

(6) Enter keywords to search and select the previously created data connection.

2.2 Create from dataset #

(1) Enter Volcano Engine, click Data Preparation --> Dataset, and click the "New" button in the upper left corner to create a new dataset.

(2) When selecting a data connection, click the Other button.

The subsequent steps are the same as steps (2) and (3) in 2.1. After completing the configuration, the pop-up window to select a data connection for the dataset will remain open, allowing you to proceed directly to the next step to create the dataset.

2.3 Directory structure description #

• Non-partitioned table

Table name: student_np

student_np              // The directory corresponding to the table name
├── student.orc         // Data file, type specified in the data connection, will read all data files of that type in the folder
├── student2.orc

• Partitioned table

Table name: student_p1

student_p1              // The directory corresponding to the table name
├── 20240909            // Partition
│   ├── _SUCCESS        // Success flag file
│   ├── student.orc     // Data file, type specified in the data connection, will read all data files of that type in the folder
│   ├── student2.orc
│   └── student3.orc    
├── 20240910
│   ├── _SUCCESS
│   ├── student.orc
│   ├── student2.orc
│   └── student3.orc

2.4 Detailed description of rules #

1. File directory format: Required, for example, data/student
   1. Rule 1: The name of the last-level directory must be the same as the table name, in the format data/{tableName}, for example, data/student
   2. Rule 2: Multi-level directory parsing is not supported. Do not define subdirectories under the directory where the data files are located. If present, the system will automatically skip them.
   3. Rule 3: It is not recommended to change the file directory. Before making changes, update the [DataWind - Data Connection] configuration item to avoid data query exceptions.
   4. Rule 4: One file directory represents one table. Sharing a file directory among multiple tables is not supported, nor is assigning multiple directories to a single table.
2. File directory contents:
   1. Data file: Required. Only ORC is supported. Multiple files are allowed, and there are no restrictions on file names.
   2. Sync success flag file: Required for partitioned tables. Do not include for non-partitioned tables. The file name must be _SUCCESS, with no extension, and it must be placed in the corresponding partition directory.
      1. Logic: Used to indicate whether data has been pushed and to construct dependency logic for S3 tasks.
      2. Rule 1: The file must exist. When the task runs, it reads the _SUCCESS file in the corresponding partition to determine whether the data for the relevant business date is ready.