Data connections support integration with Volcano Cloud TOS.

The following describes two methods for creating a Volcano Cloud TOS data connection.

2.1 Create from data connection #

(1) Enter Volcano Engine, select a specific project, click Data Preparation, find Data Connection in the drop-down list, and create a new data connection.

(2) On the page, select Volcano Cloud TOS.

(3) Fill in the required basic information, test the connection, and click Save after the connection is successful.

Data connection parameter description:

• Server: The access domain name corresponding to S3Endpoint, with https:// added in front. For example, TOS Beijing corresponds to https://tos-s3-cn-beijing.volces.com
• Username: Corresponds to accessId
• Password: Corresponds to accessKey
• Bucket name: The bucket name
• File format: Optional csv, txt
• Table type: Optional partitioned table (supports yyyMMdd and yyyy-MM-dd), or non-partitioned table
• File directory: The path corresponding to the table. Note: Do not start with / when filling in this field.

(4) After confirming that the basic information of the data connection is correct, the data connection is complete.
(5) This connection can be used to create a dataset: Click Data Preparation --> Dataset, and click the "New" button in the upper left corner to create a new dataset.

(6) Enter keywords to search and select the previously created data connection.

2.2 Create from dataset #

(1) Enter Volcano Engine, click Data Preparation --> Dataset, and click the "New" button in the upper left corner to create a new dataset.

(2) When selecting a data connection, click the Other button.

The subsequent steps are the same as steps (2) and (3) in 2.1. After completing the configuration, you will remain in the pop-up window to select a data connection for the dataset, and you can proceed directly to the next step to create the dataset.

2.3 Directory structure description #

• Non-partitioned table

Table name: student_np

student_np              // The directory corresponding to the table name
├── student.txt         // Data file, type specified in the data connection, will read all data files of that type in the folder
├── student2.txt 
└── student_np.yml      // The schema file should have a name that matches the table name

• Partitioned table

Table name: student_p1

student_p1              // The directory corresponding to the table name
├── 20240909            // Partition
│   ├── _SUCCESS        // Success flag file
│   ├── student.csv     // Data file, type specified in the data connection, will read all data files of that type in the folder
│   ├── student2.csv
│   └── student3.csv    
├── 20240910
│   ├── _SUCCESS
│   ├── student.csv
│   ├── student2.csv
│   └── student3.csv
└── student_p1.yml      // The schema file should have a name that matches the table name

2.4 Detailed rule description #

1. File directory format: Required, for example, data/student
   1. Rule 1: The name of the last-level directory must be the same as the table name, in the format data/{tableName}, for example, data/student
   2. Rule 2: Multi-level directory parsing is not supported. Do not define subdirectories under the directory where the data files are located. If present, the system will automatically skip them.
   3. Rule 3: It is not recommended to change the file directory. Before making any changes, update the [DataWind-数据连接] configuration item to avoid data query exceptions.
   4. Rule 4: Each file directory represents a single table. Multiple tables cannot share one file directory, and a single table cannot have multiple directories.
2. File directory contents:
   1. Data file: Required. Only CSV and TXT formats are supported. Multiple files are allowed, and there are no restrictions on file names.
      1. Rule: The column (field) order in the data files must be consistent with the metadata file, and the data files must not contain a header row (field names).
      2. Rule: For each query, all data is retrieved based on the file directory path (excluding files in subdirectories).
      3. Rule: The total size of files in a single file directory is recommended to be less than 500 GB.
      4. Rule: CSV files should preferably be exported directly from database systems or similar sources. Do not edit them a second time, as this may cause date type anomalies.
   2. Metadata file: Required. {tableName}.yml (if there are structural changes, this file needs to be modified), placed directly in the directory corresponding to the table name.
      1. Rule 1: Used to parse data files and map them to structured tables.
      2. Rule 2: {tableName} is the table name. Table name changes are not supported. {tableName} must be consistent with the directory name.
      3. Rule 3: For partitioned tables, partition fields do not need to be specified in the metadata file. The system will automatically add the aeolus_partition field as the partition.
      4. Rule 4: For the file format, see the appendix.
   3. Sync success marker file: Required for partitioned tables. Do not provide for non-partitioned tables. The file name is _SUCCESS, with no extension, and it should be placed in the corresponding partition directory.
      1. Logic: Used to indicate whether data has been successfully pushed and to build dependency logic for S3 tasks.
      2. Rule 1: The file must exist. When the task is executed, the _SUCCESS file in the corresponding partition is read to determine whether the data for the relevant business date is ready.