Data connections support integration with Volcano Cloud TOS.

The following describes two methods for creating a Volcano Cloud TOS data connection.

2.1 Create from data connection #

(1) Enter BytePlus, navigate to a specific project, click Data Preparation, find Data Connection in the dropdown list, and create a new data connection.

(2) Select Volcano Cloud TOS on the page.

(3) Fill in the required basic information, test the connection, and click Save after the connection is successful.

Data connection parameter description:

• Server: The access domain name corresponding to S3Endpoint, with https:// added in front. For example, TOS Beijing corresponds to https://tos-s3-cn-beijing.volces.com
• Username: Corresponds to accessId
• Password: Corresponds to accessKey
• Bucket name: The bucket name
• File format: Optional csv, txt
• Table type: Optional partitioned table (supports yyyMMdd and yyyy-MM-dd formats) or non-partitioned table
• File directory: The path corresponding to the table. Note that the entry should not start with /

(4) After confirming that the basic information of the data connection is correct, the data connection is complete.
(5) You can use this connection to create a dataset: Click Data Preparation --> Dataset, and select the "Create" button in the upper left corner to create a new dataset.


(6) Enter keywords to search and select the previously created data connection.

2.2 Create from dataset #

(1) Enter BytePlus, click Data Preparation --> Dataset, and select the "Create" button in the upper left corner to create a new dataset.

(2) When selecting a data connection, click the Other button.

The subsequent steps are the same as steps (2) and (3) in 2.1. After completing the configuration, you will remain in the popup window for selecting a data connection for the dataset, and you can proceed directly to the next step of dataset creation.

2.3 Directory structure description #

• Non-partitioned table

Table name: student_np

student_np              // The directory corresponding to the table name
├── student.txt         // Data file, type specified in the data connection, will read all data files of that type in the folder
├── student2.txt 
└── student_np.yml      // Schema file, which should have same name as the table name

• Partitioned table

Table name: student_p1

student_p1              // The directory corresponding to the table name
├── 20240909            // partition
│   ├── _SUCCESS        // success file
│   ├── student.csv     // Data file, type specified in the data connection, will read all data files of that type in the folder
│   ├── student2.csv
│   └── student3.csv    
├── 20240910
│   ├── _SUCCESS
│   ├── student.csv
│   ├── student2.csv
│   └── student3.csv
└── student_p1.yml      // Schema file, which should have same name as the table name

2.4 Detailed rule description #

1. File directory format: Required, for example data/student
   1. Rule 1: The name of the last-level directory must match the table name, with the format data/{tableName}, for example data/student
   2. Rule 2: Multi-level directory parsing is not supported. Do not define subdirectories under the directory where the data files are located. If present, the system will automatically skip them
   3. Rule 3: Changing the file directory is not recommended. Before making any changes, update the [DataWind-data connection] configuration item to avoid data query exceptions.
   4. Rule 4: Each file directory represents a single table. Sharing a file directory among multiple tables is not supported, nor is assigning multiple directories to a single table.
2. File directory contents:
   1. Data files: Required. Only CSV and TXT formats are supported. Multiple files are allowed, and there are no restrictions on file names.
      1. Rule: The column (field) order in the data files must be consistent with the metadata file, and the data files must not contain a header row (field name row).
      2. Rule: For each query, all data is retrieved based on the file directory address (excluding files in subdirectories).
      3. Rule: The total size of files in a single file directory is recommended to be less than 500 GB.
      4. Rule: CSV files should preferably be exported directly from a database or similar systems. Do not edit them a second time, as this may cause date type anomalies.
   2. Metadata file: Required. {tableName}.yml (if there are structural changes, this file must be modified), placed directly in the directory corresponding to the table name.
      1. Rule 1: Used to parse data files and map them to structured tables.
      2. Rule 2: {tableName} is the table name. Table name changes are not supported. {tableName} must match the directory name.
      3. Rule 3: For partitioned tables, partition fields do not need to be specified in the metadata file. The system will automatically add the aeolus_partition field as the partition.
      4. Rule 4: See the appendix for file format.
   3. Sync success marker file: Required for partitioned tables. Do not provide for non-partitioned tables. The file name is _SUCCESS, with no suffix, and it is placed in the corresponding partition directory.
      1. Logic: Used to indicate whether data has been fully pushed, and to construct dependency logic for S3 tasks.
      2. Rule 1: The file must exist. When executing a task, the system reads the _SUCCESS file in the corresponding partition to determine whether the data for the relevant business date is ready.