Data connections support integration with Alibaba Cloud OSS.

The following describes two methods for creating an Alibaba Cloud OSS data connection.

2.1 Create from data connection #

(1) Enter Volcano Engine, click to access a specific project, click Data Preparation, find Data Connection in the dropdown list, and create a new data connection.

(2) On the page, select OSS.

(3) Fill in the required basic information, test the connection, and click Save after the connection is successful.

Description of data connection parameters:

• Server: Corresponds to the S3Endpoint access domain name, and add https:// in front. For example, OSS Beijing corresponds to http://oss-cn-beijing.aliyuncs.com
• Username: Corresponds to accessId
• Password: Corresponds to accessKey
• Bucket name: The bucket name
• File format: Optional csv, txt
• Table type: Optional partitioned table (supports yyyMMdd and yyyy-MM-dd), or non-partitioned table
• File directory: The path corresponding to the table. Note that when filling in this field, do not start with /

(4) After confirming that the basic information of the data connection is correct, the data connection is complete.
(5) You can use this connection to create a dataset: Click Data Preparation --> Dataset, and click the "New" button in the upper left corner to create a new dataset.

(6) Enter keywords to search and select the previously created data connection.

2.2 Create from dataset #

(1) Enter Volcano Engine, click Data Preparation --> Dataset, and click the "New" button in the upper left corner to create a new dataset.

(2) When selecting a data connection, click the Other button.

The subsequent steps are the same as steps (2) and (3) in 2.1. After completing the configuration, you will remain on the popup window to select a data connection for the dataset, and you can proceed directly to the next step of dataset creation.

2.3 Directory structure description #

• Non-partitioned table

Table name: student_np

student_np              // The directory corresponding to the table name
├── student.txt         // Data file, type specified in the data connection, will read all data files of that type in the folder
├── student2.txt 
└── student_np.yml      // The schema file should have a name that matches the table name

• Partitioned table

Table name: student_p1

student_p1              // The directory corresponding to the table name
├── 20240909            // Partition
│   ├── _SUCCESS        // Success flag file
│   ├── student.csv     // Data file, type specified in the data connection, will read all data files of that type in the folder
│   ├── student2.csv
│   └── student3.csv    
├── 20240910
│   ├── _SUCCESS
│   ├── student.csv
│   ├── student2.csv
│   └── student3.csv
└── student_p1.yml      // The schema file should have a name that matches the table name

2.4 Detailed description of rules #

1. File directory format: Required, for example data/student
   1. Rule 1: The name of the last-level directory must be consistent with the table name, in the format data/{tableName}, for example data/student
   2. Rule 2: Multi-level directory parsing is not supported. Do not define subdirectories under the directory where the data files are located. If present, the system will automatically skip them.
   3. Rule 3: It is not recommended to change the file directory. Before making any changes, update the [DataWind-数据连接] configuration item to avoid data query errors.
   4. Rule 4: One file directory represents one table. Multiple tables cannot share a single file directory, and one table cannot have multiple directories.
2. File directory contents:
   1. Data files: Required. Only CSV and TXT formats are supported. Multiple files are allowed, and there are no restrictions on file names.
      1. Rule: The column (field) order in the data files must match the metadata file, and the data files must not contain a header row (field names).
      2. Rule: Each time a query is performed, all data is retrieved based on the file directory path (excluding files in subdirectories).
      3. Rule: The total size of files in a single file directory is recommended to be less than 500 GB.
      4. Rule: It is best if CSV files are exported directly from databases or similar systems. Do not edit them a second time, as this may cause date type errors.
   2. Metadata file: Required. {tableName}.yml (if there are structural changes, this file needs to be modified), placed directly in the directory corresponding to the table name.
      1. Rule 1: Used to parse data files and map them to a structured table.
      2. Rule 2: {tableName} is the table name. Table name changes are not supported. {tableName} must match the directory name.
      3. Rule 3: For partitioned tables, partition fields do not need to be specified in the metadata file. The system will automatically add the aeolus_partition field as the partition.
      4. Rule 4: For the file format, see the appendix.
   3. Sync success marker file: Required for partitioned tables. Do not include for non-partitioned tables. The file name is _SUCCESS, with no extension, and it is placed in the corresponding partition directory.
      1. Logic: Used to indicate whether data has been successfully pushed and to build dependency logic for S3 tasks.
      2. Rule 1: The file must exist. When the task is executed, the _SUCCESS file in the corresponding partition is read to determine whether the data for the relevant business date is ready.