Docs

DataAgent (OnPremise)

Web Connector

Alibaba Cloud OSS

The system supports access to Alibaba Cloud OSS. This article introduces how to access Alibaba Cloud OSS.

Conditions of use

Users need to have project editing or permissions - by user management - module - data connection - new connection permissions to use this function.

Operation steps

In the console, select Data Access > Knowledge Access, then click Create.
Select Alibaba Cloud OSS on the data connection interface.
Fill in the required basic information.

parameter	explain
server	Corresponding to the access domain name of S3Endpoint, and add https://in front of it, for example, OSS Beijing corresponds to http://oss-cn-beijing.aliyuncs.com
user name	Corresponding AccessID
password	Corresponding AccessKey
bucket name	OSS bucket name
file format	Optional csv, txt
Table type	Optional partitioned tables (divided into yyyMMdd and yyyy-MM-dd), non-partitioned tables
file directory	That is, the path corresponding to the table, be careful not to start with/when filling in here. File directory format : Required, such as data/student Rule 1: The last directory name must be the same as the table name, in the format data/{tableName}, such as data/student Rule 2: Multi-level directory parsing is not supported. Do not define subdirectories under the directory where the data file is located. If there is one, the system will automatically skip it. Rule 3: It is not recommended to change the file directory. Please update the [DataWind- Data Connection] configuration item before changing to avoid abnormal data query. Rule 4: A file directory represents a table, and multiple tables sharing a file directory is not supported, nor is it supported to have multiple directories for a table File directory content: Data file: Required, only support CSV, TXT, can have multiple files, name is not limited Rule: The order of columns (fields) in the data file must be consistent with the metadata file, and the data file does not contain a header (field name line). Rule: Each time you query, pull all the data according to the file directory address (excluding subdirectory files) Rule: The total file size in a single file directory is recommended to be less than 500G. Rule: CSV is best for databases and other systems to directly export files, do not re-edit, which may cause abnormal date types Meta information file: Required, {tableName} .yml (if there are structural changes, you need to change this file), directly in the directory corresponding to the table name Rule 1: Use to parse data files mapped to structured tables Rule 2: {tableName} is the table name, table name change is not supported, {tableName} needs to be consistent with the directory name Rule 3: If it is a partition table, the partition field does not need to be written in the meta information file, the system will automatically add aeolus_partition field as partition Rule 4: See Appendix for file format Synchronization success flag file: Partition table is required, do not fill in the non-partition table, the file name is _SUCCESS, no suffix, placed in the corresponding partition directory Logic: used to mark whether the data is pushed, used to build dependency logic on S3 tasks Rule 1: The file must exist. When the task is executed, read the _SUCCESS under the corresponding partition to determine whether the data of the corresponding business date is ready

Click Test Connection and click Save after successful connection.
After confirming that the basic information of the data connection is correct, the data connection is completed.

Directory structure description

non-partitioned table

Table name: student_np

student_np              // The directory corresponding to the table name
├── student.txt         // Data file, type specified in the data connection, will read all data files of that type in the folder
├── student2.txt 
└── student_np.yml      // schema file, whose name must be same as the table name

partition table

Table name: student_p1

student_p1              // The directory corresponding to the table name
├── 20240909            // partition
│   ├── _SUCCESS        // Success file
│   ├── student.csv     // Data file, type specified in the data connection, will read all data files of that type in the folder
│   ├── student2.csv
│   └── student3.csv    
├── 20240910
│   ├── _SUCCESS
│   ├── student.csv
│   ├── student2.csv
│   └── student3.csv
└── student_p1.yml      // schema file, whose name must be same as the table name

Last updated: 2026.06.30 13:51:53