This article describes how to integrate BytePlus's EMR Serverless Spark to create datasets.

• This feature currently supports SaaS or on-premises deployments of version V2.75.0 and above. To purchase, please contact the sales team for consultation.
• The current product only supports integration with BytePlus's EMR Serverless Spark.

• Before connecting data, contact the database administrator to collect the following information:
  • The IP address and port number of the server where the database is located
  • The username and password of the database
  • The queue and compute group of the database
• Currently, DataWind does not support the default compute group. A separate compute group of type spark must be created in EMR.

1. Go to BytePlus, enter a specific project, click Data Preparation, find Data Connection in the drop-down list, and create a new data connection.

   

2. Select EMR Serverless Spark on the page.

3. Fill in the required basic information, test the connection, and click Save after the connection is successful.

• The parameter configuration instructions for the SaaS version are as follows:
  

  Parameters	Configuration description
  Data connection name	Name for the new data connection, used to distinguish between different database connections in the system. The default is Emr_Serverless_Spark_timestamp. The name can be modified, but please enter a standardized name. Only combinations of Chinese, English, numbers, and underscores are allowed, and spaces cannot be included at the beginning or end.
  Connection method	Two options: IP/Port and JDBC connection string.
  Server	Required when the connection method is IP/Port. The address for connecting to EMR Serverless Spark varies by region. North China: serverless-spark-jdbc.emr.cn-beijing.volces.com East China: serverless-spark-jdbc.emr.cn-shanghai.volces.com South China: serverless-spark-jdbc.emr.cn-guangzhou.volces.com Johor: serverless-spark-jdbc.emr.ap-southeast-1.volces.com
  Port	Required when the connection method is IP/Port. The EMR instance port, typically 10009 for public cloud. For details, refer to the documentation: Cluster component port description.
  Username	The tenant's AccessKey. For information, refer to the documentation: Access Key (secret) management.
  Password	The tenant's SecretKey. For information, refer to the documentation: Access Key (secret) management.
  Queue	Name of the queue used for the connection. Currently, only dedicated queues are supported; public queues are not supported. Queue retrieval path in EMR: "Console" - "Resource Management" - "Serverless" - "Queue Details".
  Compute group	Compute group name. warning The default compute group is not supported. You need to create a separate compute group of type Spark.
  EMR cluster region	Specify the region where the EMR cluster is located. Options include North China (Beijing), East China (Shanghai), South China (Guangzhou), or Asia Pacific (Johor).
  Bucket name	To accelerate extraction using TOS, enter the TOS bucket name here. For details, see Optional: Extraction acceleration.

• The parameter configuration instructions for the on-premises version are as follows:

1. Go to BytePlus, click Data Preparation --> Dataset, and click the "New" button in the upper left corner to create a new dataset.

   

2. When selecting a data connection, click the Other button.

   

3. The subsequent steps are the same as steps 2 and 3 of Create from data connection. After completing the configuration, you will remain in the pop-up window for selecting a data connection for the dataset, where you can proceed directly to the next step of dataset creation.

1. Use the previously created data connection to create a dataset: Click Data Preparation --> Dataset, and click the "New" button in the upper left corner to create a new dataset.

   

2. Search for or select from the drop-down list the previously created data connection.

   

Currently, DataWind connects to EMR Serverless Spark using a query gateway approach, which has limitations in scenarios involving large-scale data extraction. To extract large volumes of data, TOS can be used to transfer and store result sets.
DataWind supports acceleration by reading result data from TOS, with extraction speeds expected to increase by tens of times.

Step 1: Create a TOS acceleration bucket #

After creating the storage bucket, refer to Create a storage bucket for detailed steps.

Step 2: Configure TOS to allow cross-tenant access #

After creating the storage bucket authorization policy, refer to Cross-account authorization for detailed steps.

Step 3: DataWind configuration #

When DataWind connects to EMR Serverless Spark, enter the created TOS bucket name in the connection parameters to enable acceleration.