Preparation #

Create a TOS bucket #

1. Log in to the TOS console and create a TOS bucket. For specific creation methods, refer to: Create TOS bucket.
2. Create the required directory based on the TOS bucket you created. This directory serves as the default storage directory for the Spark Doris Connector.

Doris data preparation #

1. Log in to the EMR Serverless console and create a Doris instance.
2. After connecting to the instance, execute the following SQL to prepare the database and tables. For connection methods, see: Connect to instance. For basic Doris usage, see: Basic Doris usage.

CREATE DATABASE if not EXISTS online_database;
USE online_database;
CREATE TABLE table1
(
    siteid INT DEFAULT '10',
    citycode SMALLINT,
    username VARCHAR(32) DEFAULT '',
    pv BIGINT SUM DEFAULT '0'
)
AGGREGATE KEY(siteid, citycode, username)
DISTRIBUTED BY HASH(siteid) BUCKETS 10;

3. After data preparation is complete, you can view the data. If the data display content is as follows, data preparation is complete:

Serverless Spark data preparation #

1. Create a resource queue.
   Log in to the EMR Serverless console to create a Serverless dedicated queue. When the queue status changes to running, you can proceed to the next step.

   warning

   When configuring a resource queue, you must enable the features related to network configuration, and the VPC network used by this queue must be exactly the same as the VPC network used by the Doris instance. This is the only way to ensure network compatibility and connectivity between the queue and the Doris instance, allowing tasks using this queue to successfully access the Doris instance.

   tip

   It is recommended to disable the global cross VPC routing feature, and only add the set serverless.cross.vpc.access.enabled=true; parameter to tasks that need to access Doris to enable network connectivity.

   
2. Create a database.
   Log in to the EMR Serverless page, click Create job, go to the Create job page, select the created dedicated queue > Default compute group, and execute the CREATE DATABASE IF NOT EXISTS ss_wr2sr; command to create the database ss_wr2sr.
3. Create data tables.
   Log in to the EMR Serverless > Create job > Edit job page, select the created dedicated queue > Default compute group, and execute the following commands to create two data tables, ss_table1 and ss_table2, in the database ss_wr2sr.

   CREATE TABLE if not exists ss_wr2sr.ss_table1(
     `siteid` int,
     `citycode` SMALLINT,
     `username` string,
     `pv` BIGINT
   );
   
   CREATE TABLE if not exists ss_wr2sr.ss_table2(
     `siteid` int,
     `citycode` SMALLINT,
     `username` string,
     `pv` BIGINT
   );
   

Procedure #

1. Read and write Doris data tables using Serverless Spark.
   Go to the EMR Serverless job editing page, enter the Spark SQL statements from the example SQL code block below, and click the Job editing page's lower left Run button to execute the SQL job.

   -- Fill in the TOS address of the JAR dependency package
   set las.spark.jar.depend.jars = [{"fileName":"{dependency jar address}"}];
   -- Optional parameter: set the default directory for Spark SQL.
   set spark.sql.catalog.{your_catalog_name}=org.apache.doris.spark.catalog.DorisTableCatalog;
   set spark.sql.catalog.{your_catalog_name}.doris.fenodes={fe_ip:fe_http_port};
   set spark.sql.catalog.{your_catalog_name}.doris.query.port=9030;
   set spark.sql.catalog.{your_catalog_name}.doris.user={user name};
   set spark.sql.catalog.{your_catalog_name}.doris.password={user password};
   
   show databases;
   SELECT * FROM {your_catalog_name}.online_database.table1;
   INSERT INTO {your_catalog_name}.online_database.table1 select * from db.table;
   

2. Read Doris data tables and write Hive data tables using Serverless Spark.
   Log in to the EMR Serverless job editing page, enter the Spark SQL statements from the example SQL code block below, and click the lower left Run button in the job editing box to execute the SQL job.

   -- Fill in the TOS address of the JAR dependency package
   set las.spark.jar.depend.jars = [{"fileName":"{dependency jar address}"}];
   
   set spark.sql.catalog.{your_catalog_name}=org.apache.doris.spark.catalog.DorisTableCatalog;
   set spark.sql.catalog.{your_catalog_name}.doris.fenodes={fe_ip:fe_http_port};
   set spark.sql.catalog.{your_catalog_name}.doris.query.port=9030;
   set spark.sql.catalog.{your_catalog_name}.doris.user={user name};
   set spark.sql.catalog.{your_catalog_name}.doris.password={user password};
   
   INSERT into ss_wr2sr.ss_table1 SELECT * FROM {your_catalog_name}.online_database.table1;
   

3. Read Hive data tables and write Doris data tables using Serverless Spark.
   Log in to the EMR Serverless job editing page, enter the Spark SQL statements from the example SQL code block below, and click the lower left Run button in the job editing box to execute the SQL job.

   -- Fill in the TOS address of the JAR dependency package
   set las.spark.jar.depend.jars = [{"fileName":"{dependency jar address}"}];
   
   set spark.sql.catalog.{your_catalog_name}=org.apache.doris.spark.catalog.DorisTableCatalog;
   set spark.sql.catalog.{your_catalog_name}.doris.fenodes={fe_ip:fe_http_port};
   set spark.sql.catalog.{your_catalog_name}.doris.query.port=9030;
   set spark.sql.catalog.{your_catalog_name}.doris.user={user name};
   set spark.sql.catalog.{your_catalog_name}.doris.password={user password};
   
   INSERT into {your_catalog_name}.online_database.table1 SELECT * FROM ss_wr2sr.ss_table2;
   

4. For more examples and parameters, refer to the Doris community documentation.

Dependency JAR addresses #

Download the required spark-doris-connector package from the Doris community documentation and upload it to the corresponding Volcano TOS path.