The following example shows how to create a new Spark Connect App from the client and return the App ID, status, and tracking URL.

from emr_spark_connect import EMRSparkConnectSession, StaticCredentials

app = (
    EMRSparkConnectSession.builder
    .region("<region>")
    .credentials(StaticCredentials("<your-ak>", "<your-sk>"))
    .queue("<your_queue_name>")
    .allowOtherUsers(True)
    
    .getOrCreateApp()
    
)

print(app.getServerId())
print(app.getStatus())
print(app.getTrackingUrl())

To create a Spark Connect App for Spark 4.x, add the parameter: serverless.spark.runtime.version=spark4.1
Example:

from emr_spark_connect import EMRSparkConnectSession, StaticCredentials

app = (
    EMRSparkConnectSession.builder
    .region("<region>")
    .credentials(StaticCredentials("<your-ak>", "<your-sk>"))
    .queue("<your_queue_name>")
    .allowOtherUsers(True)
    
    .config("serverless.spark.runtime.version", "spark4.1") #Create Connect App for Spark 4.x
    .getOrCreateApp()
)

To close the Spark Connect App after creation, call .cancel(), for example:

app.cancel()

Note:

The following example shows how to create a new Spark Connect App from the client and output the App ID, status, and tracking URL.

import com.volc.emr.spark.connect.EMRSparkConnectSession;
import com.volc.emr.spark.connect.EMRSparkConnectProvider;
import com.volc.emr.spark.connect.StaticCredentials;
import com.volc.emr.spark.connect.SparkApp;
import org.apache.spark.sql.SparkSession;

public class SparkConnectJavaExample {
    public static void main(String[] args) {
        SparkApp app = EMRSparkConnectSession.builder()
                .region("<region>")
                .credentials(new StaticCredentials("<your-ak>", "<your-sk>"))
                .queue("<your_queue_name>")
                .allowOtherUsers(true)
                
                .getOrCreateApp();
        
        System.out.println(app.getConnectId());
        System.out.println(app.getStatus());
        System.out.println(app.getTrackingUrl());
    }
}

To close the Spark Connect App after creation, call .cancel(), for example:

app.cancel()

Note:

1. Log in to EMR Serverless console .

2. In the top menu bar, select the region and project space from the dropdown according to your scenario.

3. Click Job center> Job management or Job instance to enter the Create job page.
   

4. Configure job parameters and submit.

   1. Configure basic information for the job. See the table below for parameter notes.

      Parameter	Configuration operation
      Job Name	The name of this Spark Connect App. It cannot be changed after creation.
      Job types	The type of Spark Connect App created.
      Execution resources	EMR resources used for job execution are divided into resource type dropdown and queue/project dropdown. You can select specific resources and search by name or ID.
      development model	UI: Submit job content through a visual interface. JSON: Submit job content in JSON format, which is more suitable for batch configuration or reusing existing configurations. For example:

   2. Configure job content. The parameter entry varies by development mode. See the table below for parameter notes.

      UI mode parameters	JSON mode parameters	Required	Note
      EngineRuntime	ServerlessSparkRuntimeVersion	Y	Select the Spark computing engine environment. You can choose the Spark version and the corresponding Java Runtime or Bolt Runtime. Java Runtime: Standard Java runtime. Bolt Runtime: VolcanoEngineSelf-developed native kernel acceleration engine that can improve the execution speed of Spark, Presto, and other jobs. For detailed information, see Spark on Native Engine: Bolt. warning The EngineRuntime parameter of the job has a higher priority than the default compute group and general compute group parameters.
      Dependency jars (jars)	Jars	N	Third-party or custom JAR packages that Spark Connect App depends on. Executors will automatically load them when executing tasks. They must be uploaded to TOS and the path specified here.
      Depend on File resources	Files	N	Resource files that Spark Connect App depends on. They must be uploaded to TOS and the path specified here.
      Dependency Archive	Archives	N	Archives that Spark Connect App depends on. They must be uploaded to TOS and the path specified here.
      Spark parameters (spark conf)	SparkConf	N	SparkConf is used to pass runtime configurations when creating a Spark Connect App, including engine type, authentication information, access control, and resource specifications. These configurations take effect during the app startup phase and directly affect the application's operating mode, accessibility, and resource allocation. For example, "spark.emr.connect.engine.allow.other.users": true. warning In UI mode, EngineRuntime parameters are passed as options. If EngineRuntime parameters are submitted here, they will not take effect.

      Example of submitting parameters in JSON mode:

      {
        "JobType": "SparkConnect",
        "ServerlessSparkRuntimeVersion": "spark<version>",
        "SparkConf": "serverless.spark.access.key=${AK}\nserverless.spark.secret.key=${SK}\nspark.emr.connect.engine.allow.other.users=true\nspark.executor.instances=2\nspark.executor.cores=2\nspark.executor.memory=4g",
        "Jars": [
          {
            "fileName": "tos://bucket/path/extra-dependency.jar"
          }
        ],
        "Files": [
          {
            "fileName": "tos://bucket/path/dependency-file"
          }
        ],
        "Archives": []
      }
      

      Notes for SparkConf parameters:

      Configuration item	Example value	Note
      serverless.spark.access.key	${AK}	Access Key (AK). Identity used for server-side authentication and signature calculation. It is recommended to inject via environment variables or key management. Do not hardcode in plain text in the configuration.
      serverless.spark.secret.key	${SK}	Secret Key (SK). Used for signature calculation. This is highly sensitive information and must be properly protected. Do not write it into logs, documents, or code repositories.
      spark.emr.connect.engine.allow.other.users	true	Whether to allow other users to connect to or access this Spark Connect engine. The default value is false.
      spark.executor.instances	2	The number of Executor instances, which determines parallelism and resource usage.
      spark.executor.cores	2	The number of CPU cores for each Executor, which, together with instances, determines the total CPU resources.
      spark.executor.memory	4g	The memory for each Executor.