Spark Connect is a remote connection capability introduced in Apache Spark starting from version 3.4. The core concept is to separate the client from the Spark server. The client sends query requests to the remote Spark service via the gRPC protocol. After receiving the request, the server performs query optimization, task scheduling, and distributed computation, then returns the results to the client.

In this architecture, the client is only responsible for constructing the logical query plan (Unresolved Logical Plan) and receiving execution results. There is no need to start a full Spark Driver locally or interact directly with the Spark cluster. Users can use Spark's DataFrame and SQL capabilities in Notebooks, IDEs, or applications as they would with local libraries, while the actual computation is performed remotely. For more information about Spark Connect, refer to: Spark Connect community documentation.

EMR Serverless Spark Connect is a managed remote Spark connection service VolcanoEnginebuilt on Apache Spark Connect. Users can connect to the remote EMR Serverless Spark Connect App using the Python Client or Java Client provided by EMR, and leverage the elastic computing capabilities of Serverless Spark without the need to manage cluster creation, scaling, or maintenance.
The overall process can be summarized as: Users write Spark DataFrame / SQL code locally → The client sends the query plan to the EMR Serverless Spark Connect App → The server executes the actual computation and returns the results.
This approach bridges the local development environment and the remote production computing environment, allowing real data and the complete Spark runtime to be used for debugging and validation during the development phase.

• Interactive Spark development and exploratory analysis in local or online development environments such as Jupyter Notebook, VS Code, and PyCharm.
• Remotely create SparkSession and execute Spark SQL or DataFrame jobs in applications such as Python and Java.
• Connect to the remote Spark environment during the local development phase to process real data, reducing differences and debugging costs between local development and production environments.
• Leverage EMR Serverless's elastic resources, access control, job monitoring, and logging capabilities to reduce the burden of cluster operations and maintenance.

• Client-server decoupling: Client applications do not need to run in the same process as the Spark Driver and can flexibly connect in different environments such as Notebooks, IDEs, and server-side applications.
• Developer-friendly experience: Supports interactive execution of SQL / DataFrame code, suitable for data exploration, logic debugging, and rapid validation.
• No cluster management required: Underlying compute resources are hosted by EMR Serverless, so users do not need to manually create, scale, or maintain Spark clusters.
• Elastic resources and controllable costs: Compute tasks are executed on remote Serverless Spark environments as needed, with resources allocated based on actual workload, avoiding local resource bottlenecks.
• Production consistency: The remote Spark runtime environment can be connected during development to process real data, helping reduce environment differences between development and production deployment.

Current client capabilities are divided into two parts:

• Java Client: Suitable for server-side programs, task platforms, and Java/Scala ecosystem integration
• Python Client: Suitable for Notebook, interactive analysis, and Python service integration

Core capability comparison:

Select the corresponding client environment based on the Spark version of the target Spark Connect App.

Limitations #

Spark Connect uses a client-server separation architecture, so some traditional Spark local mode or internal Driver capabilities cannot be used directly. The main current limitations are as follows:

• SparkContext-related capabilities, such as parallelize, textFile, broadcast, and accumulator, are not currently supported. It is recommended to use DataFrame / SQL capabilities instead.
• The RDD programming model, including RDD API and DataFrame.rdd, is not currently supported. It is recommended to rewrite related processing logic using DataFrame API, SQL, or UDF.
• Direct access to underlying JVM interfaces such as _jvm and _jsc is not currently supported. It is recommended to implement through Spark configuration, built-in functions, or server-side preset capabilities.
• MLlib is not currently supported in Spark 3.5.1.

Recommendations #

1. Before creating or connecting to a Spark Connect App, confirm the Spark version of the target App.
2. The pyspark / py4j versions of the Python Client must match the Spark version of the target Spark Connect App. The Java Client requires selecting the JAR corresponding to the target Spark version.
3. It is recommended to use serverId + getOrCreate() to connect to an existing Spark Connect App to avoid duplicate creation.
4. After use, it is recommended to call spark.stop() to release the current SparkSession.
5. For Python environments, third-party packages, or large file dependencies, it is recommended to configure them when creating the Spark Connect App. Temporary dependencies can be dynamically uploaded using addArtifact. For detailed operations, see Add Spark Connect App dependencies.

Usage process #

The main steps for using EMR Serverless Spark Connect are as follows:

1. Install and configure Spark Connect Client。
2. Create Spark Connect App, to add dependencies, see Add Spark Connect App dependencies.
3. Connect to Spark Connect App and submit job. To perform visual analysis tasks with third-party libraries, refer to Job visualization and analysis.

related operations #

To view job progress and job runtime information, you can use plugins to enhance the job management experience. For detailed operations, refer to Job management: Notebook progress bars and UI plugins.