Version requirements #

Python >= 3.8

Environment installation #

Install Spark Connect SDK #

emr_spark_connect-3.23.1-py3-none-any.whl

Unknown size

python -m pip install --force-reinstall --no-deps emr_spark_connect-3.23.1-py3-none-any.whl

Install EMR Python Query SDK #

Download SDK package link: Preparation: Install the SDK.

python -m pip install --force-reinstall python_serverless-1.0.3.11-py3-none-any.whl

Install PySpark environment (version 3.5.1) #

EMR pyspark client

pyspark-3.5.1-py2.py3-none-any.whl

Unknown size

python -m pip install --force-reinstall "pyspark[connect] @ file:///xxxxx/pyspark-3.5.1-py2.py3-none-any.whl"

Install Spark environment (version 4.1.2) #

You can choose to install the EMR pyspark client or the open-source client.

[Recommended] EMR pyspark client

pyspark-4.1.2-py2.py3-none-any.whl

Unknown size

python -m pip install --force-reinstall "pyspark[connect] @ file:///xxxxx/pyspark-4.1.2-py2.py3-none-any.whl"

Open-source client

python -m pip install --force-reinstall pyspark-client==4.1.2

Add the following repository configuration and dependencies to the project's pom.xml :
Specify repo:

<repositories>
    <repository>
        <id>bytedance-public</id>
        <name>bytedance maven</name>
        <url>https://artifact.bytedance.com/repository/releases</url>
    </repository>
</repositories>

Spark 3.5.1：

<dependencies>
    <dependency>
        <groupId>com.bytedance.emr</groupId>
        <artifactId>emr-spark-connect-client</artifactId>
        <version>3.23.1.2-spark3.5.1</version>
    </dependency>
</dependencies>

Spark 4.1.2：

<dependencies>
    <dependency>
        <groupId>com.bytedance.emr</groupId>
        <artifactId>emr-spark-connect-client</artifactId>
        <version>3.23.1.2-spark4.1.2</version>
    </dependency>
</dependencies>