Docs

E-MapReduce

Get to know EMR Serverless queue EMR on ECS EMR on VKE Serverless instance Component and API Document

E-MapReduce

Component and API

CLI reference

EMR CLI reference

API reference

EMR on ECS API reference

Calling methods

Operations management

ListOperations - Operation list

Cluster management

UpdateClusterAttribute - Update cluster attributes

ReleaseCluster - Release or unsubscribe a cluster

ListClusters - Cluster list

GetCluster - Retrieve a cluster

CreateCluster - Create a cluster

Order management

ListOrders - order list

Node group management

ScaleOutNodeGroup - scale out node group

CreateNodeGroup - Create a node group

UpdateNodeGroupChargeType - Update node group charge type

ScaleInNodeGroup - Scale in node group

UpdateNodeGroupEcsSpec - Update node group ECS specifications

ScaleUpNodeGroupDisk - Scale up node group disk

ListNodeGroups - Node group list

DeleteNodeGroup - Delete node group

UpdateNodeGroupAttribute - Update node group attributes

ListNodes - Node list

ListNodeGroupElasticScheduledInstances - Retrieve the list of ESI instances in a node group

User management

GetClusterUsers - Retrieve EMR managed user details

UpdateClusterUserPassword - Update cluster user password

ListClusterUsers - Get cluster user list

DeleteClusterUser - Delete cluster user

UpdateClusterUser - Modify information of an existing user

CheckUserCredentials - Validate cluster user

CreateClusterUser - Create cluster user

User group management

GetClusterUserGroups - Cluster user group details

ListClusterUserGroups - Retrieve cluster user group list

CreateClusterUserGroup - Create a cluster user group

DeleteClusterUserGroup - Delete cluster user group

UpdateClusterUserGroup - Update cluster user group

Application management

ListApplicationConfigs - configuration item list

GetApplicationConfigFile - Retrieve cluster configuration file information

RunApplicationAction - Execute application operation

ListApplicationConfigFiles - Cluster configuration file list

UpdateApplicationConfig - update configuration items

ListApplicationConfigHistories - application configuration modification history list

GetApplicationPacket - Retrieve application installation package information

ListApplications - Cluster list

ListComponentInstances - Component instance list

ListComponents - component list

API version note

EMR on VKE API reference

Read before use

API overview

Cluster management

CreateVirtualCluster - Create a cluster

GetVirtualCluster - View cluster details

DeleteVirtualCluster - Delete cluster

ListVirtualClusters - Cluster list

Operations management

GetOperation - View operation log information

Application management

UndeployApplications - Uninstall applications

DeployApplications - Deploy applications

RunApplicationAction - Run application Action

ListApplications - Application list

EMR Serverless API reference

Invocation methods

SDK reference

EMR on ECS SDK reference

SDK overview

EMR Serverless SDK reference

Spark-submit tool usage instructions

Java Query SDK

Python Query SDK

Guide to operating components

Airflow

Basic usage

Integrating Airflow logs with TOS

Advanced usage

FAQs and important notes

Celeborn

ClickHouse

Delta Lake

DolphinScheduler

DolphinScheduler on EMR Serverless

Use the task plugin (Task Plugin)

Spark-submit

Dolphin Serverless task output JobID

best practices

Best practices for DolphinScheduler resource path settings

DolphinScheduler supports Volcano Voice alerts

DolphinScheduler shell kill application practices

Doris

Advanced usage

Best practices

Doris connects to Tableau

FAQs

Flink

Overview

Basic Flink usage

Flink SQL Client usage reference

Flink supports the Protobuf format

HBase

Advanced usage

HDFS

Hive

Basic usage

Hive basic operations

Hive connection methods

Determine the HiveServer2 address

Important notes for Hive3 usage

Advanced usage

Manage LDAP authentication

Hive job optimization

HiveServer2 load balancing

Hive supports Chinese field names and Chinese partitions

Development guide

User-defined function (UDF)

Best practices

Hive access to Hudi data

Run the TPC-DS Benchmark on an EMR on ECS cluster

FAQs

Hue

Usage instructions

Best practices

Querying data using Hue

Hue high availability: Accessing other services within the cluster

Iceberg

Instructions

Basic usage

Basic usage of Iceberg

Basic usage of Iceberg (applicable to EMR 2.x versions)

Access the Iceberg Catalog through a Doris instance

Advanced usage

Iceberg parameter configuration

Spark streaming read and write Iceberg

Spark streaming read and write Iceberg (for EMR 2.x versions)

Batch reading and writing Iceberg tables with Spark

Iceberg integration with Trino/Presto

Iceberg and Flink integration

Operate Iceberg data in Hive

Operate Iceberg tables in TOS

Operate Iceberg tables in TOS (applicable to EMR 2.x versions)

Use Iceberg in Dataleap

Impala

Impala overview

Basic usage

Impala basic operations

Impala connection methods

Impala-shell tool

Advanced usage

Manage LDAP authentication

Best practices

Access Kudu using Impala

Kafka

Kerberos

Use Kerberos-authenticated clusters

Configure cross-realm trust between Kerberos clusters

EMR uses external KDC

Knox

Note

Kudu

Kudu overview

Basic usage

Basic Kudu operations

Advanced usage

Impala integration with Kudu

Synchronize data to Kudu using Flume

Kyuubi

Kyuubi overview

Kyuubi quickstart

Kyuubi advanced features

Lance

Lance overview

Quick Start

Lake format conversion

Lake format conversion: Ray reads WebDataset and writes to Lance

Ecosystem and integration

EMR Serverless Spark integration with Lance

Operate Lance data with Ray

Use SparkSQL Warehouse to access Lance tables

Best practices

Access Lance data on TOS using the Lance Python SDK

Train a CLIP multimodal model using the Lance data format

Perform multimodal vector data retrieval using the Lance data format

EMR Daft integration with Lance: operation guide and best practices

Best practices for adding and modifying columns in Lance datasets using Spark

PySpark+PyLance best practices for avoiding large field shuffle

Use cases

Model training

Version management

Task-level fault tolerance

Mixed text and image layout

Multimodal retrieval

Model labeling

Smart driving scenario

Livy

MapReduce2

OpenLDAP

Best practices

Manually configure LDAP authentication

Paimon

Phoenix

Presto

Proton

Release versions

Proton release versions

JobCommitter version

Basic usage

Identity and permission configuration

DolphinScheduler uses Proton

Getting started

Proton cache mode

Proton cache mode (TOS Accelerator)

Proton no-cache mode

Advanced usage

Proton Cli

Determining Proton cache size

Proton metadata synchronization

Distcp copy data to TOS

Proton parameter configuration

Proton on-demand cache user manual

DistCp copies data from OSS to TOS

Mount Proton using FUSE

Best practices

EMR storage-compute separation Job Committer best practices

Proton SDK optimization guide

Proton On VKE

PyProton

Introduction and usage of FSSpec

PyProton introduction and usage documentation

PyProton best practices

Use FSSpec in a Python environment

Use TOSFS in Ray Data

Using FSSpec in the PyTorch training framework

Pulsar

Ranger

Component integration

Advanced features

Ranger User Sync integration with LDAP

View Ranger audit log information

Configure DataLeap data development permissions

Ray

Ray on VKE product introduction

Use RayHistoryServer to view historical jobs

Ray vs Spark comparison

Lance-Ray库

EMR Lance-Ray general operation guide

How to resolve Ray cluster load and memory overflow issues

Use RayJob

RayJob Quick Start

RayJob submission guide

View task status through Ray Dashboard

Monitor RayJob using EMR

KubeRay RayJob submission guide

Streaming data processing based on Ray

Using RayCluster

RayCluster Quick Start

User guide for submitting Ray jobs

Monitor jobs using Ray Dashboard

Examples of accessing TOS with Ray

Word frequency statistics using Ray (WordCount)

Ray Placement Group usage guide

Ray GCS usage guide

RayCluster supports multiple WorkerGroups

Mount PersistentVolumeClaim to RayCluster

Guidance for using KubeRay to manage RayClusters across different namespaces

Using RayData

AutoScale: Using AdaptiveStreamingExecutor

Practical application of Ray Data Checkpoint in multimodal data processing

Ray Data file output: Two-phase write

Best practices

Ray tasks run on GPUs

Enable Ray Dashboard metrics capability

Replace the K8S Default Scheduler with Volcano Scheduler to run Spark and Ray jobs

Enable Autoscaler for RayCluster

Ray integration with log service TLS

Practical guide for converting traditional Python programs to Ray programs

Ray Data best practices for LLM data preprocessing

Ray Data backpressure enhancement analysis

Ray architectural practices in multimodal image processing

Build online model inference services based on Ray Serve

Spark

Basic usage

Basic usage of PySpark

Basic usage of Spark RDD

DML statements

Best practices

Spark on GPU best practices

MaxCompute Buildin Function Plugin

Spark optimization guide

Sqoop

StarRocks

Plan StarRocks clusters

quickstart

Operation and maintenance management

StarRocks UI usage

StarRocks parameter configuration

Implement load balancing using CLB

User and permission management

Basic usage

Explanation of observed metrics for the StarRocks engine

Advanced usage

Using StarRocks partitioned tables

StarRocks access TOS

StarRocks Spark Connector

StarRocks Flink Connector

Slow SQL analysis

Data lake analysis

Catalog

Default Catalog

External Catalog

Best practices

StarRocks high availability best practices

Implement data integration from MySQL to StarRocks using Serverless Flink

Analyze StarRocks data using Intelligent Data Insights

StarRocks table model design

Use DataSail for MySQL to StarRocks data integration

Starrocks major version upgrade

TensorFlow

Tez

Trino

YARN

Basic usage

Yarn Fair Scheduler configuration

Advanced usage

YARN enhancements

YARN queue management

YARN resource pool management

FAQs

Best practices

Configure user group queue mapping

ZooKeeper

Overview

Basic usage

Advanced usage

【不发布】Daft

Best practices: dynamic dependencies and business code distribution

Documentation

E-MapReduce

Guide to operating components

Proton

Release versions

JobCommitter version

Copy page

Download PDF

Release versions

JobCommitter version

Copy page

Download PDF

JobCommitter version

Spark engine

Spark version	Proton-Spark version
Spark-3.1.2	proton-spark-3.1.2-1.0.jar
Spark-3.3.3	proton-spark-3.3.3-1.4.jar
Spark-3.5.1	proton-spark-3.5.1-2.2.jar

Last updated: 2026.06.01 16:30:30

Public security network record in Beijing: No. 11010802032137

E-MapReduce

Spark engine #

Spark engine