prep.job.datax.channel

Read concurrency, which needs to be used together with splitPk.
Recommended not to exceed: 5 (concurrency will put pressure on the target database)

Not enabled (1)

No

• No unique key: 1
• Contains a unique key such as id: less than 5

prep.job.datax.partition.num

After DataX extracts the data, the number of partitions for subsequent ETL processing is calculated on the Spark engine side.
Usage scenario: When the target data volume to be extracted is very large (more than 10 million rows), it is recommended to enable this to improve downstream processing speed. The recommended value is 512 (a power of 2).

Not enabled (1)

No

• Small data volume: 1
• Large data volume: 512

prep.job.datax.splitPk

Description: When MysqlReader performs data extraction, if splitPk is specified, it means the user wants to use the field represented by splitPk for data sharding. DataX will then launch concurrent tasks for data synchronization, which can greatly improve the efficiency of data synchronization.
It is recommended that splitPk users use the table's primary key, as the primary key is generally evenly distributed, making it less likely for the resulting partitions to have data hotspots. Currently, splitPk only supports partitioning integer data and does not support other types such as floating-point, string, or date. If the user specifies other unsupported types, MysqlReader will report an error. If splitPk is not specified, including cases where splitPk is not provided or its value is empty, DataX will treat it as using a single channel to synchronize the table data. Required: No Default value: Empty In fact, according to test results, partitioning must be used together with channel. If only splitPk is enabled but the channel configuration is set to 1, there will still be no concurrency effect.

Do not enable

No

• If there is a unique key, configure it. For example, if the unique key is uid, enter: uid
• If there is no unique key, do not configure.

prep.job.datax.timeout

Advanced configuration for DataX tasks: the wait timeout for MaxCompute data source partition detection, in minutes. If not configured, the task will fail directly when the queried table partition is empty.

Do not enable

No

Not recommended to enable

prep.job.datax.disable

Disable DataX for the task

false

No

false

spark.driver.cores

The number of CPUs for the driver.

2

int

spark.driver.memory

The memory size of the driver.

4g

string

spark.driver.maxResultSize

The upper limit of data pulled to the Driver. Generally, no adjustment is needed.

2g (defaults to half of spark.driver.memory)

string

spark.executor.cores

The number of CPUs for each executor. The maximum concurrency of a Spark task is spark.executor.cores * number of executors

2

int

spark.executor.memory

The memory size of the executor.

4g

string

spark.dynamicAllocation.maxExecutors

The maximum number of executors for a single Spark task.

100

int

spark.sql.autoBroadcastJoinThreshold

The broadcast threshold on the data source side. Set to -1 to disable. PS: Disabling this does not affect Spark AQE's own broadcast.

int

spark.sql.broadcastTimeout

Broadcast timeout period. Set to -1 to disable the timeout limit.

int

spark.sql.files.maxPartitionBytes

The maximum file size for each partition (Spark Partition), used for splitting large files.

268435456

int

spark.sql.files.openCostInBytes

Files smaller than this value will be merged. This is used for merging small files.

8388608

int