DolphinScheduler supports storing resource files in HDFS or object storage, and you can switch between them during cluster creation or operation according to business needs. The system uses /dolphinscheduler as the default resource directory. To switch the storage mode or modify the path, modify the relevant parameters in common.properties. This document introduces the best way to configure the resource path.

During cluster creation: DolphinScheduler's resource paths support two storage modes: HDFS and TOS.
During cluster operation: Switching to storage types officially supported by DolphinScheduler is supported, including HDFS, TOS, S3, MinIO, and OSS. For details, see DolphinScheduler resource center configuration details.

For detailed operations and parameter descriptions for cluster creation, see Create clusters.
The DolphinScheduler resource path configuration for the cluster supports two storage modes: HDFS and TOS.

• EMR-4.0.0 and later versions:
  • Default path = cluster storage root path +/dolphinscheduler. The cluster storage root path supports HDFS or TOS. To set it as an HDFS path, enable the HDFS service.
  • To specify a TOS path, follow the steps below Specify a TOS path.
• Versions earlier than EMR-4.0.0:
  • The HDFS path is used by default. When initializing and installing services for the cluster, the HDFS resource path for DolphinScheduler is configured automatically. Default path = resource.hdfs.fs.defaultFS + /dolphinscheduler.
  • To switch to a TOS path, follow the steps below Specify a TOS path.

Preparation #

Before configuring the path, you need to enable the TOS service and prepare a TOS bucket. For detailed steps, see Create bucket .

Configuration steps #

In Create cluster>Advanced settings>Custom parameters, pass the parameter resource.tos.full.upload.path of common.properties to DOLPHIN_SCHEDULER:

[
    {
        "serviceName": "DOLPHIN_SCHEDULER",
        "fileName": "common.properties",
        "key": "resource.tos.full.upload.path",
        "value": "tos://<your_tos_bucket>/<your_tos_dir>/dolphinscheduler"
    }
]

During cluster operation, you can modify the resource path of DolphinScheduler. All storage types officially supported by DolphinScheduler can be configured, including: HDFS, TOS, S3, MinIO (S3 compatible), and OSS.

In **Service list> DolphinScheduler >Service parameters> common.properties **, add the parameters as shown below. After you save the configuration, you must restart api-server and worker-server for the configuration to take effect.

HDFS #

# Resource storage types supported: HDFS, S3, OSS, NONE
resource.storage.type=HDFS
# The base path for resources in storage. Ensure that this directory exists and that the current user has read and write permissions. It is recommended to use the default /dolphinscheduler
resource.storage.upload.base.path=/dolphinscheduler
# User for accessing HDFS; must have permission to create directories under the HDFS root path
resource.hdfs.root.user=hdfs
# Default file system address of HDFS
resource.hdfs.fs.defaultFS=hdfs://master-1-1xxx

TOS #

# Resource storage types: HDFS, S3, OSS, NONE (TOS is accessed via an HDFS-compatible interface; keep HDFS selected)
resource.storage.type=HDFS
# Base path for resources on storage. Ensure that this directory exists and that the current user has read and write permissions. It is recommended to use /dolphinscheduler
resource.storage.upload.base.path=/dolphinscheduler
# Access user; keep the default in TOS scenarios
resource.hdfs.root.user=hdfs
# Resource storage path, format: tos://${tos_bucket}
resource.hdfs.fs.defaultFS=tos://${tos_bucket}

# Endpoint of the Region where the TOS Bucket is located (using Beijing as an example; for the same Region, it is recommended to use the internal domain name, for cross-Region use the public domain name)
fs.tos.endpoint=tos-cn-beijing.volces.com
# Region where the TOS Bucket is located
fs.tos.region=cn-beijing
# AK/SK for accessing TOS
fs.tos.access-key-id=<your-access-key-id>
fs.tos.secret-access-key=<your-secret-access-key>

S3 #

# Resource storage types: HDFS, S3, OSS, NONE
resource.storage.type=S3
# The base path for resources in storage. Ensure that this directory exists and that the current user has read and write permissions. It is recommended to use the default /dolphinscheduler
resource.storage.upload.base.path=/dolphinscheduler

# AK/SK for accessing S3
resource.aws.access.key.id=<your-access-key-id>
resource.aws.secret.access.key=<your-secret-access-key>
# Region where the S3 Bucket is located
resource.aws.region=cn-north-1
# Name of the S3 Bucket for storing DolphinScheduler resources
resource.aws.s3.bucket.name=dolphinscheduler
# Endpoint of the S3 service (AWS China example; replace with the corresponding address for different regions)
resource.aws.s3.endpoint=https://s3.cn-north-1.amazonaws.com.cn

# Resource storage path, format: s3a://${bucket}
resource.hdfs.fs.defaultFS=s3a://dolphinscheduler

MinIO #

MinIO is compatible with the S3 protocol. When configuring, set resource.storage.type to S3 and point resource.aws.s3.endpoint to the MinIO service address.

# Resource storage type: MinIO is compatible with the S3 protocol; fill in S3 here
resource.storage.type=S3
# The base path for resources in storage. Ensure that this directory exists and that the current user has read and write permissions. It is recommended to use the default /dolphinscheduler
resource.storage.upload.base.path=/dolphinscheduler

# AK/SK for accessing MinIO (corresponds to MinIO's accessKey / secretKey)
resource.aws.access.key.id=<minio-access-key>
resource.aws.secret.access.key=<minio-secret-key>
# Region declared for MinIO deployment; for self-built clusters, usually fill in us-east-1
resource.aws.region=us-east-1
# Name of the Bucket for storing DolphinScheduler resources
resource.aws.s3.bucket.name=dolphinscheduler
# Access address of the MinIO service
resource.aws.s3.endpoint=http://<minio-host>:9000

# Resource storage path, format: s3a://${bucket}
resource.hdfs.fs.defaultFS=s3a://dolphinscheduler

OSS #

# Resource storage types: HDFS, S3, OSS, NONE
resource.storage.type=OSS
# The base path for resources in storage. Ensure that this directory exists and that the current user has read and write permissions. It is recommended to use the default /dolphinscheduler
resource.storage.upload.base.path=/dolphinscheduler

# AK/SK used to access Alibaba Cloud OSS
resource.alibaba.cloud.access.key.id=<your-access-key-id>
resource.alibaba.cloud.access.key.secret=<your-access-key-secret>
# Region where the OSS bucket is located
resource.alibaba.cloud.region=cn-hangzhou
# Name of the OSS bucket for storing DolphinScheduler resources
resource.alibaba.cloud.oss.bucket.name=dolphinscheduler
# Endpoint of the OSS service (using Hangzhou as an example; replace with the corresponding address for other regions)
resource.alibaba.cloud.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com