Docs

Big Data File Storage

Best practices

Configure Spark to access CloudFS

Spark is an open-source distributed computing framework designed for large-scale data analysis and processing. This article describes how to configure the Spark service in EMR to use CloudFS.

Prerequisites

Enable the big data file storage service, create a file storage instance, and obtain the mount target information. For details, see Cross-service authorization.

Complete cluster creation in E-MapReduce. For details, see E-MapReduce cluster creation.

Prepare a test file.

Step 1: Configure the CloudFS service

tip

All nodes in the cluster must modify the following configuration.

Connect to the E-MapReduce cluster using one of the following methods:

Use a local terminal and SSH to connect to the master node's public IP address.

Use a cloud server instance in the same region to connect to the master node's private IP address.

Run the following command to download the CloudfsSDK package version 1.5.3.49.

wget https://cloudfs.tos-cn-beijing.volces.com/sdk/prod/cloudfs-client-1.5.3.49.tar.gz

Run the following command to extract the CloudfsSDK package.

tar -zxvf cloudfs-client-1.5.3.49.tar.gz

Copy the extracted file cloudfs-client/lib/cloudfs-sdk-1.5.3.49.jar to the cluster path /share/hadoop/hdfs.

cp {YOUR_DOWNLOAD_PATH}/cloudfs-client/lib/cloudfs-sdk-1.5.3.49.jar /opt/{Directory}/hadoop/share/hadoop/hdfs/

Configure the core-site.xml file.

Run the following command to open the core-site.xml file in the Hadoop installation directory:

vim {hadoop_installation_directory}/hadoop/conf/core-site.xml

Add the following configuration to core-site.xml:

<property>
<name>fs.defaultFS</name>
<value>cfs://xxxx.cfs-cn-beijing.ivolces.com</value>

</property>

<property>
<name>fs.cfs.impl</name>
<value>com.volcengine.cloudfs.fs.CfsFileSystem</value>
</property>

<property>
<name>fs.AbstractFileSystem.cfs.impl</name>
<value>com.volcengine.cloudfs.fs.CFS</value>
</property>

<property>
<name>cfs.access.key</name>
<value>AKxxxxxxxxxxx</value>

</property>
<property>
<name>cfs.secret.key</name>
<value>SKxxxxxxxxxxx</value>

</property>


<property>
<name>cfs.security.token</name>
<value>STSTokenxxxxxxxx</value>
</property>


<property>
<name>cfs.client.network.segment</name>
<value><VPC subnet, for example 192.168.0.0/16></value>
</property>

Step 2: Configure Spark

Copy the cloudfs-client/lib/cloudfs-sdk-1.5.3.49.jar file to the Spark /{Directory}/spark/jars/ directory.

cp {YOUR_DOWNLOAD_PATH}/cloudfs-client/lib/cloudfs-sdk-1.5.3.49.jar /{Directory}/spark/jars/

Restart the Spark service.

On the cluster management page, locate the target E-MapReduce cluster.

Click Service list to find the Spark service.

Click the Restart button on the right side of the list page, and select Restart All components. After restarting, reconnect to the cluster.

Step 3: Verify the Spark configuration

There are two ways to use Spark to process data in CloudFS. The specific steps are as follows:

Use Spark-sql to process data in CloudFS.

The method for using Spark-sql to process data in CloudFS is similar to Hive SQL; when creating a table, simply specify the location as the CloudFS file system. For details, see Use Hive to process data in CloudFS.

Use Spark-shell to process data in CloudFS. The specific steps are as follows:

Go to the bin directory of Spark and run the following command to open the interface.

./spark-shell

If the following information is returned, the execution is successful.

Query data in CloudFS.

Read existing files in the CloudFS path.

val a =sc.textFile("cfs://{Directory}");

a.count

Write files to the specified CloudFS path.

a.saveAsTextFile("cfs://{Directory}")

Last updated: 2026.02.02 11:22:42