Docs

Big Data File Storage

Best practices

Configure MapReduce HDFS service to use CloudFS

VolcanoEngine E-MapReduce supports integration and management of Hadoop, Spark, Hive, Flink, and other ecosystem components. This topic describes how to configure the MapReduce HDFS service to use CloudFS.

Prerequisites

Before configuring MapReduce to use CloudFS, ensure the following is prepared:

CloudFS service is enabled; a file storage instance is created; and mount information is obtained. For detailed actions, see Create file storage instance.

Enable the E-MapReduce service and create a cluster. For detailed actions, see E-MapReduce cluster creation.

MapReduce data migration is complete.

Step 1: Configure HDFS service for MapReduce

tip

Modify the following settings on all cluster nodes.

Run the following command to download the CloudfsSDK package version 1.5.3.49.

wget https://cloudfs.tos-cn-beijing.volces.com/sdk/prod/cloudfs-client-1.5.3.49.tar.gz

Run the following command to unzip the CloudfsSDK rar.

tar -zxvf cloudfs-client-1.5.3.49.tar.gz

Copy the extracted file cloudfs-client/lib/cloudfs-sdk-1.5.3.49.jar to the cluster directory /share/hadoop/hdfs.

cp {YOUR_DOWNLOAD_PATH}/cloudfs-client/lib/cloudfs-sdk-1.5.3.49.jar /opt/{Directory}/hadoop/share/hadoop/hdfs/

Configure core-site.xml.

<property>
<name>fs.cfs.impl</name>
<value>com.volcengine.cloudfs.fs.CfsFileSystem</value>
</property>

<property>
<name>fs.AbstractFileSystem.cfs.impl</name>
<value>com.volcengine.cloudfs.fs.CFS</value>
</property>

<property>
<name>cfs.access.key</name>
<value>AKxxxxxxxxxxx</value>
</property>
<property>
<name>cfs.secret.key</name>
<value>SKxxxxxxxxxxx</value>
</property>

.
<property>
<name>cfs.security.token</name>
<value>STSTokenxxxxxxxx</value>
</property>


<property>
<name>cfs.client.network.segment</name>
<value><VPC subnet, for example, 192.168.0.0/16></value>
</property>

tip

The address xxxx.cfs-cn-beijing.ivolces.com is the mount target for cloud file storage. Replace it with the complete mount target address from your file storage instance. Replace this based on your specific business needs.

AKxxx and SKxxx are the access key ID and secret access key, respectively. For details, refer to Key management.

On the cluster management page, locate the target E-MapReduce cluster where you want to mount the file storage instance.

Select Service list > HDFS.

In the Actions panel on the right side of the page, click Restart > Restart All Components to restart the service.

Step 2: Use MapReduce to process CloudFS data

Use the built-in test package hadoop-mapreduce-examples-*.jar in E-MapReduce Hadoop to test. This test package is located by default in the directory /opt/{YOUR_PATH}/hadoop/2.x.x-1.x.x/package/hadoop-2.x.x-1.x.x/share/hadoop/mapreduce/.

Run the following command to generate and view target data in the /mr/input directory. For example:

hadoop jar /opt/{YOUR_PATH}/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar \
teragen 100000 cfs://{YOUR_ENDPOINT}/mr/input

hadoop jar /opt/tiger/current/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar \
terasort cfs://{YOUR_ENDPOINT}/mr/input cfs://{YOUR_ENDPOINT}/mr/output

tip

Replace {YOUR_ENDPOINT} with the mount target domain name of your file storage instance (cfs://xxxx.cfs-cn-beijing.ivolces.com).

Last updated: 2026.02.02 11:22:42