Docs

Big Data File Storage

Best practices

Migrate Hadoop file system data to CloudFS

This article describes how to migrate IDC or self-built Hadoop file system data to CloudFS. After migration, you can manage and access data in CloudFS.

Prerequisites

Before migrating Hadoop file system data to CloudFS, ensure you have completed the following preparations:

Enable Cloud File Storage (CloudFS edition) and create a file system instance and mount target. For detailed instructions, refer to Create a file storage instance.

A Hadoop cluster built on Volcano Engine ECS (hereinafter referred to as the "migration cluster") is used to access Cloud File Storage CloudFS and migrate data, and meets the following conditions:

The migration cluster and the Cloud File Storage CloudFS instance are in the same region and the same Availability Zone.

The migration cluster and the mount target of Cloud File Storage CloudFS use the same Volcano Engine VPC network and switch.

The JDK version installed on the migration cluster is not lower than 1.8.

The Hadoop version installed on the migration cluster is not lower than 2.6.0.

tip

If the original cluster meets the above conditions and has sufficient computing resources, you can directly use the original cluster as the migration cluster without creating a new cluster.

Step 1: Configure the CloudFS service

tip

All nodes in the cluster must modify the following configuration.

Run the following command to download the CloudfsSDK package version 1.5.3.49.

wget https://cloudfs.tos-cn-beijing.volces.com/sdk/prod/cloudfs-client-1.5.3.49.tar.gz

Run the following command to extract the CloudfsSDK package.

tar -zxvf cloudfs-client-1.5.3.49.tar.gz

Copy the extracted file cloudfs-client/lib/cloudfs-sdk-1.5.3.49.jar to the cluster path /share/hadoop/hdfs.

cp {YOUR_DOWNLOAD_PATH}/cloudfs-client/lib/cloudfs-sdk-1.5.3.49.jar /opt/{Directory}/hadoop/share/hadoop/hdfs/

Configure core-site.xml.

<property>
<name>fs.defaultFS</name>
<value>cfs://xxxx.cfs-cn-beijing.ivolces.com</value>
</property>
<property>
<name>fs.cfs.impl</name>
<value>com.volcengine.cloudfs.fs.CfsFileSystem</value>
</property>

<property>
<name>fs.AbstractFileSystem.cfs.impl</name>
<value>com.volcengine.cloudfs.fs.CFS</value>
</property>

<property>
<name>cfs.access.key</name>
<value>AKxxxxxxxxxxx</value>
</property>
<property>
<name>cfs.secret.key</name>
<value>SKxxxxxxxxxxx</value>
</property>


<property>
<name>cfs.security.token</name>
<value>STSTokenxxxxxxxx</value>
</property>


<property>
<name>cfs.client.network.segment</name>
<value><VPC subnet, for example 192.168.0.0/16></value>
</property>

tip

xxxx.cfs-cn-beijing.ivolces.com is the mount target address of Cloud File Storage. Replace it with the mount target address of the created file storage system.

AKxxx and SKxxx are the access key ID and secret access key, respectively. For details, refer to Key management.

Step 2: Migrate Hadoop data to CloudFS

To migrate data directly from the original cluster to Cloud File Storage CloudFS, first connect the network environments of the original cluster and the migration cluster, then perform the data migration task.

Verify the connectivity between the migration cluster and the Cloud File Storage CloudFS file storage system.

Run the following command to create a directory (for example, /dfs_links) on Cloud File Storage CloudFS:

hdfs dfs -mkdir /dfs_links

Run the following command to verify connectivity.

hdfs dfs -ls cfs://xxxx.cfs-cn-beijing.ivolces.com/dfs_links

tip

xxxx.cfs-cn-beijing.ivolces.com is the mount target of the cloud file storage system. Please modify as needed.

Use the Hadoop Shell to migrate the cluster's HDFS data to CloudFS.

hdfs dfs -cp hdfs:///README.md cfs://{CFS_ENDPOINT}/

tip

{CFS_ENDPOINT} should be replaced with the mount target of your file storage system, for example, cfs://xxxx.cfs-cn-beijing.ivolces.com.

Last updated: 2026.02.02 11:22:43