Docs

File Storage vePFS

Best Practices

Copy data between vePFS and object storage TOS using the tosutil tool

This article explains how to copy data between vePFS and object storage TOS using the tosutil tool.

Background information

vePFS provides data mobility features. For a better experience, we recommend using these features. For more details, see Data Mobility Overview.

Use cases

Dataset loading scenario (TOS > vePFS): Data moves from object storage to vePFS for data cleansing and GPU training.

Training data archiving scenario (vePFS > TOS): CheckPoint data and training results from GPU training are archived to object storage.

Prerequisites

The bandwidth of ECS and object storage meets the data transfer requirements.

Network bandwidth varies by ECS instance type. We recommend selecting ECS instances with outbound bandwidth of 20 Gbps or higher. For ECS specifications, see ECS Specification Description.

For bandwidth constraints of object storage, see Constraints.

A bucket has been created. For more information, see Creating buckets.

vePFS file storage has been mounted on ECS. For more information, see Mounting vePFS.

tosutil tool has been installed and initialized on ECS. For more information, see Install tosutil and Initialize tosutil.

The tmux tool has been installed on the ECS instance to prevent remote connection interruptions and avoid disrupting data synchronization. The installation command is as follows:

# Using the CentOS system as an example
yum install tmux -y

The commonly used tmux commands are as follows.

In the newly created tmux window, sessioname is a variable, your custom session name.
# tmux new -s sessioname
Exit the tmux window
# tmux detach
View the tmux window
# tmux ls
Load the already created tmux window, where sessionname is the session name retrieved by "tmux ls".
# tmux a -t sessioname

Steps

Copy object storage data to vePFS

This article takes an ECS instance running CentOS as an example to show how to copy object storage data to vePFS.

Use the console to log in to a Linux instance.

Run the following command to switch to tmux, which helps prevent session interruption during migration.

tmux new -s target1

Run the following command to copy data from the object storage bucket to vePFS.

./tosutil cp tos://bucketname/sourcedir /mnt/vepfs/ -r -u -j=XX -p=XX -nfj=XX

The main parameters are explained as follows.

tip

For detailed information about copying objects with tosutil, see Copy Objects.

tosutil supports optimizing transfer performance by adjusting concurrency using the -j and -p parameters. For more information, see Copy Objects.

Parameter	Note
bucketname	Object storage Bucket name.
sourcedir	Name of the directory containing the data to copy.
-r	Indicates batch execution.
-j	Maximum concurrency for batch tasks, adjustable based on business needs.
-p	Maximum concurrency for each shard task; you can adjust it according to your business needs.
-u	Download the incremental dataset from object storage. After setting this parameter, each download compares object storage and vePFS data, only downloading new and updated files.
-nfj	The maximum concurrency for handling regular files or objects (non-multipart tasks) in batch jobs. This is effective only during batch operations.
-exclude	The pattern used to match file names to exclude files during upload or download, supporting ? and * as wildcards.
-flat	Do not include the relative parent directory when uploading or downloading objects.

View the results of loading the dataset. Example output is shown below.

The table below lists all possible response fields.

Parameter	Note
Succeed count is	The number of files successfully downloaded from the object storage dataset, including newly added and skipped existing files.
Failed count is	The number of files that failed to download from the object storage dataset.
Skip count is	The number of skipped files already present in vePFS.
Succeed bytes is	Capacity of files successfully downloaded from the object storage dataset.
Task id is	The task ID uploaded by tosutil can be used to check the list of files that were downloaded successfully or failed.

tip

When uploading or downloading object storage files in batches with tosutil, result files are generated after task completion to record successful and failed tasks. By default, these files are stored in the folder specified by the defaultOutputDir parameter in the configuration file, with the default path as /root/.tosutil_output. For more details, see tosutil Configuration File.

Copy vePFS data to object storage

This article uses an ECS instance running the CentOS system as an example to illustrate how to copy data from vePFS to object storage.

Use the console to log in to a Linux instance.

Run the following command to switch to tmux and keep your session active during migration.

tmux new -s target1

Run the following command to copy the data from vePFS to object storage.

./tosutil cp /mnt/vepfs/sourcedir tos://bucketname -r -u -j=XX -p=XX -nfj=XX

The main parameters are explained below.

tip

For detailed instructions on copying objects with tosutil, see Copying Objects.

tosutil supports adjusting concurrency using the -j and -p options to optimize transfer performance. For recommendations, see Copying Objects.

Parameter	Note
sourcedir	Directory name to be copied in vePFS.
bucketname	Name of the destination bucket in object storage.
-r	Indicates execution of batch actions.
-j	The maximum concurrency for batch jobs, adjustable based on business requirements.
-p	The maximum concurrency for each multipart task can be adjusted based on business requirements.
-u	Incrementally download the dataset from object storage. Once this parameter is set, each download will compare the files between object storage and vePFS, and only download new or updated files. warning Data deleted from vePFS will not be deleted from object storage.
-nfj	The maximum concurrency for processing regular files or objects (non-multipart tasks) during batch operations. This only applies to batch actions.
-exclude	Upload or download using name matching patterns that do not include files, supporting ? and * as wildcards.

View the data archiving results.

The returned parameters are explained below.

Parameter	Note
Succeed count is	The number of files successfully uploaded to the object storage dataset, including both new files and those skipped because they already exist.
Failed count is	Number of files that failed to upload to the object storage dataset.
Skip count is	The number of skipped files that already exist in object storage.
Succeed bytes is	The size of files successfully uploaded to the object storage dataset.
Task id is	The task ID for tosutil uploads. You can use this ID to check the list of files that were uploaded successfully or failed.

tip

When using the tosutil tool to batch upload or download object storage files, successful and failed task reports are generated after the tasks are completed. The default storage folder is specified by the defaultOutputDir parameter in the configuration file, with the default path being /root/.tosutil_output. For more details, see tosutil configuration file.

Last updated: 2026.03.03 10:39:10