Docs

File Storage vePFS

Best Practices

Back up data from vePFS to object storage on a regular schedule using the tosutil tool

This article describes how to use the tosutil tool to back up data from vePFS to TOS object storage.

Background information

vePFS provides data migration features. We recommend using these features for a better experience. For details, refer to Data Migration Overview.

Use cases

The core training data stored in vePFS is highly sensitive and requires regular backups to ensure data security.

Checkpoint data generated during training should be regularly synced to the object storage to promptly free up vePFS storage space.

Prerequisites

The bandwidth of both ECS and object storage meets backup requirements.

The network bandwidth for ECS instances varies depending on the specification. For details, see ECS specification description.

For object storage bandwidth constraints, refer to Limitations.

A storage bucket has already been created. For step-by-step instructions, see Create a storage bucket.

File storage vePFS has already been mounted on ECS. For step-by-step instructions, see Mount vePFS.

The tosutil tool has already been installed and initialized on ECS. For step-by-step instructions, see Install tosutil and Initialize tosutil.

Steps

Step 1: Set up scheduled backups for vePFS data

Run the following command to create directories for storing backup scripts and log files. This example creates the backp directory.

cd /opt
mkdir backup

Run the following command to create a backup script. for example, backup_vepfs_to_tos.sh , and open it with a text editor.

vim backup_vepfs_to_tos.sh

Add backup information to the script.

tip

This backup command performs incremental backups, copying only new and modified files in vePFS to object storage.

#!/bin/bash

# Specify the source and target paths for backup. Adjust them as needed for your actual business scenario.
source_path="/mnt/vepfs/backup/files"
bucket_name="your-tos-bucket"
log_file="/opt/backup/backup.log"

# Backup parameter command
jobs=32
parallel=4

# Backup command. Records start and end times, and outputs tosutil logs to the log file.
start_time=$(date +%Y-%m-%d_%H:%M:%S)
echo "Start vepfs backup to tos: $start_time" >> $log_file
tosutil cp -r -u -j $jobs -p $parallel $source_path tos://$bucket_name >> $log_file 2>&1
end_time=$(date +%Y-%m-%d_%H:%M:%S)
echo "Finish vepfs backup to tos: $end_time" >> $log_file

Backup parameter details:

Parameter	Is modification required	Note
source_path	Yes	File path in vePFS to back up
bucket_name	Yes	Name or directory of the object storage bucket for data backup
log_file	Yes	Data backup log file
jobs	Not supported	The number of concurrent tasks for tosutil data synchronization
parallel	Not supported	Number of chunks for large file synchronization

Step 2: Configure scheduled tasks

Run the following command to grant execution permission to the backup script.

chmod +x backup_vepfs_to_tos.sh

Run the following command to create and edit a cron job.

crontab -e

Add the backup content to the cron table. For example, back up once daily at midnight. For detailed command configuration, see Appendix.

0 0 * * * /bin/bash /opt/backup_vepfs_to_tos.sh

Check the start and end time of the backup, as well as backup details, in the backup.log file.

tip

Backup details include: length of time, number of successfully backed up files, number of failed files, number of skipped files, and backup file size.

Appendix: crontab scheduled policy description

Cron format: * * * * * [your command]

The parameters are described below.

Location	Note	Example
The first column *	Indicates the execution minute, with a value range of 0~59.	5 * * * *: Runs every five minutes.
Second column*	Specifies the hour of execution. The valid range is 0~23.	0 * * * *: Runs hourly.
The third column *	Indicates the execution date, with a value range of 1~31.	0 0 * * *: Executes every day.
Fourth column *	Indicates the month of execution, with a value range of 1~12.	0 0 1 * *: Runs monthly.
The fifth column *	The execution week ranges from 0~7, where both 0 and 7 represent Sunday.	0 0 * * 0: Runs every week.
your command	The command to be executed.	0 0 1 1 *: Runs annually.

Here are some common examples.

Example	Note
30 8 * * *[your command]	Runs the command every morning at 08:30.
15 4 1,11,21 * *[your command]	This command runs at 04:15 AM on the 1st, 11th, and 21st day of every month.
10 0 * * 0,3 [your command]	Run the command at 00:10 every Sunday and Wednesday.
0,30 18-21 * * * [your command]	This command runs every 30 minutes between 18:00 and 21:00 each day.

Last updated: 2026.03.03 10:39:10