Docs

Elastic File Storage

Best Practices

Migrating data between FileNAS and TOS using tosutil

Overview

tosutil is an official command-line tool that simplifies the transfer of data across storage systems. Use tosutil to efficiently transfer data between FileNAS and Torch Object Storage (TOS).

Installing and initializing tosutil

tosutil is compatible with Windows, Linux, and macOS. Download and install the version that is appropriate for your environment.

Before importing or exporting data, install and initialize tosutil. For instructions, see Downloading and installing tosutil and Initialize tosutil.

Command syntax

General format

Linux/macOS

Windows

./tosutil [command] [args...] [options...]

Parameter	Description
[command]	Stands for the name of the command to run, for example, ls, cp and rm. For a list of tosutil commands, see Common commands.
[args...]	Stands for the basic parameters of commands, which are usually mandatory. When specifying multiple parameters, use spaces to separate them. For a list of tosutil basic parameters, see Common arguments.
[options...]	Stands for the optional parameters of commands. When specifying multiple parameters, use - as the prefix and = to indicate the parameter value, for example, -limit=100. For a list of tosutil optional parameters, see Global options. warning For Boolean options, use only - (without =) to indicate that the value is true.

Common commands

The following is a list of commonly used tosutil commands.

Category	Command	Description
Common commands	ls	Lists buckets, objects, or in-progress multipart exports.
		mkdir	Creates a folder.
		du	Calculates the size and number of objects and parts within the bucket.
		mb	Creates a new bucket using the provided bucket name and parameters.
		cp	Exports files or folders.
		setmeta	Sets object metadata.
		stat	Queries bucket or object properties.
		rm	Deletes buckets or objects, or aborts multipart exports.
		share	Creates authorization codes for sharing objects. These authorization codes can be used to list or download the objects.
		set-acl	Sets the predefined access control lists (ACLs) for buckets or objects.
		mount	Mounts a bucket as a local file system directory.
Auxiliary commands	config	Returns a configuration file's path and updates key configurations of the file.
		help	Returns a command's help information.
		probe	Tests export speed.
		netdig	Identifies issues with a single click. warning This command only works on Linux and macOS.
		hash	Computes hash values for files.
		fcp	Copies local files or folders.
		clear	Deletes checkpoint files in the specified folder and removes unnecessary files.
		version	Queries the version number.
		ping	Measures network latency.
		connect	Calculates the time it takes to establish a network connection.
		traceroute	Traces the path that an Internet Protocol (IP) packet takes to its destination.
		curl	Sends HTTP requests.

Common arguments

Argument	Description
cloud_url	Specifies the path on TOS. Paths are formatted as tos://bucket/[prefix], where the prefix is optional and indicates the object name or prefix of the object name.
file_url	Specifies the path for a file.
folder_url	Specifies the path for a folder.

Global options

All commands, except for some special auxiliary commands, can include optional parameters. These parameters are described as follows:

Option	Description
-e	Specifies the endpoint of the TOS server that will run the command.
-re	Specifies the region of the TOS server that will run the command.
-i	Specifies the access key ID used to run the command. tip Since access key IDs are not required for anonymous access, set the parameter to -i=.
-k	Specifies the secret access key used to run the command. tip Since secret access keys are not required for anonymous access, set the parameter to -k=.
-t	Specifies the security token used to run the command. Security tokens are only required when using temporary access credentials. tip Since security tokens are not required for anonymous access, set the parameter to -t=.
-conf	Specifies the path of the configuration file used to run the command.

warning

When both the -conf and -e/-re/-i/-k/-t parameters are set for a command, the -conf parameter remains effective while the access key ID, secret access key, and other relevant parameters in the configuration file under the specified path become ineffective. The -e/-re/-i/-k/-t parameters specified when running the command take priority.

Importing or exporting data

You can use the cp command to export files from FileNAS to TOS or import objects from TOS to FileNAS.

Command syntax

Linux/macOS

Windows

Exporting a file

./tosutil cp file_url tos://bucket[/key] [-dryRun] [-link] [-u] [-vchecksum] [-p=1] [-threshold=52428800] [-ps=auto] [-cpd=xxx] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-sse=xxx] [-sseKeyId=xxx] [-contentType=xxx] [-contentDisposition=xxx] [-contentEncoding=xxx] [-contentLanguage=xxx] [-cacheControl=xxx] [-expires=xxx] [-arcDir=xxx] [-fr] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Importing a file

./tosutil cp tos://bucket/key file_url|folder_url [-dryRun] [-tempFileDir=xxx] [-u] [-vchecksum] [-p=1] [-threshold=104857600] [-ps=auto] [-cpd=xxx] [-versionId=xxx] [-fr] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Exporting a folder

./tosutil cp folder_url tos://bucket[/prefix] -r [-dryRun] [-link] [-u] [-vchecksum] [-flat] [-j=1] [-p=1] [-threshold=104857600] [-nfj=1] [-ps=auto] [-cpd=xxx] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-sse=xxx] [-sseKeyId=xxx] [-contentType=xxx] [-contentDisposition=xxx] [-contentEncoding=xxx] [-contentLanguage=xxx] [-cacheControl=xxx] [-expires=xxx] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-at] [-ddo] [-dmf] [-arcDir=xxx] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Exporting a list of files

./tosutil cp file1_url,folder1_url|filelist_url tos://bucket[/prefix] -msm=1 [-r] [-dryRun] [-link] [-u] [-vchecksum] [-flat] [-j=1] [-p=1] [-threshold=104857600] [-nfj=1] [-ps=auto] [-cpd=xxx] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-sse=xxx] [-sseKeyId=xxx] [-contentType=xxx] [-contentDisposition=xxx] [-contentEncoding=xxx] [-contentLanguage=xxx] [-cacheControl=xxx] [-expires=xxx] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-at] [-ddo] [-dmf] [-arcDir=xxx] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Importing files in a batch

./tosutil cp tos://bucket[/prefix] folder_url -r [-dryRun] [-tempFileDir=xxx] [-u] [-vchecksum] [-flat] [-j=1] [-p=1] [-threshold=104857600] [-nfj=1] [-ps=auto] [-cpd=xxx] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-dmf] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Exporting a file

tosutil cp file_url tos://bucket[/key] [-dryRun] [-link] [-u] [-vchecksum] [-p=1] [-threshold=52428800] [-ps=auto] [-cpd=xxx] [-acl=xxx] [-meta=aaa:bbb#ccc:ddd] [-sse=xxx] [-sseKeyId=xxx] [-contentType=xxx] [-contentDisposition=xxx] [-contentEncoding=xxx] [-contentLanguage=xxx] [-cacheControl=xxx] [-expires=xxx] [-arcDir=xxx] [-fr] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Importing a file

tosutil cp tos://bucket/key file_url|folder_url [-dryRun] [-tempFileDir=xxx] [-u] [-vchecksum] [-p=1] [-threshold=104857600] [-ps=auto] [-cpd=xxx] [-versionId=xxx] [-fr] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Exporting a folder

tosutil cp folder_url tos://bucket[/prefix] -r [-dryRun] [-link] [-u] [-vchecksum] [-flat] [-j=1] [-p=1] [-threshold=104857600] [-nfj=1] [-ps=auto] [-cpd=xxx] [-acl=xxx] [-meta=aaa:bbb#ccc:ddd] [-sse=xxx] [-sseKeyId=xxx] [-contentType=xxx] [-contentDisposition=xxx] [-contentEncoding=xxx] [-contentLanguage=xxx] [-cacheControl=xxx] [-expires=xxx] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-at] [-ddo] [-dmf] [-arcDir=xxx] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Exporting a list of files

tosutil cp file1_url,folder1_url|filelist_url tos://bucket[/prefix] -msm=1 [-r] [-dryRun] [-link] [-u] [-vchecksum] [-flat] [-j=1] [-p=1] [-threshold=104857600] [-nfj=1] [-ps=auto] [-cpd=xxx] [-acl=xxx] [-meta=aaa:bbb#ccc:ddd] [-sse=xxx] [-sseKeyId=xxx] [-contentType=xxx] [-contentDisposition=xxx] [-contentEncoding=xxx] [-contentLanguage=xxx] [-cacheControl=xxx] [-expires=xxx] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-at] [-ddo] [-dmf] [-arcDir=xxx] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Importing files in a batch

tosutil cp tos://bucket[/prefix] folder_url -r [-dryRun] [-tempFileDir=xxx] [-u] [-vchecksum] [-flat] [-j=1] [-p=1] [-threshold=104857600] [-nfj=1] [-ps=auto] [-cpd=xxx] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-dmf] [-o=xxx] [-bt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Exporting data

Exporting a file

Use the following command to export the file1 file from the /NAS/export folder to the bucketname bucket. The exported file will be named object.

Command

./tosutil cp /NAS/export/file1 tos://bucketname/object

Output

Start at 2022-10-27 10:49:12.407874 +0000 UTC

Parallel: 1 Jobs: 5
Threshold: 100.00MB PartSize: auto
VerifyChecksum: false
CheckpointDir: /.tosutil_checkpoint

[-------------------------------------------] 100.00% ?/s 50.50KB/50.50KB 162ms

Upload successfully, 50.50KB, n/a, /NAS/export/file1 --> tos://bucketname/object, cost [164], status [200], request id [2643485a6228acbb635a6228-ac1173a1-1oo0Rk-PuO-cb-tos-bj-3]

Exporting a folder

Use the following command to export all files in the /NAS/export folder to the bucketname bucket. The exported folder will be named cpfile.

tip

This command exports the folder while preserving the directory structure. All exported files reside within the cpfile folder.

Command

./tosutil cp /NAS/export tos://bucketname/cpfile -r

Output

Start at 2022-10-27 10:50:43.872787 +0000 UTC

Parallel: 1 Jobs: 5
Threshold: 100.00MB PartSize: auto
VerifyChecksum: false
CheckpointDir: /Users/.tosutil_checkpoint
OutputDir: /Users/.tosutil_output

[----------------------------] 100.00% tps:0.00 ?/s 4/4 181.45KB/181.45KB 183ms
Succeed count is: 4 Failed count is: 0
Succeed bytes is: 181.45KB
Metrics [max cost:178 ms, min cost:103 ms, average cost:143.75 ms, average tps:21.62]

Task id is: 34c8480e-55e5-11ed-806b-aa665a3c7f26

Importing data

Importing a file

Use the following command to import the object1.png object from the bucketname bucket to the /NAS/import folder.

Command

./tosutil cp tos://bucketname/object1.png /NAS/import

Output

Start at 2022-10-27 10:56:20.611282 +0000 UTC

Parallel: 1 Jobs: 5
Threshold: 100.00MB PartSize: auto
VerifyChecksum: false
CheckpointDir: /.tosutil_checkpoint
TempFileDir:

[-------------------------------------------] 100.00% ?/s 55.98KB/55.98KB 106ms

Download successfully, 55.98KB, n/a, tos://bucketname/object1.png --> /Downloads/TOS/tosutil/object1.png, cost [106], status [200], request id [5c81485a63d4b1d7635a63d4-ac17b21f-1oo0Ye-GO-cb-tos-bj]

Importing files in a batch

Use the following command to import objects with the object prefix from the bucketname bucket to the /NAS/import folder.

Command

./tosutil cp tos://bucketname/object /NAS/import -r

Output

Start at 2022-10-27 10:56:35.434563 +0000 UTC

Parallel: 1 Jobs: 5
Threshold: 100.00MB PartSize: auto
VerifyChecksum: false
CheckpointDir: /Users/.tosutil_checkpoint
OutputDir: /Users/.tosutil_output
TempFileDir:

[----------------------] 100.00% tps:51.09 3.02MB/s 4/4 238.21KB/238.21KB 278ms
Succeed count is: 4 Failed count is: 0
Succeed bytes is: 238.21KB
Metrics [max cost:216 ms, min cost:163 ms, average cost:191.50 ms, average tps:14.23]

Task id is: 0653482e-55e6-11ed-afb8-aa665a3c7f26

Parameters

For descriptions of common optional parameters, see Global options.

Parameter	Type	Description
file_url	String	The path for a file to be imported or exported. tip If you need to import or export a list of files, add multiple file_url values to the command and separate them with commas (,).
folder_url	String	The path for the folder you want to import or export. This parameter must be used in conjunction with the -r parameter.
bucket	String	Bucket name.
key	String	Object name.
prefix	String	The prefix used for filtering objects in batch operations.
-r	Bool	Perform batch import, export, or copy operations.
-dryRun	Bool	Run in test mode without executing any actual import, export, or copy operations.
-link	Bool	Export the actual target path pointed to by symlinks for both files and directories, rather than exporting the symlink files themselves.
-msm	Enum	Sets the mode for exporting multiple files or folders at once. The options are as follows: -msm=1: The export path is a list of files or folders separated by commas (,). -msm=2: The export path is a manifest file containing a list of files or folders. warning Do not use the -msm=1 mode if a file or folder name contains a comma. Unless the -r parameter is set, folders in the list will not be exported.
-tempFileDir	String	The folder where temporary files generated during multipart imports are stored. The default location is specified in the configuration file as defaultTempFileDir. tip If the -tempFileDir parameter is left blank, the temporary files will be stored in the same directory as the objects to be imported, with names ending in .tos.temp. Since these temporary files will be saved in this directory, ensure that the user running tosutil has write permissions to this path. The available space in this directory must exceed the size of the objects to be imported.
-recover	String	The ID of a file import, export, or copy task to be resumed in the result file.
-u	Bool	This parameter is used for incremental imports, exports, or copies of files. It compares the source and destination. We recommend using this setting in the following cases: The destination does not exist. The file size at the destination differs from that at the source. The last modified time at the destination is earlier than that at the source.
-vchecksum	Bool	Validate data integrity during import, export, or copy operations using the CRC-64 algorithm.
-flat	Bool	Do not include the relative parent directory when importing, exporting, or copying files.
-j	Integer	The maximum concurrency for batch tasks. This value is set to 3 by default, which is the defaultJobs value in the configuration file.
-p	Integer	The maximum concurrency for each multipart task. This value is set to 1 by default, which is the defaultParallels value in the configuration file.
-threshold	Integer	The limit, in bytes, for a multipart task. This value is set to the defaultBigfileThreshold value in the configuration file by default. tip This parameter supports configuration with capacity units. For example, there is no need to manually calculate how many bytes are in 1 MB — just write "1 MB" and the program will recognize it.
-nfj	Integer	The maximum concurrency for batch processing of regular files or objects (non-multipart tasks). Setting this parameter divides the concurrency specified by the -j parameter into two parts: one for non-multipart tasks (nfj) and one for multipart tasks (j-nfj). This parameter only takes effect during batch operations.
-ps	Integer/String	The size, in bytes, of each part during a multipart task. This value is set to the defaultPartSize value in the configuration file by default. tip This parameter supports configuration with capacity units. For example, there is no need to manually calculate how many bytes are in 1 MB — just write "1 MB" and the program will recognize it.
-cpd	String	The folder where checkpoint files are stored. The default location is specified in the configuration file as defaultCheckpointDir.
-versionId	String	The version ID of an object to be imported or copied.
-acl	String	Sets the predefined access control lists (ACLs) for files to be exported or copied. Possible values: private public-read public-read-write authenticated-read bucket-owner-read bucket-owner-full-control bucket-owner-entrusted
-sc	Enum	Sets the storage type of files to be exported. The options are as follows: STANDARD: Standard Storage, which is the default value. IA: Infrequent Access Storage. ARCHIVE_FR: Flashback Archive. ARCHIVE: Archive. COLD_ARCHIVE: Cold Archive. DEEP_COLD_ARCHIVE: Deep Cold Archive. tip Cold Archive is generally available in China (Beijing) and China (Shanghai) and is in closed beta in China (Guangzhou). If you would like to use Cold Archive in China (Guangzhou), please contact your account manager for access. Deep Cold Archive is currently in closed beta in China (Beijing), China (Shanghai), and China (Guangzhou). Please contact your account manager for access.
-meta	String	Sets the user-defined metadata of files to be exported or copied.
-sse	String	Sets the server-side encryption algorithm used when exporting or copying files. Possible values: aes256 kms This value is set to the defaultSse value in the configuration file by default.
-sseKeyId	String	The key used when exporting or copying files with server-side encryption with Key Management Service (KMS). This value is set to the defaultSseKeyId value in the configuration file by default.
-contentType	String	Sets the Content-Type metadata for files during export or copy operations.
-contentDisposition	String	Sets the Content-Disposition metadata for files during export or copy operations.
-contentEncoding	String	Sets the Content-Encoding metadata for files during export or copy operations.
-contentLanguage	String	Sets the Content-Language metadata for files during export or copy operations.
-cacheControl	String	Sets the Cache-Control metadata for files during export or copy operations.
-expires	String	Sets the Expires metadata for files during export or copy operations, in the format of YYYYMMDDHHmmSS.
-include	String	The file matching pattern used to include files for import, export, or copying. The following characters have special behavior: ?: Matches any single character within a filename. : Matches zero or more characters within a filename. #: Serves as a delimiter. For example, -include=.png#*.txt matches all files ending in .png and .txt. tip To avoid parsing failures caused by the operating system escaping special characters, enclose the file matching pattern in quotes. This pattern only takes effect when exporting folders or a list of files. This pattern applies to the full file path, including the filename and directory structure.
-exclude	String	The file matching pattern used to exclude files from import, export, or copy operations. The following characters have special behavior: ?: Matches any single character within a filename. : Matches zero or more characters within a filename. #: Serves as a delimiter. For example, -exclude=.png#*.txt matches all files ending in .png and .txt. tip To avoid parsing failures caused by the operating system escaping special characters, enclose the file matching pattern in quotes. This pattern only takes effect when exporting folders or a list of files. This pattern applies to the full file path, including the filename and directory structure.
-timeRange	String	Sets the time range for the last modified time of files during import, export, or copy, in the format of YYYYMMDDHHmmSS-YYYYMMDDHHmmSS.
-at	Bool	Export a list of files whose last access time falls within the -timeRange.
-ddo	Bool	Do not export the folder itself as an object.
-dmf	Bool	Do not apply the parameters, -include/-exclude and -timeRange, to folders.
-marker	String	Sets the starting position for batch importing.
-arcDir	String	The archive folder for successfully exported files.
-fr	Bool	Generate a result file when working with a single file.
-o	String	The folder where result files are stored. The default location is specified in the configuration file as defaultOutputDir.
-bt	String	The type of the bucket. The options are as follows: fns: A flat namespace (FNS) bucket. hns: A hierarchical namespace (HNS) bucket. If the bucket type is not specified, the default will be the type obtained from the bucket metadata.

Recommendations for optimizing performance

tosutil supports adjusting concurrency via the -j and -p parameters. When you import or export large files in batches, the actual concurrency equals the -j value multiplied by the -p value. Recommendations:

When resources are limited

To prevent performance degradation or failure due to resource exhaustion when resources like network bandwidth or memory are limited, do not set the concurrency level too high. Start with a lower concurrency value and gradually increase it until you find the optimal setting for your resources.

Improving overall QPS and bandwidth

When exporting small files, set a higher concurrency to improve QPS.

When exporting large files, avoid excessively high concurrency to prevent bandwidth contention and performance degradation.

Use the -nfj and -j parameters to manage the concurrency of non-multipart and multipart tasks separately. For example, setting -j=100, -nfj=80 results in an overall concurrency of 100. In this case, the maximum concurrency for non-multipart tasks (typically smaller files) is 80, and the maximum concurrency for multipart tasks (typically large files) is 20.

When using public networks

Exports may be slow or prone to failure due to unstable public networks and limited client bandwidth. Adjust the following parameters:

Increase the maxRetryCount value in the configuration file to address instability through retries.

If the client's bandwidth is limited, adjust the -j and -p parameters accordingly to prevent exhaustion of the bandwidth and connection failures.

Reduce part size by adjusting the -ps parameter to improve the success rate of individual copies.

warning

In scenarios involving a large number of small files, setting a high -j value can result in a TPS greater than 1,000. In this case, increase the maxConnections parameter accordingly. For more information, see tosutil configuration parameters.

Last updated: 2026.06.01 11:55:09