You can use the cp command to copy objects.

Notes

Copying objects directly between the parent and subdirectories is not supported.

Command syntax

Linux/macOS

Windows

Copy an object

./tosutil cp tos://srcbucket/srckey tos://dstbucket/[dstkey] [-dryRun] [-u] [-pipe] [-pe=xxx] [-p=1] [-threshold=104857600] [-ps=auto] [-cpd=xxx] [-versionId=xxx] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-sse=xxx] [-sseKeyId=xxx] [-contentType=xxx] [-contentDisposition=xxx] [-contentEncoding=xxx] [-contentLanguage=xxx] [-cacheControl=xxx] [-expires=xxx] [-fr] [-o=xxx] [-bt=xxx] [-sbt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Batch copy objects

./tosutil cp tos://srcbucket[/srcprefix] tos://dstbucket[/dstprefix] -r [-dryRun] [-u] [-pipe] [-pe=xxx] [-flat] [-j=1] [-p=1] [-threshold=104857600] [-nfj=1] [-ps=auto] [-cpd=xxx] [-acl=xxx] [-sc=xxx] [-meta=aaa:bbb#ccc:ddd] [-sse=xxx] [-sseKeyId=xxx] [-contentType=xxx] [-contentDisposition=xxx] [-contentEncoding=xxx] [-contentLanguage=xxx] [-cacheControl=xxx] [-expires=xxx] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-dmf] [-o=xxx] [-bt=xxx] [-sbt=xxx] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Recommendations for optimizing performance

tosutil supports adjusting concurrency via the -j and -p parameters. When you upload large objects in batches, the actual concurrency equals the -j value multiplied by the -p value. Recommendations:

When resources are limited

To prevent performance degradation or failure due to resource exhaustion when resources like network bandwidth or memory are limited, do not set the concurrency level too high. Start with a lower concurrency value and gradually increase it until you find the optimal setting for your resources.

Improving overall QPS and bandwidth

When uploading, downloading, or copying small objects, set a higher concurrency to improve QPS.

When uploading, downloading, or copying large objects, avoid excessively high concurrency to prevent bandwidth contention and performance degradation.

Use the -nfj and -j parameters to manage the concurrency of non-multipart and multipart tasks separately. For example, setting -j=100, -nfj=80 results in an overall concurrency of 100. In this case, the maximum concurrency for non-multipart tasks (typically smaller objects) is 80, and the maximum concurrency for multipart tasks (typically larger objects) is 20.

When using public networks

Uploads may be slow or prone to failure due to unstable public networks and limited client bandwidth. Adjust the following parameters:

Increase the maxRetryCount value in the configuration file to address instability through retries.

If the client's bandwidth is limited, adjust the -j and -p parameters accordingly to prevent exhaustion of the bandwidth and connection failures.

Reduce part size by adjusting the -ps parameter to improve the success rate of individual uploads, downloads, and copies.

Parameters

For descriptions of common optional parameters, see Global options.

Parameter	Type	Description
srcbucket	String	The name of the source bucket.
srckey	String	The name of the source object.
dstbucket	String	The name of the destination bucket.
dstkey	String	The name of the target object.
srcprefix	String	Sets the prefix used for filtering source objects when batch copying.
dstprefix	String	Sets the prefix for target objects that are copied in batches.
-r	Bool	Perform batch upload, download, or copy operations.
-dryRun	Bool	Run in test mode without executing any actual uploads, downloads, or copy operations.
-recover	String	The ID of an object upload, download, or copy task to be resumed in the result file.
-u	Bool	This parameter is used for incremental uploads, downloads, or copies of objects. It compares the source and destination. We recommend using this setting in the following cases: The destination does not exist. The file size at the destination differs from that at the source. The last modified time at the destination is earlier than that at the source.
-pipe	Bool	Use the pipeline streaming mode to copy objects.
-pe	String	The endpoint of the destination bucket specified when copying objects using the pipeline streaming mode.
-flat	Bool	Do not include relative parent directories when uploading, downloading, or copying objects.
-j	Integer	The maximum concurrency for batch operations. For example, setting -j=1 means that the maximum concurrency for batch operations is 1. This value is set to 3 by default, which is the defaultJobs value in the configuration file.
-p	Integer	The maximum concurrency for a multipart task. For example, setting -p=1 means that the maximum concurrency for a multipart task is 1. This value is set to 20 by default, which is the defaultParallels value in the configuration file.
-threshold	Integer	The limit, in bytes, for a multipart task. This value is set to the defaultBigfileThreshold value in the configuration file by default. tip This parameter supports configuration with capacity units. For example, there is no need to manually calculate how many bytes are in 1 MB—just write "1 MB," and the program will recognize it.
-nfj	Integer	The maximum concurrency for regular object batch processing (non-multipart tasks). Setting this parameter divides the concurrency specified by the -j parameter into two parts: one for non-multipart tasks (nfj) and one for multipart tasks (j-nfj). This parameter only takes effect during batch operations.
-ps	Integer/String	The size, in bytes, of each part during a multipart task. This value is set to the defaultPartSize value in the configuration file by default. tip This parameter supports configuration with capacity units. For example, there is no need to manually calculate how many bytes are in 1 MB—just write "1 MB," and the program will recognize it.
-cpd	String	The folder where checkpoint files are stored. The default location is specified in the configuration file as defaultCheckpointDir.
-versionId	String	The version ID of an object to be downloaded or copied.
-acl	String	Sets the predefined access control lists (ACLs) for objects to be uploaded or copied. Possible values: private public-read public-read-write authenticated-read bucket-owner-read bucket-owner-full-control bucket-owner-entrusted
-sc	Enum	Sets the storage type of objects to be copied. The options are as follows: STANDARD: Standard Storage, which is the default value. IA: Infrequent Access Storage. ARCHIVE_FR: Flashback Archive. ARCHIVE: Archive. COLD_ARCHIVE: Cold Archive. DEEP_COLD_ARCHIVE: Deep Cold Archive. tip Cold Archive is only supported in China (Beijing), China (Shanghai), and China (Guangzhou). Deep Cold Archive is only supported in China (Beijing), China (Shanghai), and China (Guangzhou). Deep Cold Archive is currently in closed beta. Please contact your account manager for access.
-meta	String	Sets the user-defined metadata of objects to be uploaded or copied.
-sse	String	Sets the server-side encryption algorithm used when uploading or copying objects. Possible values: aes256 kms This value is set to the defaultSse value in the configuration file by default.
-sseKeyId	String	The key used when uploading or copying objects with server-side encryption with Key Management Service (KMS). This value is set to the defaultSseKeyId value in the configuration file by default.
-contentType	String	Sets the Content-Type metadata for objects during upload or copy operations.
-contentDisposition	String	Sets the Content-Disposition metadata for objects during upload or copy operations.
-contentEncoding	String	Sets the Content-Encoding metadata for objects during upload or copy operations.
-contentLanguage	String	Sets the Content-Language metadata for objects during upload or copy operations.
-cacheControl	String	Sets the Cache-Control metadata for objects during upload or copy operations.
-expires	String	Sets the Expires metadata for objects during upload or copy operations, in the format of YYYYMMDDHHmmSS.
-include	String	The object matching pattern used to include objects for upload, download, or copying. The following characters have special behavior: ? : Matches any single character within an object name. : Matches zero or more characters within an object name. #: Serves as a delimiter. For example, -include=.png#*.txt matches all objects ending in .png and .txt. tip To avoid parsing failures caused by the operating system escaping special characters, enclose the object matching pattern in quotes. This pattern only takes effect when uploading folders or a list of objects. This pattern applies to the full file path, including the object name and directory structure.
-exclude	String	The object matching pattern used to exclude objects from upload, download, or copy operations. The following characters have special behavior: ? : Matches any single character within an object name. : Matches zero or more characters within an object name. #: Serves as a delimiter. For example, -exclude=.png#*.txt matches all objects ending in .png and .txt. tip To avoid parsing failures caused by the operating system escaping special characters, enclose the object matching pattern in quotes. This pattern only takes effect when uploading folders or a list of objects. This pattern applies to the full file path, including the object name and directory structure.
-timeRange	String	Sets the time range for the last modified time of objects during upload, download, or copy, in the format of YYYYMMDDHHmmSS-YYYYMMDDHHmmSS.
-dmf	Bool	Do not apply the parameters, -include/-exclude and -timeRange, to folders.
-marker	String	Sets the starting position for batch copying.
-fr	Bool	Generate a result file when working with a single object.
-o	String	The folder where result files are stored. The default location is specified in the configuration file as defaultOutputDir.
-bt	String	The type of the destination bucket. The options are as follows: fns: A flat namespace (FNS) bucket. hns: A hierarchical namespace (HNS) bucket. If the bucket type is not specified, the default will be the type obtained from the bucket metadata.
-sbt	String	The type of the source bucket. The options are as follows: fns: A flat namespace (FNS) bucket. hns: A hierarchical namespace (HNS) bucket. If the bucket type is not specified, the default will be the type obtained from the bucket metadata.
-bfs	Integer	Uses the hierarchical listing method to batch copy objects and sets the concurrency level for those listings.

Examples

Copy an object

Use the following command to copy the object1.png object from the bucketname bucket to the tosutil bucket. The copied object will be named destobject.png.

Command

./tosutil cp tos://bucketname/object1.png tos://tosutil/destobject.png

Output

Start at 2022-10-27 10:57:24.353686 +0000 UTC

Parallel: 1 Jobs: 5
Threshold: 100.00MB PartSize: auto
CheckpointDir: /.tosutil_checkpoint

Copy successfully, 55.98KB, tos://bucketname/object1.png --> tos://tosutil/destobject.png, cost [154], status [200], request id [55fa485a6414248e635a6414-ac13a11a-1oo0Zg-CO-cb-tos-bj]

Batch copy objects

Use the following command to copy objects with the object prefix from the bucketname bucket to the tosutil bucket. The prefix of the copied objects will be destobject.

Command

./tosutil cp tos://bucketname/object tos://tosutil/destobject -r

Output

Start at 2022-10-27 10:57:43.32094 +0000 UTC

Parallel: 1 Jobs: 5
Threshold: 100.00MB PartSize: auto
CheckpointDir: /Users/.tosutil_checkpoint
OutputDir: /Users/.tosutil_output

[-------------------------------------------------] 100.00% tps:23.37 4/4 371ms
Succeed count is: 4 Failed count is: 0
Metrics [max cost:330 ms, min cost:92 ms, average cost:186.25 ms, average tps:10.75]

Task id is: 2ee1abfa-55e6-11ed-b269-aa665a3c7f26

Torch Object Storage