You can use the cp command to download objects from a bucket.

Command syntax

Linux/macOS

Windows

Download an object

./tosutil cp tos://bucket/key file_url|folder_url [-dryRun] [-tempFileDir=xxx] [-u] [-vchecksum] [-p=1] [-threshold=104857600] [-ps=auto] [-cpd=xxx] [-versionId=xxx] [-fr] [-o=xxx] [-bt=xxx] [-nftr] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Batch download objects

./tosutil cp tos://bucket[/prefix] folder_url -r [-f] [-dryRun] [-tempFileDir=xxx] [-u] [-vchecksum] [-flat] [-j=1] [-p=1] [-threshold=104857600] [-nfj=1] [-ps=auto] [-cpd=xxx] [-include=*.xxx] [-exclude=*.xxx] [-timeRange=time1-time2] [-dmf] [-o=xxx] [-bt=xxx] [-nftr] [-e=xxx] [-re=xxx] [-i=xxx] [-k=xxx] [-t=xxx] [-conf=xxx]

Recommendations for optimizing performance

tosutil supports adjusting concurrency via the -j and -p parameters. When you upload large objects in batches, the actual concurrency equals the -j value multiplied by the -p value. Recommendations:

When resources are limited

To prevent performance degradation or failure due to resource exhaustion when resources like network bandwidth or memory are limited, do not set the concurrency level too high. Start with a lower concurrency value and gradually increase it until you find the optimal setting for your resources.

Improving overall QPS and bandwidth

When uploading, downloading, or copying small objects, set a higher concurrency to improve QPS.

When uploading, downloading, or copying large objects, avoid excessively high concurrency to prevent bandwidth contention and performance degradation.

Use the -nfj and -j parameters to manage the concurrency of non-multipart and multipart tasks separately. For example, setting -j=100, -nfj=80 results in an overall concurrency of 100. In this case, the maximum concurrency for non-multipart tasks (typically smaller objects) is 80, and the maximum concurrency for multipart tasks (typically larger objects) is 20.

When using public networks

Uploads may be slow or prone to failure due to unstable public networks and limited client bandwidth. Adjust the following parameters:

Increase the maxRetryCount value in the configuration file to address instability through retries.

If the client's bandwidth is limited, adjust the -j and -p parameters accordingly to prevent exhaustion of the bandwidth and connection failures.

Reduce part size by adjusting the -ps parameter to improve the success rate of individual uploads, downloads, and copies.

Parameters

For descriptions of common optional parameters, see Global options.

Parameter	Type	Description
file_url	String	The path for an object to be downloaded.
folder_url	String	The path for the local folder where downloaded objects will be stored.
bucket	String	Bucket name.
key	String	Object name.
prefix	String	The prefix used for filtering objects in batch operations.
-r	Bool	Perform batch upload, download, or copy operations.
-f	Bool	Force the operation. When downloading an object locally, the system will automatically overwrite an existing local file with the same name without prompting for confirmation.
-dryRun	Bool	Run in test mode without executing any actual uploads, downloads, or copy operations.
-tempFileDir	String	The folder where temporary files generated during multipart downloads are stored. The default location is specified in the configuration file as defaultTempFileDir. tip If the -tempFileDir parameter is left blank, the temporary files will be stored in the same directory as the objects to be downloaded, with filenames ending in .tos.temp. Since these temporary files will be saved in this directory, ensure that the user running tosutil has write permissions to this path. The available space in this directory must exceed the size of the objects to be downloaded.
-recover	String	The ID of an object upload, download, or copy task to be resumed in the result file.
-u	Bool	This parameter is used for incremental uploads, downloads, or copies of objects. It compares the source and destination. We recommend using this setting in the following cases: The destination does not exist. The file size at the destination differs from that at the source. The last modified time at the destination is earlier than that at the source.
-vchecksum	Bool	Validate data integrity during uploads or downloads using the CRC-64 algorithm.
-flat	Bool	Do not include relative parent directories when uploading, downloading, or copying objects.
-j	Integer	The maximum concurrency for batch operations. For example, setting -j=1 means that the maximum concurrency for batch operations is 1. This value is set to 3 by default, which is the defaultJobs value in the configuration file.
-p	Integer	The maximum concurrency for a multipart task. For example, setting -p=1 means that the maximum concurrency for a multipart task is 1. This value is set to 5 by default, which is the defaultDownloadParallels value in the configuration file.
-threshold	Integer	The limit, in bytes, for a multipart task. This value is set to the defaultBigfileThreshold value in the configuration file by default. tip This parameter supports configuration with capacity units. For example, there is no need to manually calculate how many bytes are in 1 MB—just write "1 MB," and the program will recognize it.
-nfj	Integer	The maximum concurrency for regular object batch processing (non-multipart tasks). Setting this parameter divides the concurrency specified by the -j parameter into two parts: one for non-multipart tasks (nfj) and one for multipart tasks (j-nfj). This parameter only takes effect during batch operations.
-ps	Integer/String	The size, in bytes, of each part during a multipart task. This value is set to the defaultPartSize value in the configuration file by default. tip This parameter supports configuration with capacity units. For example, there is no need to manually calculate how many bytes are in 1 MB—just write "1 MB," and the program will recognize it.
-cpd	String	The folder where checkpoint files are stored. The default location is specified in the configuration file as defaultCheckpointDir.
-versionId	String	The version ID of an object to be downloaded or copied.
-include	String	The object matching pattern used to include objects for upload, download, or copying. The following characters have special behavior: ? : Matches any single character within an object name. : Matches zero or more characters within an object name. #: Serves as a delimiter. For example, -include=.png#*.txt matches all objects ending in .png and .txt. tip To avoid parsing failures caused by the operating system escaping special characters, enclose the object matching pattern in quotes. This pattern only takes effect when uploading folders or a list of objects. This pattern applies to the full file path, including the object name and directory structure.
-exclude	String	The object matching pattern used to exclude objects from upload, download, or copy operations. The following characters have special behavior: ? : Matches any single character within an object name. : Matches zero or more characters within an object name. #: Serves as a delimiter. For example, -exclude=.png#*.txt matches all objects ending in .png and .txt. tip To avoid parsing failures caused by the operating system escaping special characters, enclose the object matching pattern in quotes. This pattern only takes effect when uploading folders or a list of objects. This pattern applies to the full file path, including the object name and directory structure.
-timeRange	String	Sets the time range for the last modified time of objects during upload, download, or copy, in the format of YYYYMMDDHHmmSS-YYYYMMDDHHmmSS.
-dmf	Bool	Do not apply the parameters, -include/-exclude and -timeRange, to folders.
-marker	String	Sets the starting position for batch downloads.
-fr	Bool	Generate a result file when working with a single object.
-o	String	The folder where result files are stored. The default location is specified in the configuration file as defaultOutputDir.
-bt	String	The type of the bucket. The options are as follows: fns: A flat namespace (FNS) bucket. hns: A hierarchical namespace (HNS) bucket. If the bucket type is not specified, the default will be the type obtained from the bucket metadata.
-bfs	Integer	Uses the hierarchical listing method to batch download objects and sets the concurrency level for those listings.
-nftr	Bool	Do not use the Truncate parameter to specify file size when creating a new file.

Examples

Download an object

Use the following command to download the object1.png object from the bucketname bucket to the /Users/Downloads/TOS/tosutil folder.

Command

./tosutil cp tos://bucketname/object1.png /Users/Downloads/TOS/tosutil

Output

Start at 2022-10-27 10:56:20.611282 +0000 UTC

Parallel: 1 Jobs: 5
Threshold: 100.00MB PartSize: auto
VerifyChecksum: false
CheckpointDir: /.tosutil_checkpoint
TempFileDir:

[-------------------------------------------] 100.00% ?/s 55.98KB/55.98KB 106ms

Download successfully, 55.98KB, n/a, tos://bucketname/object1.png --> /Downloads/TOS/tosutil/object1.png, cost [106], status [200], request id [5c81485a63d4b1d7635a63d4-ac17b21f-1oo0Ye-GO-cb-tos-bj]

Batch download objects

Use the following command to download objects with the object prefix from the bucketname bucket to the /Users/Downloads/TOS/tosutil folder.

Command

./tosutil cp tos://bucketname/object /Users/Downloads/TOS/tosutil -r

Output

Start at 2022-10-27 10:56:35.434563 +0000 UTC

Parallel: 1 Jobs: 5
Threshold: 100.00MB PartSize: auto
VerifyChecksum: false
CheckpointDir: /Users/.tosutil_checkpoint
OutputDir: /Users/.tosutil_output
TempFileDir:

[----------------------] 100.00% tps:51.09 3.02MB/s 4/4 238.21KB/238.21KB 278ms
Succeed count is: 4 Failed count is: 0
Succeed bytes is: 238.21KB
Metrics [max cost:216 ms, min cost:163 ms, average cost:191.50 ms, average tps:14.23]

Task id is: 0653482e-55e6-11ed-afb8-aa665a3c7f26

Torch Object Storage