Docs

Torch Object Storage

FSX best practices

Uploading large objects

This topic explains how to upload large objects to a TOS bucket mounted via FSX.

How uploads work

To improve write performance, FSX uses a mechanism called multipart upload. This mechanism uploads a single object to TOS as a set of parts and implements concurrent uploads. Consequently, the maximum size of an object that can be written to TOS is determined by the number of parts and the size of each one. This is calculated using the following formula:

Maximum object size = Number of parts × part size

Due to TOS constraints, a single object can have up to 10,000 parts. The default part size is 8 MB. Therefore, the maximum size of an object that can be written to TOS by default is 80,000 MB.

Configuring large object uploads

The part_size parameter specifies the size of each part of a multipart upload in MB, with a valid range of 4 to 5,120 MB. Increase the part_size value if an object exceeds 80,000 MB to ensure successful upload.

Calculate the value of part_size, i.e., the size of each part of a multipart upload.

part_size = Size of the object to be uploaded (MB) ÷ 10,000

For example, if the object is 82,000 MB, the part size would be 82,000 ÷ 10,000, which equals 8.2 MB. Round up because the part size must be an integer. Therefore, set part_size to 9.

tip

part_size must be an integer. If it has decimals, round it up.

Run this command to specify part_size when mounting a bucket.

mount -t fsx /tos/<bucket name> <mount point> -o region="<your region>",endpoint="<your endpoint>",credential_filepath="<absolute path of credentials file>",part_size=<part size>,no_writeback_cache

tip

You can only configure the part_size parameter when mounting a bucket. If the bucket is already mounted, first unmount it, then remount it with the new parameter. For steps to unmount, see Unmounting mount points.

For instructions on how to mount a bucket via FSX, see Configuration and Mounting.

For more information on TOS regions and endpoints, see Regions and endpoints.

Once you've finished the configuration, you can upload large objects to the mounted bucket.

Last updated: 2025.11.25 14:05:57