Docs

Torch Object Storage

tosfsspec

Advanced use cases

This topic primarily covers the advanced capabilities of tosfsspec, with a focus on performance improvement, data consistency, and specific business scenarios.

Multipart uploads and downloads

When working with large objects, tosfsspec automatically enables multipart transfers. This splits an object into multiple parts, which are then uploaded or downloaded in parallel to improve throughput and stability.

Part size (part_size and block_size)

TosFileSystem(part_size=...): The part_size value specified when initializing TosFileSystem is a global default. This setting primarily affects server-side or multipart copying operations, such as cp_file.

fs.open(..., block_size=...): The block_size parameter in the open() method is used more often and directly determines the size of the buffer used when writing (wb or ab) or reading (rb) objects.

Write mode (wb or ab):

block_size in write mode determines the size of each part uploaded to TOS. To satisfy TOS requirements, this value must be greater than or equal to 4 MB. If it is not explicitly set, the part_size configured for TosFileSystem is used instead.

tip

When writing large objects, we recommend setting block_size between 8 MB and 128 MB depending on the network conditions and the size of the object. For most scenarios, 8 MB is a good starting point.

Read mode (rb):

block_size in read mode determines the size of each data block downloaded from TOS. It is the amount of data that the underlying client prefetches when you call file.read(size). Using the appropriate block_size can significantly improve random read performance.

Code example:

# Set the part size for writing to 10 MB
with fs.open('tos://my-bucket/large_file.dat', 'wb', block_size=10 * 1024 * 1024) as f:
# Write 20 MB of data, which will be uploaded in two parts
f.write(b'a' * 20 * 1024 * 1024)

Controlling concurrency

tosfsspec uses a high-performance kernel internally to execute large numbers of concurrent network requests. The size of the shared worker pools can be adjusted through initialization parameters.

shared_prefetch_tasks: Maximum number of concurrent prefetch tasks when reading data.

shared_upload_part_tasks: Number of concurrent multipart upload tasks.

shared_upload_part_copy_tasks: Number of concurrent multipart copy tasks.

Although increasing concurrency can enhance parallelism and potentially improve transfer speed under good network conditions, it also increases CPU and memory usage. Code example:

Append-only writes ('ab' mode)

tosfsspec supports opening objects in append mode (ab), which makes it ideal for continuous logging and incremental updates. This feature is implemented via the TOS AppendObject API. Code example:

warning

Append-only writes require the target object to be appendable. When an object is first created in wb mode, tosfsspec marks it as appendable. Subsequent append operations on objects not created this way (e.g., regular objects uploaded by other tools) will fail.

Content types (Content-Type)

tosfsspec automatically infers the MIME type (Content-Type) of uploaded objects, allowing browsers and other clients to handle them correctly.

put_file(): Infers Content-Type based on the extension of the local file.

open(..., 'wb') or touch(): Infers Content-Type based on the extension of the object key.

You can override the inferred value by explicitly specifying ContentType in open(), touch(), or put_file(). Code example:

Data integrity checks (enable_crc)

With the default setting of enable_crc=True, tosfsspec calculates a CRC64 checksum after each data transfer to ensure that the local and remote data are identical. Only disable this verification if you require high performance and can tolerate a low risk of data corruption. Code example:

Restrictions on HNS/FNS buckets (maxdepth)

TOS has two main types of buckets:

FNS (Flat Namespace): A flat namespace in which all objects are at a single level and directory structures are simulated.

HNS (Hierarchical Namespace): A hierarchical namespace that provides a native directory structure and better performance, similar to Hadoop Distributed File System (HDFS).

tosfsspec works with both bucket types, but some operations behave differently for each.

mv(): Moving a directory within an HNS bucket is an atomic operation, making it significantly more efficient. By contrast, moving a directory within an FNS bucket requires a multi-step process of copying each object to the new location and then individually deleting the original objects.

walk(), find(), rm(), copy(): The maxdepth parameter controls recursion depth for recursive operations. It is only valid for FNS buckets. Using the maxdepth parameter in an HNS bucket will result in a ValueError. This is because HNS buckets natively support hierarchical directory structures and do not allow control over recursion depth.

Last updated: 2026.03.31 18:14:37