Docs

Torch Object Storage

tosfsspec

Overview

This topic introduces the advanced features and best practices of tosfsspec, helping you gain a comprehensive understanding of the library.

Introduction

tosfsspec is a Python library that adheres to the pythonic file systems specification (fsspec). This enables developers to read from and write to TOS as conveniently and efficiently as they would with a local file system. Whether you are a data analyst or a backend developer, tosfsspec provides a standard, concise, and feature-rich interface for using TOS as the storage backend in Python applications.

With standard file handling functions such as open(), ls(), and rm(), you can directly access TOS and easily upload and download data, as well as traverse and manage directories. More importantly, tosfsspec works well with many popular data analysis libraries. This allows you to read data directly from TOS when processing large datasets, eliminating the need to integrate and wrap SDKs yourself.

Core features

Standardized interface: tosfsspec provides a file system interface that is fully compatible with fsspec. It is easy to use and has a low learning curve.

Seamless ecosystem integration: tosfsspec integrates seamlessly with libraries such as pandas, dask, and pyarrow. It can be configured via the storage_options parameter.

Rich file operations: tosfsspec can handle most file system operations including reading and writing, appending, copying, moving, deleting, pattern matching (glob), and recursive traversal (walk).

High-performance I/O: Built-in mechanisms such as multipart upload/download, concurrent task handling, and cyclic redundancy check (CRC-64) ensure stable and efficient large-scale data transfer.

TOS support: tosfsspec supports both hierarchical namespace (HNS) and flat namespace (FNS) buckets in TOS, with specific operation support and restrictions for each.

tosfsspec vs. s3fs

s3fs is a well-established library in the fsspec ecosystem for integrating Amazon S3. Although tosfsspec shares a similar design with s3fs, their functionality differs due to the different underlying storage services (TOS vs. S3) and how they are implemented.

Feature	s3fs	tosfsspec	Remarks
Underlying dependencies	boto3/botocore	tosnativeclient (a high-performance kernel).	tosfsspec uses a proprietary high-performance kernel, instead of the boto ecosystem.
Asynchronous support	Supported (via aiobotocore).	Not supported.	All tosfsspec operations are currently synchronous and blocking. They cannot be awaited in an asyncio event loop.
Credentials configuration	Supports multiple methods, such as through environment variables, a configuration file (~/.aws/), Identity and Access Management (IAM) roles, and explicit passing.	Primarily relies on the explicit passing of key, secret and region.	To keep configuration explicit within the fsspec framework, we recommend explicitly passing credentials during tosfsspec initialization rather than relying on global configuration.
Caching strategies	Supports multiple cache_type options, such as readahead, blockcache, and simplecache.	cache_type is fixed at none.	tosfsspec does not use the default caching layer provided by fsspec. Instead, it relies on the caching system built into the high-performance kernel.
Object versioning	Supported (via version_aware=True).	Not supported.	tosfsspec cannot handle all versions of an object. All operations work on the most recent version.
Requester Pays	Supported (via requester_pays=True).	Not supported.	tosfsspec currently does not support the Requester Pays feature.
Server-side encryption	Supported (via s3_additional_kwargs).	Partially supported (via storage_class and others).	TOS handles encryption settings differently from S3. tosfsspec manages server-side encryption indirectly via the storage_class parameter, lacking a direct passthrough option like s3_additional_kwargs in s3fs.
Fork safety in multiprocessing	Requires manual configuration. Users must set the start method to spawn or forkserver. Otherwise, it is unsafe.	Automatically reinitializes the client in child processes with the built-in fork safety mechanism.	Using tosfsspec with the fork method is a more robust option. However, we recommend using the spawn method to reduce potential issues.
Hierarchical namespace	Not supported (S3 uses a flat namespace).	Explicitly supports and distinguishes between HNS and FNS buckets (e.g., maxdepth restrictions).	tosfsspec provides adaptations for the two distinct TOS bucket types.

When migrating from s3fs to tosfsspec, pay attention to the following:

Passing credentials: Explicitly pass key, secret, and region in storage_options.

Removing unsupported parameters: Remove s3fs-specific constructor parameters, such as version_aware and requester_pays.

Asynchronous code migration: Rewrite asynchronous codes to run synchronously.