Torch Object Storage
Torch Object Storage
Torch Object Storage
Document
What's New
Announcements
Billing
Billable items
Billing method
Getting Started
User Guide
Working with buckets
Basic configurations
Managing lifecycle
Back-to-origin rules
Event notification
Intelligent Tiering
Permissions
Domain management
Fault tolerance and disaster recovery
Cross-region replication
Same-region replication
Data redundancy
Image processing
ObjectSet
Working with objects
Managing folders
Monitoring and alerting
Metric monitoring
Access control
Bucket policies
Common configuration scenarios
Authorizing IAM users of the current account
Authorizing other accounts
Managing access via tools and SDKs
Managing public read access
Data security
Access points
Multi-region access points (MRAP)
Single-region access points (SRAP)
Converged access points (CAP)
DevMind
TOS accelerator
SenseFlow
TOS Agent
Data processing
Basic concepts
Image processing
Media processing
Document preview
Task processing
Data subscription
Flow control management
Quality of Service (QoS) policies
Hierarchical namespace (HNS)
User guide
Tool reference
Best practices
TOS Vectors
User guide
API reference
Vector bucket
Vector index
SDK reference
Python-Vectors
Vector bucket
Vector index
Java-Vectors
Vector bucket
Vector index
Node.js-Vectors
Vector bucket
Vector index
Tool Guide
tosutil
Common commands
Upload, download, and copy objects
Share objects
Auxiliary commands
Common examples
TOS Browser
FSX
FSX best practices
TOS PyTorch Connector
tosfsspec
API Reference
Signature Mechanism
The header contains a signature
Bucket
Basic Bucket Operations
Bucket Lifecycle
Access Monitor
Bucket ACL
Bucket CORS
Bucket Policy
Back-to-origin Rules
Bucket Replication
Versioning
Static Website
Event Notification
Custom Domain
Server-side encryption
Transfer Acceleration
Data Processing
Image processing
Media processing
Video transcoding
Audio transcoding
video to GIF
Document preview
File decompression
Log Storage
Retention Policy
Requester pays
TLS version
Object
Basic Object Operations
Object ACL
Fetch Object
Symbolic Link
WORM object
Multipart Upload
SDK Reference
Java
Pre-signing Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Upload and Download Speed Limit
Copy Object
List Object
Data Processing
Image Processing
Video Processing
Audio Processing
Log Configuration
Hierarchical Namespace
Bucket base interface
Introduction to Typical Scenes
Python
Data processing
Pre-signature mechanism
Bucket interface
Basic bucket interface
Manage bucket configuration
Object interface
Upload object
Download object
Copy object
List objects
Introduction to typical scenes
C
Object Interface
Upload object
C++
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Data Processing
Go
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Bucket Interface
Manage Bucket Configuration
Data Processing
Log Configuration
Introduction to Typical Scenes
Pre-signature Mechanism
Node.js
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload object
Download object
Copy object
List objects
Data Processing
Image Processing
Introduction to Typical Scenes
Browser.js
Pre-signature Mechanism
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Data Processing
.NET
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects
Introduction to Typical Scenes
PHP
Bucket Interface
Object Interface
Download Object
Data Processing
Exception Handling
Android
Bucket Interface
Object Interface
Upload Object
Download Object
Exception Handling
iOS
Bucket Interface
Object Interface
Upload Object
Download Object
Harmony
Pre-signature Mechanism
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects v2
Introduction to Typical Scenarios
Rust
Pre-signing mechanism
Bucket interface
Basic bucket operations
Lifecycle rules
Bucket ACLs
Cross-origin resource sharing
Bucket inventory
Back-to-origin rules
Bucket replication
Static website hosting
Event notification
Custom domain
Server-side encryption
Object interface
Basic operation
Access control
Error Codes
01-SERVICE_ERROR
02-AUTH
03-ACCESS_CONTROL
04-QOS
05-REQUEST
06-BUCKET
07-LIFECYCLE
08-ACL
09-CORS
10-POLICY
11-MIRROR
12-REPLICATION
13-WEBSITE
14-NOTIFICATION
15-ENCRYPT
16-CUSTOMDOMAIN
17-OBJECT
18-TAG
19-MULTIPART
20-STORAGE_CLASS
21-CALLBACK
22-DATA_PROCESS
23-BATCH
24-STORAGE_LENS
25-INVENTORY
26-DIRECTORY_BUCKET
FAQ
Best Practices
AIGC-Doubao LLMs
Accessing TOS with third-party tools
Granting temporary access
Data migration
Data processing
Checking data integrity
Performance optimization
- Documentation
Torch Object StorageTool GuidetosfsspecIntegrating tosfsspec with the ecosystem
Copy page
Download PDFtosfsspec
Integrating tosfsspec with the ecosystem
Copy pageDownload PDFIntegrating tosfsspec with the ecosystem
tosfsspec fully implements the fsspec specification. It can be used directly in Python and integrated with major data analysis and AI frameworks, including Pandas, Dask, Ray, PyArrow, and PyTorch.
tip
Before passing constructor parameters, you must configure long-term access credentials. For instructions, see Steps.
Using tosfsspec with fsspec
warning
Instead of using the tos:// protocol scheme, you must use the tos-new:// protocol scheme. tos:// is currently occupied by an unofficial, third-party tosfs package that has already been registered with fsspec.
You can access TOS resources directly using fsspec.open(). Use "tos-new://" in URLs and pass the required constructor parameters via storage_options.
Code example
import fsspec
import os
import tos
fsspec.register_implementation('tos-new', 'tosfsspec.TosFileSystem')
url = 'tos-new://your-bucket/path/to/data.csv'
# Get AK and SK from the environment variables
key = os.getenv('TOS_ACCESS_KEY')
secret = os.getenv('TOS_SECRET_KEY')
storage_options = {
'key','secret',
'region': 'cn-beijing',
}
with fsspec.open(url, 'rb', **storage_options) as f:
head = f.read(1024)
print(head)
When using fsspec.register_implementation, fsspec automatically selects TosFileSystem as the underlying file system implementation.
Using tosfsspec in Python
You can use tosfsspec's common APIs to read from and write to TOS in Python, similar to how you would work with a local file system.
Code example
import os
import tosfsspec
import tos
# Get AK and SK from the environment variables
key = os.getenv('TOS_ACCESS_KEY')
secret = os.getenv('TOS_SECRET_KEY')
fs = tosfsspec.TosFileSystem(
key,secret,
region='cn-beijing'
)
bucket = 'your-bucket'
local_file_path = 'localfile.txt'
remote_file_path = f'tos://{bucket}/remote_file.txt'
# Create a local file to upload
with open(local_file_path, 'w') as f:
f.write('Hello TOSFS.')
# Upload to TOS
fs.put_file(local_file_path, remote_file_path)
print(f'Uploaded {local_file_path} to {remote_file_path}')
# Download from TOS
downloaded_file_path = 'downloaded_file.txt'
fs.get_file(remote_file_path, downloaded_file_path)
print(f'Downloaded {remote_file_path} to {downloaded_file_path}')
# Read content from downloaded local file
with open(downloaded_file_path, 'r') as f:
content = f.read()
print(f'Content of {downloaded_file_path}: {content}')
# Delete tos file
fs.rm(remote_file_path)
# Write to TOS
with fs.open(remote_file_path, 'w') as f:
f.write('Hello TOSFS.')
# Read from TOS
with fs.open(remote_file_path, 'r') as f:
tos_content = f.read()
print(f'Content of {remote_file_path}: {tos_content}')
# Clean local files
os.remove(local_file_path)
os.remove(downloaded_file_path)
Using tosfsspec in a multi-process scenario
tosfsspec works well in multiple processes. However, we recommend using the spawn mode to start child processes.
Code example
import multiprocessing
import tosfsspec
import os
import tos
bucket = 'your-bucket'
# Get AK and SK from the environment variables
key = os.getenv('TOS_ACCESS_KEY')
secret = os.getenv('TOS_SECRET_KEY')
def action(fs):
result = fs.ls(f'tos://{bucket}/')
print(result)
if __name__ == '__main__':
# We recommend using the spawn mode
multiprocessing.set_start_method('spawn', force=True)
fs = tosfsspec.TosFileSystem(
key,secret,
region='cn-beijing'
)
processes = [multiprocessing.Process(target=action, args=(fs,)) for i in range(4)]
for p in processes:
p.start()
for p in processes:
p.join()
Integrating tosfsspec with Pandas
Since tosfsspec follows the fsspec specification, Pandas can use tosfsspec to read from or write to TOS in formats like CSV and Parquet without downloading the data locally.
- Reading a CSV file from TOS
Code example
import fsspec
import pandas as pd
import tosfsspec
import os
import tos
fsspec.register_implementation('tos-new', 'tosfsspec.TosFileSystem')
# Get AK and SK from the environment variables
key = os.getenv('TOS_ACCESS_KEY')
secret = os.getenv('TOS_SECRET_KEY')
# Create a CSV file in TOS
fs = tosfsspec.TosFileSystem(
key,secret,
region='cn-beijing'
)
bucket = 'your-bucket'
csv_path = f'tos-new://{bucket}/datasets/iris.csv'
csv_content = 'sepal_length,sepal_width,petal_length,petal_width,species\n5.1,3.5,1.4,0.2,setosa\n'
with fs.open(csv_path, 'wb') as f:
f.write(csv_content.encode())
# Configure storage_options
storage_options = {
'key','secret',
'region': 'cn-beijing'
}
# Use Pandas to read data from TOS
df = pd.read_csv(csv_path, storage_options=storage_options)
print('Successfully read data from TOS:')
print(df)
- Writing a Pandas DataFrame to TOS as a Parquet file
Code example
import fsspec
import pandas as pd
import os
import tos
fsspec.register_implementation('tos-new', 'tosfsspec.TosFileSystem')
bucket = 'your-bucket'
# Get AK and SK from the environment variables
key = os.getenv('TOS_ACCESS_KEY')
secret = os.getenv('TOS_SECRET_KEY')
# Create a DataFrame
data = {'col1': [1, 2], 'col2': [3, 4]}
df_to_write = pd.DataFrame(data=data)
# Define the path to the Parquet file you want to write
parquet_path = f'tos-new://{bucket}/output/data.parquet'
# Configure storage_options
storage_options = {
'key','secret',
'region': 'cn-beijing'
}
# Write directly to TOS. Make sure that pyarrow or fastparquet has been installed.
df_to_write.to_parquet(parquet_path, storage_options=storage_options, index=False)
print(f'DataFrame successfully written to: {parquet_path}')
Integrating tosfsspec with Dask
Dask is a flexible parallel computing library that natively supports fsspec‑compatible file systems. With tosfsspec, you can use dask.dataframe to read multiple CSV files from TOS in parallel and create a distributed DataFrame.
- Suppose your TOS bucket contains several CSV files structured as follows:
Code example
your-bucket/
└── stock_data/
├── part.0.csv
├── part.1.csv
└── part.2.csv
- You can load all of them at once using Dask:
Code example
import dask.dataframe as dd
import fsspec
import os
import tos
fsspec.register_implementation('tos-new', 'tosfsspec.TosFileSystem')
bucket = 'your-bucket'
# Get AK and SK from the environment variables
key = os.getenv('TOS_ACCESS_KEY')
secret = os.getenv('TOS_SECRET_KEY')
# Configure storage_options
storage_options = {
'key','secret',
'region': 'cn-beijing'
}
# Wildcards are supported in file paths
dask_path = f'tos-new://{bucket}/stock_data/part.*.csv'
# Use Dask to read data. Dask automatically handles wildcards and loads files in parallel.
ddf = dd.read_csv(dask_path, storage_options=storage_options)
# At this point, the data has not actually been loaded because Dask uses lazy evaluation.
print('Number of Dask DataFrame partitions:', ddf.npartitions)
# Only when you trigger a computation, such as calculating the mean, does the actual I/O operation occur.
mean_value = ddf.some_column.mean().compute()
print('Computed value:', mean_value)
Integrating tosfsspec with Ray (distributed computing)
Ray can leverage tosfsspec to directly read and process data stored in TOS in a distributed manner.
Code example
import ray
import tosfsspec
import os
import tos
# Initialize Ray. Ray is usually initialized in a cluster environment.
ray.init()
bucket = 'your-bucket'
# Get AK and SK from the environment variables
key = os.getenv('TOS_ACCESS_KEY')
secret = os.getenv('TOS_SECRET_KEY')
fs = tosfsspec.TosFileSystem(
key,secret,
region='cn-beijing'
)
# csv file path
input_csv_path = f'tos://{bucket}/input.csv'
output_csv_path = f'tos://{bucket}/output.csv'
with fs.open(input_csv_path, 'w') as f:
f.write('id,name,age\n1,John Doe,30\n2,Jane Smith,25\n3,Bob Johnson,40\n')
ds = ray.data.read_csv(input_csv_path, filesystem=fs)
# Processing data
# ds = ds.map_batches(your_processing_function)
ds.repartition(1).write_csv(output_csv_path, filesystem=fs)
Integrating tosfsspec with PyArrow
PyArrow can leverage tosfsspec via the filesystem parameter to efficiently read from and write to TOS.
Code example
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq
import tosfsspec
import os
import tos
bucket = 'your-bucket'
# Get AK and SK from the environment variables
key = os.getenv('TOS_ACCESS_KEY')
secret = os.getenv('TOS_SECRET_KEY')
fs = tosfsspec.TosFileSystem(
key,secret,
region='cn-beijing'
)
# Create a Parquet dataset
table = pa.table({'year': [2020, 2022, 2021, 2022, 2019, 2021],
'n_legs': [2, 2, 4, 4, 5, 100],
'animal': ['Flamingo', 'Parrot', 'Dog', 'Horse',
'Brittle stars', 'Centipede']})
pq.write_table(table, f'tos://{bucket}/data.parquet', filesystem=fs)
# Read the Parquet dataset
dataset = ds.dataset(f'tos://{bucket}/data.parquet', filesystem=fs)
# Return dataset information
print(f'Rows: {dataset.count_rows()}')
print(dataset.schema)
# Convert to a Pandas DataFrame
df = dataset.to_table().to_pandas()
print(df.head())
Integrating tosfsspec with PyTorch
PyTorch can be integrated with tosfsspec for tasks such as loading datasets and reading and writing checkpoints.
Code example
import fsspec
import torch
import tosfsspec
from torchdata.datapipes.iter import FSSpecFileLister, FSSpecFileOpener
import os
import tos
bucket = 'your-bucket'
# Get AK and SK from the environment variables
key = os.getenv('TOS_ACCESS_KEY')
secret = os.getenv('TOS_SECRET_KEY')
# Load the dataset
fsspec.register_implementation('tos-new', 'tosfsspec.TosFileSystem', )
storage_options = {
'key','secret',
'region': 'cn-beijing'
}
file_lister = FSSpecFileLister(root=f'tos-new://{bucket}/dataset/', **storage_options)
iterable_dataset = FSSpecFileOpener(file_lister, mode='rb', **storage_options)
for _, item in iterable_dataset:
data = item.read()
# Read/write checkpoints
fs = tosfsspec.TosFileSystem(
key=,secret,
region='cn-beijing'
)
model = torch.nn.Linear(10, 10)
with fs.open(f'tos://{bucket}/models/model.pt', 'wb') as f:
torch.save(model.state_dict(), f)
with fs.open(f'tos://{bucket}/models/model.pt', 'rb') as f:
state_dict = torch.load(f)
# print(state_dict)
model.load_state_dict(state_dict)
Last updated: 2026.03.31 18:14:37
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
