Torch Object Storage
Torch Object Storage
Torch Object Storage
Document
What's New
Announcements
Billing
Billable items
Billing method
Getting Started
User Guide
Working with buckets
Basic configurations
Managing lifecycle
Back-to-origin rules
Event notification
Intelligent Tiering
Permissions
Domain management
Fault tolerance and disaster recovery
Cross-region replication
Same-region replication
Data redundancy
Image processing
ObjectSet
Working with objects
Managing folders
Monitoring and alerting
Metric monitoring
Access control
Bucket policies
Common configuration scenarios
Authorizing IAM users of the current account
Authorizing other accounts
Managing access via tools and SDKs
Managing public read access
Data security
Access points
Multi-region access points (MRAP)
Single-region access points (SRAP)
Converged access points (CAP)
DevMind
TOS accelerator
SenseFlow
TOS Agent
Data processing
Basic concepts
Image processing
Media processing
Document preview
Task processing
Data subscription
Flow control management
Quality of Service (QoS) policies
Hierarchical namespace (HNS)
User guide
Tool reference
Best practices
TOS Vectors
User guide
API reference
Vector bucket
Vector index
SDK reference
Python-Vectors
Vector bucket
Vector index
Java-Vectors
Vector bucket
Vector index
Node.js-Vectors
Vector bucket
Vector index
Tool Guide
tosutil
Common commands
Upload, download, and copy objects
Share objects
Auxiliary commands
Common examples
TOS Browser
FSX
FSX best practices
TOS PyTorch Connector
tosfsspec
API Reference
Signature Mechanism
The header contains a signature
Bucket
Basic Bucket Operations
Bucket Lifecycle
Access Monitor
Bucket ACL
Bucket CORS
Bucket Policy
Back-to-origin Rules
Bucket Replication
Versioning
Static Website
Event Notification
Custom Domain
Server-side encryption
Transfer Acceleration
Data Processing
Image processing
Media processing
Video transcoding
Audio transcoding
video to GIF
Document preview
File decompression
Log Storage
Retention Policy
Requester pays
TLS version
Object
Basic Object Operations
Object ACL
Fetch Object
Symbolic Link
WORM object
Multipart Upload
SDK Reference
Java
Pre-signing Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Upload and Download Speed Limit
Copy Object
List Object
Data Processing
Image Processing
Video Processing
Audio Processing
Log Configuration
Hierarchical Namespace
Bucket base interface
Introduction to Typical Scenes
Python
Data processing
Pre-signature mechanism
Bucket interface
Basic bucket interface
Manage bucket configuration
Object interface
Upload object
Download object
Copy object
List objects
Introduction to typical scenes
C
Object Interface
Upload object
C++
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Data Processing
Go
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Bucket Interface
Manage Bucket Configuration
Data Processing
Log Configuration
Introduction to Typical Scenes
Pre-signature Mechanism
Node.js
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload object
Download object
Copy object
List objects
Data Processing
Image Processing
Introduction to Typical Scenes
Browser.js
Pre-signature Mechanism
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Data Processing
.NET
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects
Introduction to Typical Scenes
PHP
Bucket Interface
Object Interface
Download Object
Data Processing
Exception Handling
Android
Bucket Interface
Object Interface
Upload Object
Download Object
Exception Handling
iOS
Bucket Interface
Object Interface
Upload Object
Download Object
Harmony
Pre-signature Mechanism
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects v2
Introduction to Typical Scenarios
Rust
Pre-signing mechanism
Bucket interface
Basic bucket operations
Lifecycle rules
Bucket ACLs
Cross-origin resource sharing
Bucket inventory
Back-to-origin rules
Bucket replication
Static website hosting
Event notification
Custom domain
Server-side encryption
Object interface
Basic operation
Access control
Error Codes
01-SERVICE_ERROR
02-AUTH
03-ACCESS_CONTROL
04-QOS
05-REQUEST
06-BUCKET
07-LIFECYCLE
08-ACL
09-CORS
10-POLICY
11-MIRROR
12-REPLICATION
13-WEBSITE
14-NOTIFICATION
15-ENCRYPT
16-CUSTOMDOMAIN
17-OBJECT
18-TAG
19-MULTIPART
20-STORAGE_CLASS
21-CALLBACK
22-DATA_PROCESS
23-BATCH
24-STORAGE_LENS
25-INVENTORY
26-DIRECTORY_BUCKET
FAQ
Best Practices
AIGC-Doubao LLMs
Accessing TOS with third-party tools
Granting temporary access
Data migration
Data processing
Checking data integrity
Performance optimization
- Documentation
Torch Object StorageSDK ReferencePythonQuick start (Python SDK)
Copy page
Download PDFPython
Quick start (Python SDK)
Copy pageDownload PDFQuick start (Python SDK)
This article explains how to use the TOS Python SDK to perform common operations such as creating buckets, uploading objects, downloading objects, and deleting objects.
Notice
In multi-process scenarios, create a separate TosClient for each process. Do not share the same TosClient across processes to avoid concurrency and data corruption issues in read/write requests.
Installation and initialization
Install python-devel
You need to install the python-devel package.
The TOS Python SDK relies on crcmod to calculate CRC checksums. The C extension mode of crcmod depends on the Python.h file from the python-devel package. If Python.h is missing, crcmod’s C extension mode installation fails, causing crcmod to run in pure Python mode. The pure Python mode has much lower CRC calculation performance than the C extension mode, resulting in inefficient upload and download operations.
tip
If enabling CRC checksum causes poor upload or download performance, you can disable CRC64 checksum.
Verify crcmod mode
Enter the Python environment and type import crcmod._crcfunext.
- If the following error message appears, it means the C extension mode of the crcmod library failed to install; the crcmod library is operating in pure Python mode. The reason is that when compiling crcmod, _crcfunext.so depends on the Python.h file, which is missing in the system, causing the _crcfunext.so library to fail to generate.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named _crcfunext
- If no error message appears, it means the C extension mode of the crcmod library installed successfully.
If an error appears, you can resolve it as follows:
- Uninstall crcmod.
- pip uninstall crcmod
- Install python-devel.
- If you are using Windows or Mac OS X, the Python-dependent header files install automatically with Python, so you do not need to install python-devel.
- If you are using CentOS, RHEL, or Fedora, you can install it using the following command:
- # Python2.x versionyum install python-devel# Python3.x versionyum install python3-devel
- If you use Debian or Ubuntu, install it with the following command:
- # Python2.x versionapt-get install python-dev# Python3.x versionapt-get install python3-dev
- Reinstall crcmod.
- pip install crcmod
Client general example
The following code is a general template using the TOS Python SDK.
import os
import tos
# Obtain AK and SK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the endpoint corresponding to the region where the Bucket is located. # For example, in China North 2 (Beijing), set "your endpoint" to tos-cn-beijing.volces.com and "your region" to cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket = "bucket-test"
key = "object-test"
data = "Hello world"
try:
# Create a TosClientV2 object
client = tos.TosClientV2(ak, sk, endpoint, region)
# Call the API to request TOS service, for example, uploading an object
resp = client.put_object(bucket, key, content=data)
print('success, request id {}'.format(resp.request_id))
except tos.exceptions.TosClientError as e:
# Operation failed, client exception caught, usually due to invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, server exception caught, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# The request ID can be used to pinpoint specific issues; it is highly recommended to log and store it
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Create a bucket
A bucket is a globally unique namespace in TOS, acting as a container to store object data. The following code creates a new bucket named bucket-test.
import os
import tos
# Obtain AK and SK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the endpoint corresponding to the region where the Bucket is located. # For example, in China North 2 (Beijing), set "your endpoint" to tos-cn-beijing.volces.com and "your region" to cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
key = "object-test"
data = "Hello world"
try:
# Create a TosClientV2 object; operations on buckets and objects are all performed through TosClientV2
client = tos.TosClientV2(ak, sk, endpoint, region)
# Set the bucket's access permissions to private (private read and write)
client.create_bucket(bucket_name, acl=tos.ACLType.ACL_Private)
except tos.exceptions.TosClientError as e:
# Operation failed, client exception caught, usually due to invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, server exception caught, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# The request ID can be used to pinpoint specific issues; it is highly recommended to log and store it
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
tip
Upload an object
After successfully creating a new bucket, you can upload objects to it. The following code uploads a string to the TOS bucket bucket-test.
import os
import tos
# Obtain AK and SK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the endpoint corresponding to the region where the Bucket is located. # For example, in China North 2 (Beijing), set "your endpoint" to tos-cn-beijing.volces.com and "your region" to cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
try:
client = tos.TosClientV2(ak, sk, endpoint, region)
# Add Object via string
client.put_object(bucket_name, object_key, content='Hello World')
except tos.exceptions.TosClientError as e:
# Operation failed, client exception caught, usually due to invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, server exception caught, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# The request ID can be used to pinpoint specific issues; it is highly recommended to log and store it
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
tip
Downloading objects
The following code downloads the specified TOS object into memory.
import os
import tos
# Obtain AK and SK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the endpoint corresponding to the region where the Bucket is located. # For example, in China North 2 (Beijing), set "your endpoint" to tos-cn-beijing.volces.com and "your region" to cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
try:
# Create a TosClientV2 object; operations on buckets and objects are all performed through TosClientV2
client = tos.TosClientV2(ak, sk, endpoint, region)
# Download the object into memory
object_stream = client.get_object(bucket_name, object_key)
# object_stream is an iterator for reading data iteratively
# for content in object_stream:
# print(content)
# You can also call the read() method to retrieve the complete data in memory at once
print(object_stream.read())
except tos.exceptions.TosClientError as e:
# Operation failed, client exception caught, usually due to invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, server exception caught, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# The request ID can be used to pinpoint specific issues; it is highly recommended to log and store it
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
tip
Listing objects
The following code lists objects in the specified bucket bucket-test, listing up to 2,000 objects.
import os
import tos
# Obtain AK and SK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the endpoint corresponding to the region where the Bucket is located. # For example, in China North 2 (Beijing), set "your endpoint" to tos-cn-beijing.volces.com and "your region" to cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
try:
# Create a TosClientV2 object; operations on buckets and objects are all performed through TosClientV2
client = tos.TosClientV2(ak, sk, endpoint, region)
# By default, up to 1,000 objects are listed per request
list_out_1 = client.list_objects(bucket_name)
for ob in list_out_1.contents:
print(ob.key)
# If not all objects have been listed, you can continue listing by setting the optional parameter 'marker'
if list_out_1.is_truncated:
list_out_2 = client.list_objects(bucket_name, marker=list_out_1.next_marker)
for ob in list_out_2.contents:
print(ob.key)
except tos.exceptions.TosClientError as e:
# Operation failed, client exception caught, usually due to invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, server exception caught, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# The request ID can be used to pinpoint specific issues; it is highly recommended to log and store it
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
tip
Deleting objects
The following code deletes the object object-test from the specified bucket bucket-test.
import os
import tos
# Obtain AK and SK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the endpoint corresponding to the region where the Bucket is located. # For example, in China North 2 (Beijing), set "your endpoint" to tos-cn-beijing.volces.com and "your region" to cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
try:
client = tos.TosClientV2(ak, sk, endpoint, region)
# Delete an object from the bucket
client.delete_object(bucket_name, object_key)
except tos.exceptions.TosClientError as e:
# Operation failed, client exception caught, usually due to invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, server exception caught, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# The request ID can be used to pinpoint specific issues; it is highly recommended to log and store it
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
tip
Close TosClient
The TOS Python SDK enables persistent connections by default. When you create multiple TosClient instances to access TOS, use the following code to close unused TosClient instances and release connection resources.
import os
import tos
# Obtain AK and SK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the endpoint corresponding to the region where the Bucket is located. # For example, in China North 2 (Beijing), set "your endpoint" to tos-cn-beijing.volces.com and "your region" to cn-beijing.
endpoint = "your endpoint"
region = "your region"
try:
# Create a TosClientV2 object; operations on buckets and objects are all performed through TosClientV2
client = tos.TosClientV2(ak, sk, endpoint, region)
# After performing the relevant operations, close the TosClient when it is no longer needed
client.close()
except tos.exceptions.TosClientError as e:
# Operation failed, client exception caught, usually due to invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, server exception caught, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# The request ID can be used to pinpoint specific issues; it is highly recommended to log and store it
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Last updated: 2025.09.05 14:00:07
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
