Torch Object Storage
Torch Object Storage
Torch Object Storage
Document
What's New
Announcements
Billing
Billable items
Billing method
Getting Started
User Guide
Working with buckets
Basic configurations
Managing lifecycle
Back-to-origin rules
Event notification
Intelligent Tiering
Permissions
Domain management
Fault tolerance and disaster recovery
Cross-region replication
Same-region replication
Data redundancy
Image processing
ObjectSet
Working with objects
Managing folders
Monitoring and alerting
Metric monitoring
Access control
Bucket policies
Common configuration scenarios
Authorizing IAM users of the current account
Authorizing other accounts
Managing access via tools and SDKs
Managing public read access
Data security
Access points
Multi-region access points (MRAP)
Single-region access points (SRAP)
Converged access points (CAP)
DevMind
TOS accelerator
SenseFlow
TOS Agent
Data processing
Basic concepts
Image processing
Media processing
Document preview
Task processing
Data subscription
Flow control management
Quality of Service (QoS) policies
Hierarchical namespace (HNS)
User guide
Tool reference
Best practices
TOS Vectors
User guide
API reference
Vector bucket
Vector index
SDK reference
Python-Vectors
Vector bucket
Vector index
Java-Vectors
Vector bucket
Vector index
Node.js-Vectors
Vector bucket
Vector index
Tool Guide
tosutil
Common commands
Upload, download, and copy objects
Share objects
Auxiliary commands
Common examples
TOS Browser
FSX
FSX best practices
TOS PyTorch Connector
tosfsspec
API Reference
Signature Mechanism
The header contains a signature
Bucket
Basic Bucket Operations
Bucket Lifecycle
Access Monitor
Bucket ACL
Bucket CORS
Bucket Policy
Back-to-origin Rules
Bucket Replication
Versioning
Static Website
Event Notification
Custom Domain
Server-side encryption
Transfer Acceleration
Data Processing
Image processing
Media processing
Video transcoding
Audio transcoding
video to GIF
Document preview
File decompression
Log Storage
Retention Policy
Requester pays
TLS version
Object
Basic Object Operations
Object ACL
Fetch Object
Symbolic Link
WORM object
Multipart Upload
SDK Reference
Java
Pre-signing Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Upload and Download Speed Limit
Copy Object
List Object
Data Processing
Image Processing
Video Processing
Audio Processing
Log Configuration
Hierarchical Namespace
Bucket base interface
Introduction to Typical Scenes
Python
Data processing
Pre-signature mechanism
Bucket interface
Basic bucket interface
Manage bucket configuration
Object interface
Upload object
Download object
Copy object
List objects
Introduction to typical scenes
C
Object Interface
Upload object
C++
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Data Processing
Go
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Bucket Interface
Manage Bucket Configuration
Data Processing
Log Configuration
Introduction to Typical Scenes
Pre-signature Mechanism
Node.js
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload object
Download object
Copy object
List objects
Data Processing
Image Processing
Introduction to Typical Scenes
Browser.js
Pre-signature Mechanism
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Data Processing
.NET
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects
Introduction to Typical Scenes
PHP
Bucket Interface
Object Interface
Download Object
Data Processing
Exception Handling
Android
Bucket Interface
Object Interface
Upload Object
Download Object
Exception Handling
iOS
Bucket Interface
Object Interface
Upload Object
Download Object
Harmony
Pre-signature Mechanism
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects v2
Introduction to Typical Scenarios
Rust
Pre-signing mechanism
Bucket interface
Basic bucket operations
Lifecycle rules
Bucket ACLs
Cross-origin resource sharing
Bucket inventory
Back-to-origin rules
Bucket replication
Static website hosting
Event notification
Custom domain
Server-side encryption
Object interface
Basic operation
Access control
Error Codes
01-SERVICE_ERROR
02-AUTH
03-ACCESS_CONTROL
04-QOS
05-REQUEST
06-BUCKET
07-LIFECYCLE
08-ACL
09-CORS
10-POLICY
11-MIRROR
12-REPLICATION
13-WEBSITE
14-NOTIFICATION
15-ENCRYPT
16-CUSTOMDOMAIN
17-OBJECT
18-TAG
19-MULTIPART
20-STORAGE_CLASS
21-CALLBACK
22-DATA_PROCESS
23-BATCH
24-STORAGE_LENS
25-INVENTORY
26-DIRECTORY_BUCKET
FAQ
Best Practices
AIGC-Doubao LLMs
Accessing TOS with third-party tools
Granting temporary access
Data migration
Data processing
Checking data integrity
Performance optimization
- Documentation
Torch Object StorageSDK ReferencePythonObject interfaceUpload objectResumable upload (Python SDK)
Copy page
Download PDFUpload object
Resumable upload (Python SDK)
Copy pageDownload PDFResumable upload (Python SDK)
Before uploading the file to TOS by resuming the upload through a breakpoint, you can set the sharding size, the number of threads to upload the sharding, the speed limit on the client side when uploading, the event callback function, etc. At the same time, the task can also be cancelled during the execution of the resume upload task at the breakpoint. If the file upload fails due to network abnormalities, etc., you can resume uploading the unfinished part from the breakpoint.
Precautions
- Before uploading an object, you must have tos: PutObjectpermission, see Overview.
- When uploading an object, the object name must meet certain specifications. For details, see Common Concepts.
- TOS is a distributed object storage product designed for mass storage, which stores object index data in internal partition. To scale out the maximum throughput when you upload and download objects and reduce the probability of hotspot partitions, please avoid lexicographically increasing object naming.
- If an object with the same name already exists in the bucket, the new object will overwrite the existing object. If versioning is enabled in your bucket, the original object is preserved and a new version number is generated to identify the newly uploaded object.
Sample code
Reenter upload after failure
The following code is used to resume uploading with a breakpoint. Upload the local file to the object-test object in the target bucket bucket-testobject-test object in the target bucket bucket-test. If TosClientError is thrown during uploading and the reason for the error is a network timeout, the user can call upload_filewith the same parameters to achieve breakpoint reentry uploading.
import os
import tos
# Retrieve AKSK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the corresponding Endpoint of the region where your Bucket is located. Taking the example of North China 2 (Beijing), fill in "your endpoint" with tos-cn-beijing.volces.com and "your region" with cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
# Local file full path, for example usr/local/testfile.txt
filename = "/usr/local/testfile.txt"
try:
# Create TosClientV2 object to perform operations on buckets and objects
client = tos.TosClientV2(ak, sk, endpoint, region)
# The optional parameter part_size can be used to specify the size of each part
# The enable_checkpoint parameter can be used to enable or disable the checkpoint feature
# The task_num parameter can be used to set the number of threads
client.upload_file(bucket_name, object_key, filename,
# Set the number of threads for resumable upload, default is 1
task_num=3,
# Set the chunk size for resumable upload, default is 20mb
part_size=1024 * 1024 * 5)
except tos.exceptions.TosClientError as e:
# Operation failed, catch client error, usually caused by invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, catch server error, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# request id can be used to locate specific issues, it is strongly recommended to save it in the log.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Processing progress bar
The following code is used to configure the breakpoint continuation upload progress bar function.
import os
import tos
from tos import DataTransferType
# Retrieve AKSK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the corresponding Endpoint of the region where your Bucket is located. Taking the example of North China 2 (Beijing), fill in "your endpoint" with tos-cn-beijing.volces.com and "your region" with cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
# Local file full path, for example usr/local/testfile.txt
filename = "/usr/local/testfile.txt"
try:
# Create a TosClientV2 object to perform operations on buckets and objects.
client = tos.TosClientV2(ak, sk, endpoint, region)
def percentage(consumed_bytes: int, total_bytes: int, rw_once_bytes: int, type: DataTransferType):
if total_bytes:
rate = int(100 * float(consumed_bytes) / float(total_bytes))
print("rate:{}, consumed_bytes:{},total_bytes{}, rw_once_bytes:{}, type:{}".format(rate, consumed_bytes,
total_bytes,
rw_once_bytes, type))
# The optional parameter part_size can be used to specify the size of each chunk.
# The enable_checkpoint parameter can be used to enable or disable the feature of resumable uploads.
# The task_num parameter can be used to set the number of threads.
client.upload_file(bucket_name, object_key, filename,
# Set the number of threads for resumable upload, default is 1
task_num=3,
# Set the size of each part for resumable upload, default is 20mb
part_size=1024 * 1024 * 5,
# Set the progress bar callback function for resumable upload
data_transfer_listener=percentage)
except tos.exceptions.TosClientError as e:
# Operation failed, catch client error, usually caused by invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, catch server error, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# request id can be used to locate specific issues, it is strongly recommended to save it in the log.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Handling event callbacks
The following code is used to customize the breakpoint continuation upload callback function.
import os
import tos
from tos import UploadEventType
from tos.models2 import PartInfo
# Retrieve AKSK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the corresponding Endpoint of the region where your Bucket is located. Taking the example of North China 2 (Beijing), fill in "your endpoint" with tos-cn-beijing.volces.com and "your region" with cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
# Local file full path, for example usr/local/testfile.txt
filename = "/usr/local/testfile.txt"
try:
client = tos.TosClientV2(ak, sk, endpoint, region)
# Custom event callback handler function
def upload_event(type: UploadEventType, err: Exception, bucket: str, key: str, upload_id: str, file_path: str,
checkpoint_file: str, part: PartInfo):
print(type, err, bucket, key, upload_id, file_path, checkpoint_file, part)
# The optional parameter part_size can be used to specify the size of each chunk.
# The enable_checkpoint parameter can be used to enable or disable the feature of resuming from checkpoints.
# The task_num can be set to determine the number of threads.
client.upload_file(bucket_name, object_key, filename,
# Set the number of threads for resumable upload, default is 1
task_num=3,
# Set the chunk size for resumable upload, default is 5mb
part_size=1024 * 1024 * 5,
# Enable resumable upload, default is enabled
enable_checkpoint=True,
# Configure custom event callbacks
upload_event_listener=upload_event)
except tos.exceptions.TosClientError as e:
# Operation failed, catch client error, usually caused by invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, catch server error, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# request id can be used to locate specific issues, it is strongly recommended to save it in the log.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Processing client side speed limit
The following code is used to set the client side speed limit for a breakpoint continuation upload.
import os
import tos
from tos import UploadEventType
from tos.models2 import PartInfo
# Retrieve AKSK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the corresponding Endpoint of the region where your Bucket is located. Taking the example of North China 2 (Beijing), fill in "your endpoint" with tos-cn-beijing.volces.com and "your region" with cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
# Local file full path, for example usr/local/testfile.txt
filename = "/usr/local/testfile.txt"
try:
# Create a TosClientV2 object to perform operations on buckets and objects.
client = tos.TosClientV2(ak, sk, endpoint, region)
# The TOS Python SDK implements client throttling using the basic token bucket algorithm, where rate is the rate at which tokens are sent and capacity is the total capacity.
# The following configuration means an average upload speed of 5mb/s, with a maximum supported upload speed of 10 + 5 mb/s.
rate_limiter = tos.RateLimiter(5 * 1024 * 1024, 10 * 1024 * 1024)
client.upload_file(bucket_name, object_key, filename,
# Set the number of threads for resumable upload to 3 by default.
task_num=3,
# Set the size of each part for resumable upload to 5mb by default.
part_size=1024 * 1024 * 5,
# Enable resumable upload by default.
enable_checkpoint=True,
# rate_limiter is an optional parameter used for client-side rate limiting.
rate_limiter=rate_limiter)
except tos.exceptions.TosClientError as e:
# Operation failed, catch client error, usually caused by invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, catch server error, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# request id can be used to locate specific issues, it is strongly recommended to save it in the log.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Cancellation mechanism
The following code is used to cancel an ongoing breakpoint continuation upload task at runtime.
import time
from threading import Thread
import os
import tos
from tos.checkpoint import CancelHook
# Retrieve AKSK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
# Fill in "your endpoint" and "your region" with the corresponding Endpoint of the region where your Bucket is located. Taking the example of North China 2 (Beijing), fill in "your endpoint" with tos-cn-beijing.volces.com and "your region" with cn-beijing.
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
# Local file full path, for example usr/local/testfile.txt
filename = "/usr/local/testfile.txt"
try:
# Create a TosClientV2 object to perform operations on buckets and objects.
client = tos.TosClientV2(ak, sk, endpoint, region)
# Inherit the CancelHook class to implement the function of canceling breakpoint resumable upload tasks.
class MyCancel(CancelHook):
def cancel(self, is_abort: bool):
# When is_abort is true, delete the context information and abort the segmented upload task. When is_abort is false, only interrupt the current execution.
# When overriding the cancel method, it is necessary to call the cancel method of the parent class.
# Simulate canceling the task after 10 seconds.
time.sleep(10)
super(MyCancel, self).cancel(is_abort=is_abort)
print('some user define')
cancel = MyCancel()
t1 = Thread(target=cancel.cancel, args=(False,))
t1.start()
upload_out = client.upload_file(bucket_name, object_key, file_path=filename,
# Set the function for canceling resumable upload.
cancel_hook=cancel,
# Set the number of threads for uploading files, default is 1.
task_num=3,
# Set the file chunk size, default is 20MB, minimum is 5MB.
part_size=1024 * 1024 * 5)
except tos.exceptions.TosClientError as e:
# Operation failed, catch client error, usually caused by invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, catch server error, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# request id can be used to locate specific issues, it is strongly recommended to save it in the log.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Last updated: 2025.09.17 18:22:10
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
