Torch Object Storage
Torch Object Storage
Torch Object Storage
Document
What's New
Announcements
Billing
Billable items
Billing method
Getting Started
User Guide
Working with buckets
Basic configurations
Managing lifecycle
Back-to-origin rules
Event notification
Intelligent Tiering
Permissions
Domain management
Fault tolerance and disaster recovery
Cross-region replication
Same-region replication
Data redundancy
Image processing
ObjectSet
Working with objects
Managing folders
Monitoring and alerting
Metric monitoring
Access control
Bucket policies
Common configuration scenarios
Authorizing IAM users of the current account
Authorizing other accounts
Managing access via tools and SDKs
Managing public read access
Data security
Access points
Multi-region access points (MRAP)
Single-region access points (SRAP)
Converged access points (CAP)
DevMind
TOS accelerator
SenseFlow
TOS Agent
Data processing
Basic concepts
Image processing
Media processing
Document preview
Task processing
Data subscription
Flow control management
Quality of Service (QoS) policies
Hierarchical namespace (HNS)
User guide
Tool reference
Best practices
TOS Vectors
User guide
API reference
Vector bucket
Vector index
SDK reference
Python-Vectors
Vector bucket
Vector index
Java-Vectors
Vector bucket
Vector index
Node.js-Vectors
Vector bucket
Vector index
Tool Guide
tosutil
Common commands
Upload, download, and copy objects
Share objects
Auxiliary commands
Common examples
TOS Browser
FSX
FSX best practices
TOS PyTorch Connector
tosfsspec
API Reference
Signature Mechanism
The header contains a signature
Bucket
Basic Bucket Operations
Bucket Lifecycle
Access Monitor
Bucket ACL
Bucket CORS
Bucket Policy
Back-to-origin Rules
Bucket Replication
Versioning
Static Website
Event Notification
Custom Domain
Server-side encryption
Transfer Acceleration
Data Processing
Image processing
Media processing
Video transcoding
Audio transcoding
video to GIF
Document preview
File decompression
Log Storage
Retention Policy
Requester pays
TLS version
Object
Basic Object Operations
Object ACL
Fetch Object
Symbolic Link
WORM object
Multipart Upload
SDK Reference
Java
Pre-signing Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Upload and Download Speed Limit
Copy Object
List Object
Data Processing
Image Processing
Video Processing
Audio Processing
Log Configuration
Hierarchical Namespace
Bucket base interface
Introduction to Typical Scenes
Python
Data processing
Pre-signature mechanism
Bucket interface
Basic bucket interface
Manage bucket configuration
Object interface
Upload object
Download object
Copy object
List objects
Introduction to typical scenes
C
Object Interface
Upload object
C++
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Data Processing
Go
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Bucket Interface
Manage Bucket Configuration
Data Processing
Log Configuration
Introduction to Typical Scenes
Pre-signature Mechanism
Node.js
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload object
Download object
Copy object
List objects
Data Processing
Image Processing
Introduction to Typical Scenes
Browser.js
Pre-signature Mechanism
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Data Processing
.NET
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects
Introduction to Typical Scenes
PHP
Bucket Interface
Object Interface
Download Object
Data Processing
Exception Handling
Android
Bucket Interface
Object Interface
Upload Object
Download Object
Exception Handling
iOS
Bucket Interface
Object Interface
Upload Object
Download Object
Harmony
Pre-signature Mechanism
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects v2
Introduction to Typical Scenarios
Rust
Pre-signing mechanism
Bucket interface
Basic bucket operations
Lifecycle rules
Bucket ACLs
Cross-origin resource sharing
Bucket inventory
Back-to-origin rules
Bucket replication
Static website hosting
Event notification
Custom domain
Server-side encryption
Object interface
Basic operation
Access control
Error Codes
01-SERVICE_ERROR
02-AUTH
03-ACCESS_CONTROL
04-QOS
05-REQUEST
06-BUCKET
07-LIFECYCLE
08-ACL
09-CORS
10-POLICY
11-MIRROR
12-REPLICATION
13-WEBSITE
14-NOTIFICATION
15-ENCRYPT
16-CUSTOMDOMAIN
17-OBJECT
18-TAG
19-MULTIPART
20-STORAGE_CLASS
21-CALLBACK
22-DATA_PROCESS
23-BATCH
24-STORAGE_LENS
25-INVENTORY
26-DIRECTORY_BUCKET
FAQ
Best Practices
AIGC-Doubao LLMs
Accessing TOS with third-party tools
Granting temporary access
Data migration
Data processing
Checking data integrity
Performance optimization
- Documentation
Torch Object StorageSDK ReferencePythonObject interfaceUpload objectMultipart upload (Python SDK)
Copy page
Download PDFUpload object
Multipart upload (Python SDK)
Copy pageDownload PDFMultipart upload (Python SDK)
For larger objects, you can divide them into multiple data blocks (parts) to upload separately, and finally call complete_multipart_upload to merge the uploaded data blocks into one object.
Precautions
- Before initiating multipart uploads, you must have the tos: PutObjectpermission. For details, see Overview.
- Before canceling multipart uploads, you must have tos: AbortMultipartUploadpermission. For details, see Overview.
- Part numbers start from 1 and go up to 10,000. Except for the last part, all other parts have a minimum size of 4 MiB.
- When uploading an object, the object name must meet certain specifications. For details, see Common concepts.
- TOS is a distributed object storage product designed for mass storage. The internal partition stores object index data. In order to scale out the maximum throughput when uploading and downloading objects, and reduce the probability of hot partitions, please avoid using lexicographically increasing object naming.
- If an object with the same name already exists in the bucket, the new object will overwrite the existing object. If versioning is enabled in your bucket, the original object is preserved and a new version number is generated to identify the newly uploaded object.
Multipart upload steps
Multipart uploading typically involves the following three steps.
- Initialize a multipart upload: Call the create_multipart_upload method to return the globally unique upload_id created by TOS.
- Upload parts: Call upload_part method to upload part data.
tip
- For the same multipart upload task (identified by upload_id), the part number ( part_number) identifies the relative position of the part throughout the object. If you upload data multiple times through the same part number, the original data source will be overwritten in TOS, and the last uploaded data shall prevail.
- When merging parts, you need to specify all the part information (part number, ETag) in the current multipart upload task.
- Complete the multipart upload: When all parts are uploaded, call complete_multipart_upload method to merge all shardings into one complete object.
Sample code
Complete multipart upload process
The following code uploads a local file to the object-test object in the target bucket bucket-testvia multipart upload.
import os
import tos
from tos.utils import SizeAdapter
# Retrieve AKSK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
# Local file full path, for example usr/local/testfile.txt
file_name = '/usr/local/testfile.txt'
total_size = os.path.getsize(file_name)
part_size = 5 * 1024 * 1024
try:
# Create a TosClientV2 object to perform operations on buckets and objects.
client = tos.TosClientV2(ak, sk, endpoint, region)
# Initialize upload task
# If you need to set the storage class of the object during initialization of the multipart upload, you can set it using the storage_class field.
# If you need to set the object ACL during initialization of the multipart upload, you can set it using the acl and grant_full_control fields.
multi_result = client.create_multipart_upload(bucket_name, object_key, acl=tos.ACLType.ACL_Public_Read,
storage_class=tos.StorageClassType.Storage_Class_Standard)
upload_id = multi_result.upload_id
parts = []
# Upload chunk data
with open(file_name, 'rb') as f:
part_number = 1
offset = 0
while offset < total_size:
num_to_upload = min(part_size, total_size - offset)
out = client.upload_part(bucket_name, object_key, upload_id, part_number,
content=SizeAdapter(f, num_to_upload, init_offset=offset))
parts.append(out)
offset += num_to_upload
part_number += 1
# Complete the chunked upload task.
client.complete_multipart_upload(bucket_name, object_key, upload_id, parts)
except tos.exceptions.TosClientError as e:
# Operation failed, catch client error, usually caused by invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, catch server error, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# request id can be used to locate specific issues, it is strongly recommended to save it in the log.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Configure progress bar
tip
For data in the form of strings, bytes, and local files, progress bar functionality is supported. Data of types such as network streams cannot obtain the size of the uploaded content, so the callback does not return the overall size of the object.
The following code is used to add a progress bar feature to the multipart upload process.
import os
import tos
from tos import DataTransferType
from tos.utils import SizeAdapter, MergeProcess
# Retrieve AKSK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
# Local file full path, for example usr/local/testfile.txt
file_name = '/usr/local/testfile.txt'
total_size = os.path.getsize(file_name)
part_size = 5 * 1024 * 1024
try:
# Create a TosClientV2 object to perform operations on buckets and objects.
client = tos.TosClientV2(ak, sk, endpoint, region)
def percentage(consumed_bytes: int, total_bytes: int, rw_once_bytes: int, type: DataTransferType):
if total_bytes:
rate = int(100 * float(consumed_bytes) / float(total_bytes))
print("rate:{}, consumed_bytes:{},total_bytes{}, rw_once_bytes:{}, type:{}".format(rate, consumed_bytes,
total_bytes,
rw_once_bytes, type))
# Configure the progress bar, which is different from regular uploads as the progress of the multipart upload needs to be aggregated.
data_transfer_listener = MergeProcess(percentage, total_size, (total_size + part_size - 1) // part_size, 0)
# Initialize the upload task
# If you need to set the storage class of the object during the initialization of the parts, you can set it through the storage_class field.
# If you need to set the object ACL during the initialization of the parts, you can set it through the acl, grant_full_control, and other fields.
multi_result = client.create_multipart_upload(bucket_name, object_key, acl=tos.ACLType.ACL_Public_Read,
storage_class=tos.StorageClassType.Storage_Class_Standard)
upload_id = multi_result.upload_id
parts = []
# Upload chunk data.
with open(file_name, 'rb') as f:
part_number = 1
offset = 0
while offset < total_size:
num_to_upload = min(part_size, total_size - offset)
out = client.upload_part(bucket_name, object_key, upload_id, part_number,
content=SizeAdapter(f, num_to_upload, init_offset=offset),
data_transfer_listener=data_transfer_listener)
parts.append(out)
offset += num_to_upload
part_number += 1
# Complete the chunked upload task.
client.complete_multipart_upload(bucket_name, object_key, upload_id, parts)
except tos.exceptions.TosClientError as e:
# Operation failed, catch client error, usually caused by invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, catch server error, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# request id can be used to locate specific issues, it is strongly recommended to save it in the log.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Cancel multipart uploads
You can cancel the multipart uploads using abort_multipart_upload. When a multipart upload is cancelled, TOS will delete the uploaded part data, and you can no longer perform any operations on the task.
The following code cancels the multipart upload task for the object object-test in the bucket-test.
import os
import tos
# Retrieve AKSK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
# The upload_id returned by the create_multipart_upload API.
upload_id = "your upload id"
try:
client = tos.TosClientV2(ak, sk, endpoint, region)
# Cancel the multipart upload event for the specified upload_id, and the uploaded parts will be deleted.
client.abort_multipart_upload(bucket_name, object_key, upload_id)
except tos.exceptions.TosClientError as e:
# Operation failed, catch client error, usually caused by invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, catch server error, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# request id can be used to locate specific issues, it is strongly recommended to save it in the log.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
List uploaded parts
The following code lists the uploaded parts of the object object-test in bucket-test.
import os
import tos
# Retrieve AKSK information from environment variables.
ak = os.getenv('TOS_ACCESS_KEY')
sk = os.getenv('TOS_SECRET_KEY')
endpoint = "your endpoint"
region = "your region"
bucket_name = "bucket-test"
object_key = "object-test"
# The upload_id returned by the create_multipart_upload API.
upload_id = "upload_id"
try:
# Create a TosClientV2 object to perform operations on buckets and objects.
client = tos.TosClientV2(ak, sk, endpoint, region)
# List the uploaded parts information corresponding to the specified upload_id.
truncated = True
marker = 0
while truncated:
# part_number_marker specifies the starting position of the chunk upload
# is_truncated is True indicates that there are still unfinished chunks to be listed
out = client.list_parts(bucket_name, object_key, upload_id, part_number_marker=marker)
for part in out.parts:
print('part_number:', part.part_number)
print('etag:', part.etag)
print('size', part.size)
marker = out.next_part_number_marker
truncated = out.is_truncated
except tos.exceptions.TosClientError as e:
# Operation failed, catch client error, usually caused by invalid request parameters or network issues
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# Operation failed, catch server error, detailed error information can be obtained from the response
print('fail with server error, code: {}'.format(e.code))
# request id can be used to locate specific issues, it is strongly recommended to save it in the log.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Related documentation
- For API documentation on creating a multipart upload task, see CreateMultipartUpload.
- For API documentation on uploading parts, see UploadPart.
- For API documentation on merging parts, see CompleteMultipartUpload.
- For API documentation on canceling multipart upload tasks, see AbortMultipartUpload.
Last updated: 2026.08.03 10:28:12
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
