Torch Object Storage
Torch Object Storage
Torch Object Storage
Document
What's New
Announcements
Billing
Billable items
Billing method
Getting Started
User Guide
Working with buckets
Basic configurations
Managing lifecycle
Back-to-origin rules
Event notification
Intelligent Tiering
Permissions
Domain management
Fault tolerance and disaster recovery
Cross-region replication
Same-region replication
Data redundancy
Image processing
ObjectSet
Working with objects
Managing folders
Monitoring and alerting
Metric monitoring
Access control
Bucket policies
Common configuration scenarios
Authorizing IAM users of the current account
Authorizing other accounts
Managing access via tools and SDKs
Managing public read access
Data security
Access points
Multi-region access points (MRAP)
Single-region access points (SRAP)
Converged access points (CAP)
DevMind
TOS accelerator
SenseFlow
TOS Agent
Data processing
Basic concepts
Image processing
Media processing
Document preview
Task processing
Data subscription
Flow control management
Quality of Service (QoS) policies
Hierarchical namespace (HNS)
User guide
Tool reference
Best practices
TOS Vectors
User guide
API reference
Vector bucket
Vector index
SDK reference
Python-Vectors
Vector bucket
Vector index
Java-Vectors
Vector bucket
Vector index
Node.js-Vectors
Vector bucket
Vector index
Tool Guide
tosutil
Common commands
Upload, download, and copy objects
Share objects
Auxiliary commands
Common examples
TOS Browser
FSX
FSX best practices
TOS PyTorch Connector
tosfsspec
API Reference
Signature Mechanism
The header contains a signature
Bucket
Basic Bucket Operations
Bucket Lifecycle
Access Monitor
Bucket ACL
Bucket CORS
Bucket Policy
Back-to-origin Rules
Bucket Replication
Versioning
Static Website
Event Notification
Custom Domain
Server-side encryption
Transfer Acceleration
Data Processing
Image processing
Media processing
Video transcoding
Audio transcoding
video to GIF
Document preview
File decompression
Log Storage
Retention Policy
Requester pays
TLS version
Object
Basic Object Operations
Object ACL
Fetch Object
Symbolic Link
WORM object
Multipart Upload
SDK Reference
Java
Pre-signing Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Upload and Download Speed Limit
Copy Object
List Object
Data Processing
Image Processing
Video Processing
Audio Processing
Log Configuration
Hierarchical Namespace
Bucket base interface
Introduction to Typical Scenes
Python
Data processing
Pre-signature mechanism
Bucket interface
Basic bucket interface
Manage bucket configuration
Object interface
Upload object
Download object
Copy object
List objects
Introduction to typical scenes
C
Object Interface
Upload object
C++
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Data Processing
Go
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Bucket Interface
Manage Bucket Configuration
Data Processing
Log Configuration
Introduction to Typical Scenes
Pre-signature Mechanism
Node.js
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload object
Download object
Copy object
List objects
Data Processing
Image Processing
Introduction to Typical Scenes
Browser.js
Pre-signature Mechanism
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Data Processing
.NET
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects
Introduction to Typical Scenes
PHP
Bucket Interface
Object Interface
Download Object
Data Processing
Exception Handling
Android
Bucket Interface
Object Interface
Upload Object
Download Object
Exception Handling
iOS
Bucket Interface
Object Interface
Upload Object
Download Object
Harmony
Pre-signature Mechanism
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects v2
Introduction to Typical Scenarios
Rust
Pre-signing mechanism
Bucket interface
Basic bucket operations
Lifecycle rules
Bucket ACLs
Cross-origin resource sharing
Bucket inventory
Back-to-origin rules
Bucket replication
Static website hosting
Event notification
Custom domain
Server-side encryption
Object interface
Basic operation
Access control
Error Codes
01-SERVICE_ERROR
02-AUTH
03-ACCESS_CONTROL
04-QOS
05-REQUEST
06-BUCKET
07-LIFECYCLE
08-ACL
09-CORS
10-POLICY
11-MIRROR
12-REPLICATION
13-WEBSITE
14-NOTIFICATION
15-ENCRYPT
16-CUSTOMDOMAIN
17-OBJECT
18-TAG
19-MULTIPART
20-STORAGE_CLASS
21-CALLBACK
22-DATA_PROCESS
23-BATCH
24-STORAGE_LENS
25-INVENTORY
26-DIRECTORY_BUCKET
FAQ
Best Practices
AIGC-Doubao LLMs
Accessing TOS with third-party tools
Granting temporary access
Data migration
Data processing
Checking data integrity
Performance optimization
- Documentation
Torch Object StorageBest PracticesChecking data integrityVerifying the integrity of uploaded objects
Copy page
Download PDFChecking data integrity
Verifying the integrity of uploaded objects
Copy pageDownload PDFVerifying the integrity of uploaded objects
You can verify the integrity of uploaded objects using Content-MD5, Content-SHA256, or CRC-64. This topic introduces validation flowcharts, sample codes, and comparisons of different methods.
Using Content-MD5
For simple uploads or multipart uploads via TOS APIs, the client calculates the MD5 checksum of the data to be uploaded and passes this value to TOS via request headers or form fields to ensure data integrity.
Flowchart
Code examples
Golang
Java
Python
package main
import (
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"github.com/volcengine/ve-tos-golang-sdk/v2/tos"
)
func main() {
var (
accessKey = "your access key"
secretKey = "your secret key"
endpoint = "your endpoint"
region = "your region"
bucket = "bucketname"
key = "objectname"
)
client, err := tos.NewClientV2(endpoint, tos.WithRegion(region), tos.WithCredentials(tos.NewStaticCredentials(accessKey, secretKey)))
if err != nil {
panic(err)
}
data := []byte("hello world")
// Calculate the MD5 checksum
hash := md5.New()
hash.Write(data)
contentMD5 := base64.StdEncoding.EncodeToString(hash.Sum(nil))
// Upload object with the specified Content-MD5 header
input := &tos.PutObjectV2Input{}
input.Bucket = bucket
input.Key = key
input.Content = bytes.NewReader(data)
input.ContentMD5 = contentMD5
_, err = client.PutObjectV2(context.Background(), input)
if err != nil {
panic(err)
}
}
package com.volcengine.tos;
import com.volcengine.tos.auth.Credentials;
import com.volcengine.tos.auth.SignV4;
import com.volcengine.tos.auth.StaticCredentials;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.security.MessageDigest;
import java.util.Base64;
public class Main {
public static void main(String[] args) {
String endpoint = "your endpoint";
String accessKey = "your access key";
String secretKey = "your secret key";
String region = "your region";
String bucket = "bucketname";
String key = "objectname";
TOSClient client = new TOSClient(endpoint);
Credentials c = new StaticCredentials(accessKey, secretKey);
client.setCredentials(c);
client.setSigner(new SignV4(c, region));
String data = "hello world";
try {
// Calculate the MD5 checksum
MessageDigest md5 = MessageDigest.getInstance("MD5");
md5.update(data.getBytes());
String contentMD5 = Base64.getEncoder().encodeToString(md5.digest());
InputStream stream = new ByteArrayInputStream(data.getBytes());
client.putObject(bucket, key, stream, RequestOptions.withContentMD5(contentMD5));
} catch (Exception e) {
e.printStackTrace();
}
}
}
import base64
import hashlib
import tos
endpoint = 'your endpoint'
access_key = 'your access key'
secret_key = 'your secret key'
region = 'your region'
bucket = 'bucketname'
key = 'objectname'
data = 'hello world'
# Calculate the MD5 checksum
hash = hashlib.md5()
hash.update(data.encode())
content_md5 = base64.b64encode(hash.digest()).decode()
client = tos.TosClientV2(access_key, secret_key, endpoint, region)
try:
resp = client.put_object(bucket, key, content=data, content_md5=content_md5)
print(resp.status_code)
except Exception as e:
print(e)
Using Content-SHA256
This process for Content-SHA256 is the same as Content-MD5.
Flowchart
Code examples
Golang
Java
Python
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"github.com/volcengine/ve-tos-golang-sdk/v2/tos"
)
func main() {
var (
accessKey = "your access key"
secretKey = "your secret key"
endpoint = "your endpoint"
region = "your region"
bucket = "bucketname"
key = "objectname"
)
client, err := tos.NewClientV2(endpoint, tos.WithRegion(region), tos.WithCredentials(tos.NewStaticCredentials(accessKey, secretKey)))
if err != nil {
panic(err)
}
data := []byte("hello world")
// Calculate the SHA-256 checksum
hash := sha256.New()
hash.Write(data)
contentSHA256 := hex.EncodeToString(hash.Sum(nil))
// Upload object with the specified Content-SHA256 header
input := &tos.PutObjectV2Input{}
input.Bucket = bucket
input.Key = key
input.Content = bytes.NewReader(data)
input.ContentSHA256 = contentSHA256
_, err = client.PutObjectV2(context.Background(), input)
if err != nil {
panic(err)
}
}
package com.volcengine.tos;
import com.volcengine.tos.auth.Credentials;
import com.volcengine.tos.auth.SignV4;
import com.volcengine.tos.auth.StaticCredentials;
import com.volcengine.tos.model.object.GetObjectOutput;
import org.apache.commons.codec.binary.Hex;
import java.io.ByteArrayInputStream;
import java.io.InputStream;
import java.security.MessageDigest;
public class Main {
public static void main(String[] args) {
String endpoint = "your endpoint";
String accessKey = "your access key";
String secretKey = "your secret key";
String region = "your region";
String bucket = "bucketname";
String key = "objectname";
TOSClient client = new TOSClient(endpoint);
Credentials c = new StaticCredentials(accessKey, secretKey);
client.setCredentials(c);
client.setSigner(new SignV4(c, region));
String data = "hello world";
try {
MessageDigest sha256 = MessageDigest.getInstance("SHA-256");
sha256.update(data.getBytes());
String contentSHA256 = Hex.encodeHexString(sha256.digest());
InputStream stream = new ByteArrayInputStream(data.getBytes());
client.putObject(bucket, key, stream, RequestOptions.withContentSHA256(contentSHA256));
GetObjectOutput output = client.getObject(bucket, key);
output.getContent();
} catch (Exception e) {
e.printStackTrace();
}
}
}
import hashlib
import tos
endpoint = 'your endpoint'
access_key = 'your access key'
secret_key = 'your secret key'
region = 'your region'
bucket = 'bucketname'
key = 'objectname'
data = 'hello world'
# Calculate the SHA-256 checksum
hash = hashlib.sha256()
hash.update(data.encode())
content_sha256 = hash.digest().hex()
client = tos.TosClientV2(access_key, secret_key, endpoint, region)
try:
resp = client.put_object(bucket, key, content=data, content_sha256=content_sha256)
print(resp.status_code)
except Exception as e:
print(e)
Using CRC-64
For simple or multipart uploads via the TOS APIs, the client first calculates a CRC-64 checksum of the original data. Then, TOS calculates a CRC-64 checksum of the received data. Then, the server compares the two values to ensure the integrity of the uploaded data.
For multipart uploads, in addition to the aforementioned check, a composite checksum is calculated based on the individual checksums of each part. This value is then compared to the checksum of the entire object that the server returns to ensure the integrity of the final stored object.
Flowchart
Code examples
Golang
Python
package main
import (
"bytes"
"context"
"hash"
"io"
"github.com/volcengine/ve-tos-golang-sdk/v2/tos"
)
type crc64Reader struct {
r io.Reader
h hash.Hash64
}
func (cr *crc64Reader) Read(p []byte) (n int, err error) {
n, err = cr.r.Read(p)
cr.h.Write(p[:n])
return n, err
}
func main() {
var (
accessKey = "your access key"
secretKey = "your secret key"
endpoint = "your endpoint"
region = "your region"
bucket = "bucketname"
key = "objectname"
)
client, err := tos.NewClientV2(endpoint, tos.WithRegion(region), tos.WithCredentials(tos.NewStaticCredentials(accessKey, secretKey)))
if err != nil {
panic(err)
}
resp, err := client.CreateMultipartUploadV2(context.Background(), &tos.CreateMultipartUploadV2Input{
Bucket: bucket,
Key: key,
})
if err != nil {
panic(err)
}
uploadID := resp.UploadID
parts := make([]tos.UploadedPartV2, 0, 3)
var crc64Final uint64
data := make([]byte, 1024*1024*5)
for i := 1; i <= 2; i++ {
input := &tos.UploadPartV2Input{}
input.Bucket = bucket
input.Key = key
input.UploadID = uploadID
input.PartNumber = i
crc64Hash := tos.NewCRC(tos.DefaultCrcTable(), 0)
input.Content = &crc64Reader{
r: bytes.NewReader(data),
h: crc64Hash,
}
output, err := client.UploadPartV2(context.Background(), input)
if err != nil {
panic(err)
}
crc64Once := crc64Hash.Sum64()
if output.HashCrc64ecma != crc64Once {
panic("crc64 not match")
}
crc64Final = tos.CRC64Combine(crc64Final, crc64Once, uint64(len(data)))
parts = append(parts, tos.UploadedPartV2{
PartNumber: input.PartNumber,
ETag: output.ETag,
})
}
output, err := client.CompleteMultipartUploadV2(context.Background(), &tos.CompleteMultipartUploadV2Input{
Bucket: bucket,
Key: key,
UploadID: uploadID,
Parts: parts,
})
if err != nil {
panic(err)
}
if output.HashCrc64ecma != crc64Final {
panic("crc64 not match")
}
}
from io import BytesIO
import os
import tos
endpoint = 'your endpoint'
access_key = 'your access key'
secret_key = 'your secret key'
region = 'your region'
bucket_name = 'bucketname'
key = 'objectname'
class Crc64IO(object):
def __init__(self, data):
self.data = data
self.crc64 = tos.utils.Crc64()
def __iter__(self):
return self
def __next__(self):
return self.next()
def next(self):
content = self.data.read(8 * 1024)
if content:
self.crc64.update(content)
return content
raise StopIteration
# Local file full path, e.g., usr/local/testfile.txt
file_name = 'usr/local/testfile.txt'
client = tos.TosClientV2(access_key, secret_key, endpoint, region, enable_crc=False)
try:
total_size = os.path.getsize(file_name)
part_size = 1024 * 1024 * 5
resp = client.create_multipart_upload(bucket_name, key)
upload_id = resp.upload_id
parts = []
crc64 = tos.utils.Crc64()
crc64_final = 0
# Multipart upload
with open(file_name, 'rb') as f:
part_number = 1
offset = 0
while offset < total_size:
num_to_upload = min(part_size, total_size - offset)
crc64_io = Crc64IO(SizeAdapter(f, num_to_upload, init_offset=offset))
part = client.upload_part(bucket_name, key, upload_id, part_number,
content=crc64_io)
if crc64_io.crc64.crc != part.hash_crc64_ecma:
raise Exception('crc64 not match')
crc64_final = crc64.combine(crc64_final, crc64_io.crc64.crc, num_to_upload)
parts.append(part)
offset += num_to_upload
part_number += 1
resp = client.complete_multipart_upload(bucket_name, key, upload_id, parts)
if crc64_final != resp.hash_crc64_ecma:
raise Exception('crc64 not match')
except tos.exceptions.TosClientError as e:
# The operation failed and a client exception was caught. This is usually due to invalid request parameters or a network exception.
print('fail with client error, message:{}, cause: {}'.format(e.message, e.cause))
except tos.exceptions.TosServerError as e:
# The operation failed and a server exception occurred. See the response for the error details.
print('fail with server error, code: {}'.format(e.code))
# Use the request ID for troubleshooting purposes. We strongly recommend including the request ID in the logs.
print('error with request id: {}'.format(e.request_id))
print('error with message: {}'.format(e.message))
print('error with http code: {}'.format(e.status_code))
print('error with ec: {}'.format(e.ec))
print('error with request url: {}'.format(e.request_url))
except Exception as e:
print('fail with unknown error: {}'.format(e))
Comparison of the different validation methods
Validation method | Content-MD5 | Content-SHA256 | CRC64 |
Validation method | Content-MD5 | Content-SHA256 | CRC64 |
Use cases |
|
|
|
Disadvantages |
|
| Data validation is performed on the client side, and the object has been saved to TOS. If validation fails, additional cleanup actions are required, such as overwriting or deleting the object. |
Supported APIs |
| ||
Recommendations |
| | |
Last updated: 2026.10.10 17:55:21
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
