Torch Object Storage
Torch Object Storage
Torch Object Storage
Document
What's New
Announcements
Billing
Billable items
Billing method
Getting Started
User Guide
Working with buckets
Basic configurations
Managing lifecycle
Back-to-origin rules
Event notification
Intelligent Tiering
Permissions
Domain management
Fault tolerance and disaster recovery
Cross-region replication
Same-region replication
Data redundancy
Image processing
ObjectSet
Working with objects
Managing folders
Monitoring and alerting
Metric monitoring
Access control
Bucket policies
Common configuration scenarios
Authorizing IAM users of the current account
Authorizing other accounts
Managing access via tools and SDKs
Managing public read access
Data security
Access points
Multi-region access points (MRAP)
Single-region access points (SRAP)
Converged access points (CAP)
DevMind
TOS accelerator
SenseFlow
TOS Agent
Data processing
Basic concepts
Image processing
Media processing
Document preview
Task processing
Data subscription
Flow control management
Quality of Service (QoS) policies
Hierarchical namespace (HNS)
User guide
Tool reference
Best practices
TOS Vectors
User guide
API reference
Vector bucket
Vector index
SDK reference
Python-Vectors
Vector bucket
Vector index
Java-Vectors
Vector bucket
Vector index
Node.js-Vectors
Vector bucket
Vector index
Tool Guide
tosutil
Common commands
Upload, download, and copy objects
Share objects
Auxiliary commands
Common examples
TOS Browser
FSX
FSX best practices
TOS PyTorch Connector
tosfsspec
API Reference
Signature Mechanism
The header contains a signature
Bucket
Basic Bucket Operations
Bucket Lifecycle
Access Monitor
Bucket ACL
Bucket CORS
Bucket Policy
Back-to-origin Rules
Bucket Replication
Versioning
Static Website
Event Notification
Custom Domain
Server-side encryption
Transfer Acceleration
Data Processing
Image processing
Media processing
Video transcoding
Audio transcoding
video to GIF
Document preview
File decompression
Log Storage
Retention Policy
Requester pays
TLS version
Object
Basic Object Operations
Object ACL
Fetch Object
Symbolic Link
WORM object
Multipart Upload
SDK Reference
Java
Pre-signing Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Upload and Download Speed Limit
Copy Object
List Object
Data Processing
Image Processing
Video Processing
Audio Processing
Log Configuration
Hierarchical Namespace
Bucket base interface
Introduction to Typical Scenes
Python
Data processing
Pre-signature mechanism
Bucket interface
Basic bucket interface
Manage bucket configuration
Object interface
Upload object
Download object
Copy object
List objects
Introduction to typical scenes
C
Object Interface
Upload object
C++
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Data Processing
Go
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Bucket Interface
Manage Bucket Configuration
Data Processing
Log Configuration
Introduction to Typical Scenes
Pre-signature Mechanism
Node.js
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload object
Download object
Copy object
List objects
Data Processing
Image Processing
Introduction to Typical Scenes
Browser.js
Pre-signature Mechanism
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Data Processing
.NET
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects
Introduction to Typical Scenes
PHP
Bucket Interface
Object Interface
Download Object
Data Processing
Exception Handling
Android
Bucket Interface
Object Interface
Upload Object
Download Object
Exception Handling
iOS
Bucket Interface
Object Interface
Upload Object
Download Object
Harmony
Pre-signature Mechanism
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects v2
Introduction to Typical Scenarios
Rust
Pre-signing mechanism
Bucket interface
Basic bucket operations
Lifecycle rules
Bucket ACLs
Cross-origin resource sharing
Bucket inventory
Back-to-origin rules
Bucket replication
Static website hosting
Event notification
Custom domain
Server-side encryption
Object interface
Basic operation
Access control
Error Codes
01-SERVICE_ERROR
02-AUTH
03-ACCESS_CONTROL
04-QOS
05-REQUEST
06-BUCKET
07-LIFECYCLE
08-ACL
09-CORS
10-POLICY
11-MIRROR
12-REPLICATION
13-WEBSITE
14-NOTIFICATION
15-ENCRYPT
16-CUSTOMDOMAIN
17-OBJECT
18-TAG
19-MULTIPART
20-STORAGE_CLASS
21-CALLBACK
22-DATA_PROCESS
23-BATCH
24-STORAGE_LENS
25-INVENTORY
26-DIRECTORY_BUCKET
FAQ
Best Practices
AIGC-Doubao LLMs
Accessing TOS with third-party tools
Granting temporary access
Data migration
Data processing
Checking data integrity
Performance optimization
- Documentation
Torch Object StorageBest PracticesAccessing TOS with AWS S3 SDK
Copy page
Download PDFBest Practices
Accessing TOS with AWS S3 SDK
Copy pageDownload PDFAccessing TOS with AWS S3 SDK
This topic will help you quickly and correctly configure the AWS S3 SDK for various major programming languages so you can access and manage TOS resources.
Overview
TOS provides Amazon S3-compatible APIs. This allows you to easily access and manage your storage resources on TOS with a simple configuration using the well-established, feature-rich AWS S3 SDKs.
- Unified technology stack: In multi-cloud environments, you can use the same AWS S3 SDK to manage storage services from different cloud providers, simplifying development and maintenance.
- Smooth migration: Applications migrating from AWS S3 or other S3-compatible storage to TOS require almost no code changes — just update the client configuration.
- Mature ecosystem: You can leverage the stable and efficient development experience of the AWS S3 SDK, along with an abundance of community examples and solutions.
Prerequisites
- You have created an FNS bucket. For more information on bucket types and how to create a bucket, see Bucket type and Creating buckets.
- To ensure secure connections, we recommend using AWS Signature Version 4 (s3v4) with AWS S3 SDKs.
- TOS S3-compatible domains only support virtual-hosted-style addressing. Path-style addressing is not supported. Therefore, when configuring the SDK, either explicitly specify virtual-hosted style or ensure that it is the default setting. If the SDK defaults to path-style addressing, disable it. This is critical for a successful connection.
Configuration parameters
- Region: The region where your TOS bucket is located, for example, cn-beijing. For more information, see Regions and endpoints.
- Endpoint: Optional. For more information, see Regions and endpoints.
- Access Key: A pair of access key ID (AK) and secret access key (SK) that serves as credentials for accessing the TOS API. For more information on creating and managing access keys, see API Access Key.
Configuring access credentials
Before initializing the SDK, you must first set up long-term access credentials. This process differs depending on the operating system.
macOS
Linux
Windows
- Open Terminal and run this command to open the file.
- nano ~/.bash_profile
- Add your access key ID and secret access key at the end of the file.
- export TOS_ACCESS_KEY=AKTP**********export TOS_SECRET_KEY=T1dJ**********
- Save the file and exit.
- Run this command to apply the configuration.
- source ~/.bash_profile
- Run this command to verify the configuration.
- echo $TOS_ACCESS_KEYecho $TOS_SECRET_KEY
- The following output confirms a successful configuration.
- AKTP**********T1dJ**********
- Open Terminal and run this command to open the file.
- sudo vim /etc/profile
- Add your access key ID and secret access key at the end of the file.
- export TOS_ACCESS_KEY=AKTP**********export TOS_SECRET_KEY=T1dJ**********
- Save the file and exit.
- Run this command to apply the configuration.
- source /etc/profile
- Run this command to verify the configuration.
- echo $TOS_ACCESS_KEYecho $TOS_SECRET_KEY
- The following output confirms a successful configuration.
- AKTP**********T1dJ**********
- Open the command line interface and run this command to enter your access key ID and secret access key.
- set TOS_ACCESS_KEY=AKTP**********set TOS_SECRET_KEY=T1dJ**********
- Run this command to apply the configuration.
- setx TOS_ACCESS_KEY "%TOS_ACCESS_KEY%"setx TOS_SECRET_KEY "%TOS_SECRET_KEY%"
- Restart the command line interface and run this command to verify the configuration.
- echo %TOS_ACCESS_KEY%echo %TOS_SECRET_KEY%
- The following output confirms a successful configuration.
- AKTP**********T1dJ**********
Examples
Java
Python
Go
C++
JavaScript(Node.js)
PHP
Rust
C#(.NET)
We recommend using the AWS SDK for Java v2, which provides a more modern API and better performance.
Environment
Make sure your development environment has JDK 8 or later installed.
Adding dependencies
Maven
Add the following dependency to your pom.xml file:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.25.27</version>
</dependency>
Gradle
Add the following dependency to your build.gradle file:
implementation 'software.amazon.awssdk:s3:2.25.27'
Code example
The following example illustrates how to initialize the S3Client and perform basic operations.
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.ResponseBytes;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.core.sync.ResponseTransformer;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;
import java.io.IOException;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Paths;
public class TosS3JavaBestPractice {
public static void main(String[] args) {
// 1. Configure authentication information and endpoint
final String endpoint = "https://tos-s3-cn-beijing.volces.com";
final String region = "cn-beijing";
final String accessKey = System.getenv("VOLCENGINE_ACCESS_KEY_ID");
final String secretKey = System.getenv("VOLCENGINE_SECRET_ACCESS_KEY");
// 2. Initialize the S3Client
// Note: The AWS SDK for Java v2 uses virtual-hosted style by default
// To disable path-style explicitly (which is required by some legacy versions or special configurations), use .forcePathStyle(false)
S3Client s3Client = S3Client.builder()
.endpointOverride(URI.create(endpoint))
.region(Region.of(region))
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create(accessKey, secretKey)))
.build();
String bucketName = "your-unique-bucket-name-" + System.currentTimeMillis();
String objectKey = "my-object-key";
String localFilePath = "local-file.txt";
String downloadFilePath = "downloaded-file.txt";
try {
// Prepare a local file
Files.write(Paths.get(localFilePath), "Hello, Volcengine TOS!".getBytes());
// 3. Create a bucket
System.out.println("Creating bucket: " + bucketName);
s3Client.createBucket(CreateBucketRequest.builder().bucket(bucketName).build());
System.out.println("Bucket created successfully.");
// 4. Upload an object
System.out.println("Uploading object to " + bucketName);
s3Client.putObject(PutObjectRequest.builder()
.bucket(bucketName)
.key(objectKey)
.build(), RequestBody.fromFile(Paths.get(localFilePath)));
System.out.println("Object uploaded successfully.");
// 5. List objects
System.out.println("Listing objects in bucket: " + bucketName);
ListObjectsV2Response listObjectsV2Response = s3Client.listObjectsV2(ListObjectsV2Request.builder()
.bucket(bucketName).build());
for (S3Object s3Object : listObjectsV2Response.contents()) {
System.out.println("- " + s3Object.key());
}
// 6. Download an object
System.out.println("Downloading object: " + objectKey);
ResponseBytes<GetObjectResponse> objectBytes = s3Client.getObject(
GetObjectRequest.builder().bucket(bucketName).key(objectKey).build(),
ResponseTransformer.toBytes());
Files.write(Paths.get(downloadFilePath), objectBytes.asByteArray());
System.out.println("Object downloaded successfully to " + downloadFilePath);
// 7. Delete an object
System.out.println("Deleting object: " + objectKey);
s3Client.deleteObject(DeleteObjectRequest.builder().bucket(bucketName).key(objectKey).build());
System.out.println("Object deleted successfully.");
// 8. Delete a bucket
System.out.println("Deleting bucket: " + bucketName);
s3Client.deleteBucket(DeleteBucketRequest.builder().bucket(bucketName).build());
System.out.println("Bucket deleted successfully.");
} catch (S3Exception | IOException e) {
System.err.println("Error: " + e.getMessage());
e.printStackTrace();
} finally {
// Remove a local file
try {
Files.deleteIfExists(Paths.get(localFilePath));
Files.deleteIfExists(Paths.get(downloadFilePath));
} catch (IOException ex) {
System.err.println("Failed to clean up local files: " + ex.getMessage());
}
s3Client.close();
}
}
}
Use the boto3 library to interact with the TOS S3-compatible APIs.
Environment
Make sure your development environment has Python 3.5, 3.6, 3.7, 3.8, or a later version installed.
Installing boto3
Install boto3 via pip:
pip install boto3
Code example
The following example illustrates how to initialize the S3Client and perform basic operations.
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError
import os
import time
def main():
# 1. Configure authentication information and endpoint
endpoint = "https://tos-s3-cn-beijing.volces.com"
region = "cn-beijing"
access_key = os.getenv("VOLCENGINE_ACCESS_KEY_ID")
secret_key = os.getenv("VOLCENGINE_SECRET_ACCESS_KEY")
# 2. Initialize the S3Client
# Key configurations:
# - endpoint_url: Points to the TOS S3 endpoint
# - region_name: Required, even if endpoint_url is specified
# - config:
# - signature_version: 's3v4'
# - s3: {'addressing_style': 'virtual'} to explicitly specify virtual-hosted style
s3_client = boto3.client(
's3',
endpoint_url=endpoint,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
region_name=region,
config=Config(
signature_version='s3v4',
s3={'addressing_style': 'virtual'} # Explicitly specify virtual-hosted style
)
)
bucket_name = f"your-unique-bucket-name-{int(time.time())}"
object_key = "my-object-key"
local_file_path = "local-file.txt"
download_file_path = "downloaded-file.txt"
try:
# Prepare a local file
with open(local_file_path, "w") as f:
f.write("Hello, Volcengine TOS!")
# 3. Create a bucket
print(f"Creating bucket: {bucket_name}")
# For the cn-beijing region, LocationConstraint is not required
s3_client.create_bucket(Bucket=bucket_name)
print("Bucket created successfully.")
# 4. Upload an object
print(f"Uploading object to {bucket_name}")
s3_client.upload_file(local_file_path, bucket_name, object_key)
print("Object uploaded successfully.")
# 5. List objects
print(f"Listing objects in bucket: {bucket_name}")
response = s3_client.list_objects_v2(Bucket=bucket_name)
if 'Contents' in response:
for obj in response['Contents']:
print(f"- {obj['Key']}")
# 6. Download an object
print(f"Downloading object: {object_key}")
s3_client.download_file(bucket_name, object_key, download_file_path)
print(f"Object downloaded successfully to {download_file_path}")
# 7. Delete an object
print(f"Deleting object: {object_key}")
s3_client.delete_object(Bucket=bucket_name, Key=object_key)
print("Object deleted successfully.")
# 8. Delete an object
print(f"Deleting bucket: {bucket_name}")
s3_client.delete_bucket(Bucket=bucket_name)
print("Bucket deleted successfully.")
except ClientError as e:
print(f"Error: {e}")
finally:
# Remove a local file
if os.path.exists(local_file_path):
os.remove(local_file_path)
if os.path.exists(download_file_path):
os.remove(download_file_path)
if __name__ == "__main__":
main()
Use aws-sdk-go-v2.
Environment
Make sure your development environment has Go 1.13 or later installed. You can download the installation package from the official Go website.
Installing dependencies
Run this command to install the modules required for the demonstration:
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/credentials
go get github.com/aws/aws-sdk-go-v2/service/s3
Code example
package main
import (
"context"
"fmt"
"io/ioutil"
"os"
"strconv"
"time"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
// "github.com/aws/aws-sdk-go-v2/service/s3/types" // [Fix 1] Remove unused package to avoid compilation error
)
func main() {
// 1. Configure authentication information and endpoint
endpoint := "https://tos-s3-cn-beijing.volces.com"
region := "cn-beijing"
accessKey := os.Getenv("VOLCENGINE_ACCESS_KEY_ID")
secretKey := os.Getenv("VOLCENGINE_SECRET_ACCESS_KEY")
// 2. Initialize the S3Client
customResolver := aws.EndpointResolverWithOptionsFunc(func(service, r string, options ...interface{}) (aws.Endpoint, error) {
return aws.Endpoint{
URL: endpoint,
SigningRegion: region,
HostnameImmutable: false, // [Fix 2] This value must be false! // Otherwise, the addressing mode is not the virtual‑hosted style required by TOS.
}, nil
})
cfg, err := config.LoadDefaultConfig(context.TODO(),
config.WithRegion(region),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")),
config.WithEndpointResolverWithOptions(customResolver),
)
if err != nil {
fmt.Printf("failed to load configuration, %v\n", err)
return
}
// Set UsePathStyle to false
s3Client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.UsePathStyle = false
})
bucketName := "your-unique-bucket-name-" + strconv.FormatInt(time.Now().Unix(), 10)
objectKey := "my-object-key"
localFilePath := "local-file.txt"
downloadFilePath := "downloaded-file.txt"
// Prepare a local file
err = ioutil.WriteFile(localFilePath, []byte("Hello, Volcengine TOS!"), 0644)
if err != nil {
fmt.Printf("failed to create local file, %v\n", err)
return
}
defer os.Remove(localFilePath)
// 3. Create a bucket
fmt.Printf("Creating bucket: %s\n", bucketName)
_, err = s3Client.CreateBucket(context.TODO(), &s3.CreateBucketInput{
Bucket: aws.String(bucketName),
})
if err != nil {
fmt.Printf("failed to create bucket, %v\n", err)
return
}
fmt.Println("Bucket created successfully.")
// Delete a bucket
defer func() {
fmt.Printf("Deleting bucket: %s\n", bucketName)
_, err = s3Client.DeleteBucket(context.TODO(), &s3.DeleteBucketInput{
Bucket: aws.String(bucketName),
})
if err != nil {
fmt.Printf("failed to delete bucket, %v\n", err)
} else {
fmt.Println("Bucket deleted successfully.")
}
}
// 4. Upload an object
fmt.Printf("Uploading object to %s\n", bucketName)
file, err := os.Open(localFilePath)
if err != nil {
fmt.Printf("failed to open file, %v\n", err)
return
}
defer file.Close()
_, err = s3Client.PutObject(context.TODO(), &s3.PutObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
Body: file,
})
if err != nil {
fmt.Printf("failed to upload object, %v\n", err)
return
}
fmt.Println("Object uploaded successfully.")
// Delete an object
defer func() {
fmt.Printf("Deleting object: %s\n", objectKey)
_, err = s3Client.DeleteObject(context.TODO(), &s3.DeleteObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
if err != nil {
fmt.Printf("failed to delete object, %v\n", err)
} else {
fmt.Println("Object deleted successfully.")
}
}
// 5. List objects
fmt.Printf("Listing objects in bucket: %s\n", bucketName)
listOutput, err := s3Client.ListObjectsV2(context.TODO(), &s3.ListObjectsV2Input{
Bucket: aws.String(bucketName),
})
if err != nil {
fmt.Printf("failed to list objects, %v\n", err)
return
}
for _, object := range listOutput.Contents {
fmt.Printf("- %s\n", *object.Key)
}
// 6. Download an object
fmt.Printf("Downloading object: %s\n", objectKey)
getOutput, err := s3Client.GetObject(context.TODO(), &s3.GetObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(objectKey),
})
if err != nil {
fmt.Printf("failed to download object, %v\n", err)
return
}
defer getOutput.Body.Close()
body, err := ioutil.ReadAll(getOutput.Body)
if err != nil {
fmt.Printf("failed to read object body, %v\n", err)
return
}
err = ioutil.WriteFile(downloadFilePath, body, 0644)
if err != nil {
fmt.Printf("failed to write downloaded file, %v\n", err)
return
}
defer os.Remove(downloadFilePath)
fmt.Printf("Object downloaded successfully to %s\n", downloadFilePath)
}
Use aws-sdk-cpp. Please note that building and integrating the C++ SDK is relatively complex. Refer to the official documentation to complete the environment setup.
Environment
Make sure your development environment has C++ 11 or later installed.
Building and installing aws-sdk-cpp
You need to build aws-sdk-cpp from source.
- Clone the repository: git clone --recurse-submodules https://github.com/aws/aws-sdk-cpp
- Build and install:
cd aws-sdk-cpp
mkdir build
cd build
cmake .. -DBUILD_ONLY="s3" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local
make
sudo make install
Code example
Make sure your CMakeLists.txt correctly links against aws-sdk-cpp:
find_package(AWSSDK REQUIRED COMPONENTS s3)
...
target_link_libraries(your-target PRIVATE ${AWSSDK_LIBRARIES})
Code example:
#include <iostream>
#include <fstream>
#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>
#include <aws/core/client/ClientConfiguration.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/CreateBucketRequest.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <aws/s3/model/GetObjectRequest.h>
#include <aws/s3/model/ListObjectsV2Request.h>
#include <aws/s3/model/DeleteObjectRequest.h>
#include <aws/s3/model/DeleteBucketRequest.h>
#include <chrono>
int main() {
Aws::SDKOptions options;
Aws::InitAPI(options);
{
// 1. Configure authentication information and endpoint
const Aws::String endpoint = "tos-s3-cn-beijing.volces.com";
const Aws::String region = "cn-beijing";
const Aws::String accessKey = std::getenv("VOLCENGINE_ACCESS_KEY_ID");
const Aws::String secretKey = std::getenv("VOLCENGINE_SECRET_ACCESS_KEY");
Aws::Client::ClientConfiguration config;
config.endpointOverride = endpoint;
config.region = region;
// The C++ SDK uses virtual-hosted style by default; no additional configuration is required
Aws::Auth::AWSCredentials credentials(accessKey, secretKey);
// 2. Initialize the S3Client
Aws::S3::S3Client s3Client(credentials, config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false);
auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(
std::chrono::system_clock::now().time_since_epoch()).count();
Aws::String bucketName = "your-unique-bucket-name-" + std::to_string(ms);
const Aws::String objectKey = "my-object-key";
const Aws::String localFilePath = "local-file.txt";
const Aws::String downloadFilePath = "downloaded-file.txt";
// Prepare a local file
std::ofstream localFile(localFilePath);
localFile << "Hello, Volcengine TOS!";
localFile.close();
// 3. Create a bucket
std::cout << "Creating bucket: " << bucketName << std::endl;
Aws::S3::Model::CreateBucketRequest createBucketRequest;
createBucketRequest.SetBucket(bucketName);
auto createBucketOutcome = s3Client.CreateBucket(createBucketRequest);
if (!createBucketOutcome.IsSuccess()) {
std::cerr << "CreateBucket error: " << createBucketOutcome.GetError().GetMessage() << std::endl;
return 1;
}
std::cout << "Bucket created successfully." << std::endl;
// 4. Upload an object
std::cout << "Uploading object to " << bucketName << std::endl;
Aws::S3::Model::PutObjectRequest putObjectRequest;
putObjectRequest.SetBucket(bucketName);
putObjectRequest.SetKey(objectKey);
std::shared_ptr<Aws::IOStream> input_data =
Aws::MakeShared<Aws::FStream>("PutObjectInputStream",
localFilePath.c_str(),
std::ios_base::in | std::ios_base::binary);
putObjectRequest.SetBody(input_data);
auto putObjectOutcome = s3Client.PutObject(putObjectRequest);
if (!putObjectOutcome.IsSuccess()) {
std::cerr << "PutObject error: " << putObjectOutcome.GetError().GetMessage() << std::endl;
} else {
std::cout << "Object uploaded successfully." << std::endl;
}
// 5. List objects
std::cout << "Listing objects in bucket: " << bucketName << std::endl;
Aws::S3::Model::ListObjectsV2Request listObjectsRequest;
listObjectsRequest.SetBucket(bucketName);
auto listObjectsOutcome = s3Client.ListObjectsV2(listObjectsRequest);
if (listObjectsOutcome.IsSuccess()) {
for (const auto& object : listObjectsOutcome.GetResult().GetContents()) {
std::cout << "- " << object.GetKey() << std::endl;
}
} else {
std::cerr << "ListObjects error: " << listObjectsOutcome.GetError().GetMessage() << std::endl;
}
// 6. Download an object
std::cout << "Downloading object: " << objectKey << std::endl;
Aws::S3::Model::GetObjectRequest getObjectRequest;
getObjectRequest.SetBucket(bucketName);
getObjectRequest.SetKey(objectKey);
auto getObjectOutcome = s3Client.GetObject(getObjectRequest);
if (getObjectOutcome.IsSuccess()) {
Aws::OFStream downloadedFile(downloadFilePath.c_str(), std::ios::out | std::ios::binary);
downloadedFile << getObjectOutcome.GetResult().GetBody().rdbuf();
std::cout << "Object downloaded successfully to " << downloadFilePath << std::endl;
} else {
std::cerr << "GetObject error: " << getObjectOutcome.GetError().GetMessage() << std::endl;
}
// 7. Delete an object
std::cout << "Deleting object: " << objectKey << std::endl;
Aws::S3::Model::DeleteObjectRequest deleteObjectRequest;
deleteObjectRequest.SetBucket(bucketName);
deleteObjectRequest.SetKey(objectKey);
auto deleteObjectOutcome = s3Client.DeleteObject(deleteObjectRequest);
if (deleteObjectOutcome.IsSuccess()) {
std::cout << "Object deleted successfully." << std::endl;
} else {
std::cerr << "DeleteObject error: " << deleteObjectOutcome.GetError().GetMessage() << std::endl;
}
// 8. Delete a bucket
std::cout << "Deleting bucket: " << bucketName << std::endl;
Aws::S3::Model::DeleteBucketRequest deleteBucketRequest;
deleteBucketRequest.SetBucket(bucketName);
auto deleteBucketOutcome = s3Client.DeleteBucket(deleteBucketRequest);
if (deleteBucketOutcome.IsSuccess()) {
std::cout << "Bucket deleted successfully." << std::endl;
} else {
std::cerr << "DeleteBucket error: " << deleteBucketOutcome.GetError().GetMessage() << std::endl;
}
// Remove a local file
remove(localFilePath.c_str());
remove(downloadFilePath.c_str());
}
Aws::ShutdownAPI(options);
return 0;
}
Use @aws-sdk/client-s3 (v3).
Environment
Make sure your development environment has Node.js 10 or later installed.
Installing the SDK
npm install @aws-sdk/client-s3
Code example
const {
S3Client,
CreateBucketCommand,
PutObjectCommand,
ListObjectsV2Command,
GetObjectCommand,
DeleteObjectCommand,
DeleteBucketCommand,
} = require("@aws-sdk/client-s3");
const { getSignedUrl } = require("@aws-sdk/s3-request-presigner");
const fs = require("fs");
const { Readable } = require("stream");
// Helper to stream to buffer
const streamToBuffer = (stream) =>
new Promise((resolve, reject) => {
const chunks = [];
stream.on("data", (chunk) => chunks.push(chunk));
stream.on("error", reject);
stream.on("end", () => resolve(Buffer.concat(chunks)));
});
async function main() {
// 1. Configure authentication information and endpoint
const endpoint = "https://tos-s3-cn-beijing.volces.com";
const region = "cn-beijing";
const accessKey = process.env.VOLCENGINE_ACCESS_KEY_ID;
const secretKey = process.env.VOLCENGINE_SECRET_ACCESS_KEY;
// 2. Initialize the S3Client
// To use S3Client V3, you must set the PathStyle parameter to false
const s3Client = new S3Client({
endpoint: endpoint,
region: region,
credentials: {
accessKeyId: accessKey,
secretAccessKey: secretKey,
},
forcePathStyle: false, // Critical: This value must be "false" to use virtual-hosted-style addressing.
});
const bucketName = `your-unique-bucket-name-${Date.now()}`;
const objectKey = "my-object-key";
const localFilePath = "local-file.txt";
const downloadFilePath = "downloaded-file.txt";
try {
// Prepare a local file
fs.writeFileSync(localFilePath, "Hello, Volcengine TOS!");
// 3. Create a bucket
console.log(`Creating bucket: ${bucketName}`);
await s3Client.send(new CreateBucketCommand({ Bucket: bucketName }));
console.log("Bucket created successfully.");
// 4. Upload an object
console.log(`Uploading object to ${bucketName}`);
const fileStream = fs.createReadStream(localFilePath);
await s3Client.send(new PutObjectCommand({
Bucket: bucketName,
Key: objectKey,
Body: fileStream,
}));
console.log("Object uploaded successfully.");
// 5. List objects
console.log(`Listing objects in bucket: ${bucketName}`);
const listResponse = await s3Client.send(new ListObjectsV2Command({ Bucket: bucketName }));
if (listResponse.Contents) {
listResponse.Contents.forEach(obj => console.log(`- ${obj.Key}`));
}
// 6. Download an object
console.log(`Downloading object: ${objectKey}`);
const getResponse = await s3Client.send(new GetObjectCommand({
Bucket: bucketName,
Key: objectKey,
}));
if (getResponse.Body instanceof Readable) {
const bodyBuffer = await streamToBuffer(getResponse.Body);
fs.writeFileSync(downloadFilePath, bodyBuffer);
console.log(`Object downloaded successfully to ${downloadFilePath}`);
}
// 7. Delete an object
console.log(`Deleting object: ${objectKey}`);
await s3Client.send(new DeleteObjectCommand({ Bucket: bucketName, Key: objectKey }));
console.log("Object deleted successfully.");
} catch (err) {
console.error("Error:", err);
} finally {
// 8. Delete a bucket
try {
console.log(`Deleting bucket: ${bucketName}`);
await s3Client.send(new DeleteBucketCommand({ Bucket: bucketName }));
console.log("Bucket deleted successfully.");
} catch (err) {
console.error("Error deleting bucket:", err);
}
// Remove a local file
if (fs.existsSync(localFilePath)) fs.unlinkSync(localFilePath);
if (fs.existsSync(downloadFilePath)) fs.unlinkSync(downloadFilePath);
}
}
main();
Use aws-sdk-php.
Environment
Make sure your development environment has PHP 5.6 or later installed.
Installing the SDK
Install via Composer:
composer require aws/aws-sdk-php
Code example
<?php
require 'vendor/autoload.php';
use Aws\S3\S3Client;
use Aws\S3\Exception\S3Exception;
// 1. Configure authentication information and endpoint
$endpoint = 'https://tos-s3-cn-beijing.volces.com';
$region = 'cn-beijing';
$accessKey = getenv('VOLCENGINE_ACCESS_KEY_ID');
$secretKey = getenv('VOLCENGINE_SECRET_ACCESS_KEY');
// 2. Initialize the S3Client
// Critical: 'use_path_style_endpoint' => false
$s3Client = new S3Client([
'version' => 'latest',
'region' => $region,
'endpoint' => $endpoint,
'credentials' => [
'key' => $accessKey,
'secret' => $secretKey,
],
'use_path_style_endpoint' => false // Explicitly specify virtual-hosted style
]);
$bucketName = 'your-unique-bucket-name-' . time();
$objectKey = 'my-object-key';
$localFilePath = 'local-file.txt';
$downloadFilePath = 'downloaded-file.txt';
try {
// Prepare a local file
file_put_contents($localFilePath, 'Hello, Volcengine TOS!');
// 3. Create a bucket
echo "Creating bucket: {$bucketName}\n";
$s3Client->createBucket(['Bucket' => $bucketName]);
echo "Bucket created successfully.\n";
// 4. Upload an object
echo "Uploading object to {$bucketName}\n";
$s3Client->putObject([
'Bucket' => $bucketName,
'Key' => $objectKey,
'SourceFile' => $localFilePath,
]);
echo "Object uploaded successfully.\n";
// 5. List objects
echo "Listing objects in bucket: {$bucketName}\n";
$objects = $s3Client->listObjectsV2(['Bucket' => $bucketName]);
if (!empty($objects['Contents'])) {
foreach ($objects['Contents'] as $object) {
echo "- " . $object['Key'] . "\n";
}
}
// 6. Download an object
echo "Downloading object: {$objectKey}\n";
$result = $s3Client->getObject([
'Bucket' => $bucketName,
'Key' => $objectKey,
'SaveAs' => $downloadFilePath
]);
echo "Object downloaded successfully to {$downloadFilePath}\n";
// 7. Delete an object
echo "Deleting object: {$objectKey}\n";
$s3Client->deleteObject([
'Bucket' => $bucketName,
'Key' => $objectKey,
]);
echo "Object deleted successfully.\n";
// 8. Delete a bucket
echo "Deleting bucket: {$bucketName}\n";
$s3Client->deleteBucket(['Bucket' => $bucketName]);
echo "Bucket deleted successfully.\n";
} catch (S3Exception $e) {
echo "Error: " . $e->getMessage() . "\n";
} finally {
// Remove a local file
if (file_exists($localFilePath)) {
unlink($localFilePath);
}
if (file_exists($downloadFilePath)) {
unlink($downloadFilePath);
}
}
Use aws-sdk-s3 (v0.34.0 or later).
Adding dependencies
Add the following dependencies to your Cargo.toml file:
[dependencies]
aws-config = "0.57.0"
aws-sdk-s3 = "0.34.0"
tokio = { version = "1", features = ["full"] }
http = "0.2.9"
Code example
use aws_config::meta::region::RegionProviderChain;
use aws_sdk_s3::{config::Region, meta::PKG_VERSION, Client, Error};
use std::fs::File;
use std::io::Write;
use std::path::Path;
use tokio::fs::File as TokioFile;
#[tokio::main]
async fn main() -> Result<(), Error> {
// 1. Configure authentication information and endpoint
let endpoint = "https://tos-s3-cn-beijing.volces.com";
let region_str = "cn-beijing";
let access_key = std::env::var("VOLCENGINE_ACCESS_KEY_ID").expect("VOLCENGINE_ACCESS_KEY_ID not set");
let secret_key = std::env::var("VOLCENGINE_SECRET_ACCESS_KEY").expect("VOLCENGINE_SECRET_ACCESS_KEY not set");
let region_provider = RegionProviderChain::first_try(Region::new(region_str));
let shared_config = aws_config::from_env()
.region(region_provider)
.endpoint_url(endpoint)
.credentials_provider(aws_config::Credentials::new(
access_key, secret_key, None, None, "default",
))
.load()
.await;
// 2. Initialize the S3Client
// The Rust SDK uses virtual-hosted style by default
let client = Client::new(&shared_config);
let bucket_name = format!("your-unique-bucket-name-{}", chrono::Utc::now().timestamp());
let object_key = "my-object-key".to_string();
let local_file_path = "local-file.txt";
let download_file_path = "downloaded-file.txt";
// Prepare a local file
let mut file = File::create(local_file_path).unwrap();
file.write_all(b"Hello, Volcengine TOS!").unwrap();
// 3. Create a bucket
println!("Creating bucket: {}", &bucket_name);
client.create_bucket().bucket(&bucket_name).send().await?;
println!("Bucket created successfully.");
// 4. Upload an object
println!("Uploading object to {}", &bucket_name);
let body = TokioFile::open(local_file_path).await.unwrap().into();
client
.put_object()
.bucket(&bucket_name)
.key(&object_key)
.body(body)
.send()
.await?;
println!("Object uploaded successfully.");
// 5. List objects
println!("Listing objects in bucket: {}", &bucket_name);
let mut response = client
.list_objects_v2()
.bucket(&bucket_name)
.into_paginator()
.send();
while let Some(result) = response.next().await {
match result {
Ok(output) => {
for object in output.contents().unwrap_or_default() {
println!("- {}", object.key().unwrap_or("Unknown"));
}
}
Err(err) => {
eprintln!("Error listing objects: {:?}", err);
break;
}
}
}
// 6. Download an object
println!("Downloading object: {}", &object_key);
let mut object = client
.get_object()
.bucket(&bucket_name)
.key(&object_key)
.send()
.await?;
let mut file = File::create(download_file_path).unwrap();
while let Some(bytes) = object.body.try_next().await.unwrap() {
file.write_all(&bytes).unwrap();
}
println!("Object downloaded successfully to {}", download_file_path);
// 7. Delete an object
println!("Deleting object: {}", &object_key);
client
.delete_object()
.bucket(&bucket_name)
.key(&object_key)
.send()
.await?;
println!("Object deleted successfully.");
// 8. Delete a bucket
println!("Deleting bucket: {}", &bucket_name);
client
.delete_bucket()
.bucket(&bucket_name)
.send()
.await?;
println!("Bucket deleted successfully.");
// Remove a local file
std::fs::remove_file(local_file_path).ok();
std::fs::remove_file(download_file_path).ok();
Ok(())
}
Use the AWSSDK.S3 NuGet package.
Environment
Make sure your development environment has .NET Framework 2.0 or later, or Mono 6.12 or later, installed.
Installing the SDK
Install via .NET CLI or NuGet package manager:
dotnet add package AWSSDK.S3
Code example
using System;
using System.IO;
using System.Threading.Tasks;
using Amazon.S3;
using Amazon.S3.Model;
namespace TosS3DotNetBestPractice
{
class Program
{
static async Task Main(string[] args)
{
// 1. Configure authentication information and endpoint
var endpoint = "https://tos-s3-cn-beijing.volces.com";
var region = "cn-beijing";
var accessKey = Environment.GetEnvironmentVariable("VOLCENGINE_ACCESS_KEY_ID");
var secretKey = Environment.GetEnvironmentVariable("VOLCENGINE_SECRET_ACCESS_KEY");
// 2. Initialize the S3Client
// Critical: ForcePathStyle = false
var config = new AmazonS3Config
{
ServiceURL = endpoint,
RegionEndpoint = Amazon.RegionEndpoint.GetBySystemName(region),
ForcePathStyle = false, // Explicitly specify virtual-hosted style
};
var s3Client = new AmazonS3Client(accessKey, secretKey, config);
var bucketName = $"your-unique-bucket-name-{DateTimeOffset.Now.ToUnixTimeSeconds()}";
var objectKey = "my-object-key";
var localFilePath = "local-file.txt";
var downloadFilePath = "downloaded-file.txt";
try
{
// Prepare a local file
await File.WriteAllTextAsync(localFilePath, "Hello, Volcengine TOS!");
// 3. Create a bucket
Console.WriteLine($"Creating bucket: {bucketName}");
await s3Client.PutBucketAsync(new PutBucketRequest
{
BucketName = bucketName,
UseClientRegion = true // Use the region configured for the client
});
Console.WriteLine("Bucket created successfully.");
// 4. Upload an object
Console.WriteLine($"Uploading object to {bucketName}");
await s3Client.PutObjectAsync(new PutObjectRequest
{
BucketName = bucketName,
Key = objectKey,
FilePath = localFilePath
});
Console.WriteLine("Object uploaded successfully.");
// 5. List objects
Console.WriteLine($"Listing objects in bucket: {bucketName}");
var listResponse = await s3Client.ListObjectsV2Async(new ListObjectsV2Request
{
BucketName = bucketName
});
foreach (var s3Object in listResponse.S3Objects)
{
Console.WriteLine($"- {s3Object.Key}");
}
// 6. Download an object
Console.WriteLine($"Downloading object: {objectKey}");
var getResponse = await s3Client.GetObjectAsync(new GetObjectRequest
{
BucketName = bucketName,
Key = objectKey
});
await getResponse.WriteResponseStreamToFileAsync(downloadFilePath, false, default);
Console.WriteLine($"Object downloaded successfully to {downloadFilePath}");
// 7. Delete an object
Console.WriteLine($"Deleting object: {objectKey}");
await s3Client.DeleteObjectAsync(new DeleteObjectRequest
{
BucketName = bucketName,
Key = objectKey
});
Console.WriteLine("Object deleted successfully.");
// 8. Delete a bucket
Console.WriteLine($"Deleting bucket: {bucketName}");
await s3Client.DeleteBucketAsync(new DeleteBucketRequest
{
BucketName = bucketName
});
Console.WriteLine("Bucket deleted successfully.");
}
catch (Exception e)
{
Console.WriteLine($"Error: {e.Message}");
}
finally
{
// Remove a local file
if (File.Exists(localFilePath)) File.Delete(localFilePath);
if (File.Exists(downloadFilePath)) File.Delete(downloadFilePath);
}
}
}
}
FAQ
- LocationConstraint in CreateBucket
- The S3 API requires the LocationConstraint parameter when creating a bucket in a region other than US East (N. Virginia), us-east-1. However, TOS S3-compatible APIs do not currently require this parameter, even in regions such as cn-beijing. Therefore, you can omit this configuration. If the SDK asks you to provide it, though, simply enter the same value for region.
- Certificate validation error
- In corporate intranet or proxy environments, SSL certificate validation may occasionally be intercepted. Make sure that your development environment trusts certificates for the *.volces.com domain.
- Some SDKs, such as the Python boto3 SDK, allow you to disable SSL certificate validation for debugging purposes using the verify=False parameter. However, doing so poses a security risk. We strongly recommend properly resolving this issue in production environments by configuring system or CA certificate trust chains.
- Signature version mismatch
- If you encounter signature-related errors when calling APIs, check your SDK initialization configuration. Make sure you have specified the AWS Signature Version 4 algorithm (s3v4) explicitly. TOS S3-compatible APIs depend heavily on this signature version.
- Incorrect virtual-hosted style configuration
- This is the most common configuration mistake when integrating with TOS. TOS S3-compatible domains do not support path-style addressing (e.g., https://tos-s3-cn-beijing.volces.com/bucket-name). All requests must use virtual-hosted-style addressing (e.g., https://bucket-name.tos-s3-cn-beijing.volces.com). Carefully check the addressing mode of the SDK you are using.
Last updated: 2026.08.25 10:39:45
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
