Torch Object Storage
Torch Object Storage
Torch Object Storage
Document
What's New
Announcements
Billing
Billable items
Billing method
Getting Started
User Guide
Working with buckets
Basic configurations
Managing lifecycle
Back-to-origin rules
Event notification
Intelligent Tiering
Permissions
Domain management
Fault tolerance and disaster recovery
Cross-region replication
Same-region replication
Data redundancy
Image processing
ObjectSet
Working with objects
Managing folders
Monitoring and alerting
Metric monitoring
Access control
Bucket policies
Common configuration scenarios
Authorizing IAM users of the current account
Authorizing other accounts
Managing access via tools and SDKs
Managing public read access
Data security
Access points
Multi-region access points (MRAP)
Single-region access points (SRAP)
Converged access points (CAP)
DevMind
TOS accelerator
SenseFlow
TOS Agent
Data processing
Basic concepts
Image processing
Media processing
Document preview
Task processing
Data subscription
Flow control management
Quality of Service (QoS) policies
Hierarchical namespace (HNS)
User guide
Tool reference
Best practices
TOS Vectors
User guide
API reference
Vector bucket
Vector index
SDK reference
Python-Vectors
Vector bucket
Vector index
Java-Vectors
Vector bucket
Vector index
Node.js-Vectors
Vector bucket
Vector index
Tool Guide
tosutil
Common commands
Upload, download, and copy objects
Share objects
Auxiliary commands
Common examples
TOS Browser
FSX
FSX best practices
TOS PyTorch Connector
tosfsspec
API Reference
Signature Mechanism
The header contains a signature
Bucket
Basic Bucket Operations
Bucket Lifecycle
Access Monitor
Bucket ACL
Bucket CORS
Bucket Policy
Back-to-origin Rules
Bucket Replication
Versioning
Static Website
Event Notification
Custom Domain
Server-side encryption
Transfer Acceleration
Data Processing
Image processing
Media processing
Video transcoding
Audio transcoding
video to GIF
Document preview
File decompression
Log Storage
Retention Policy
Requester pays
TLS version
Object
Basic Object Operations
Object ACL
Fetch Object
Symbolic Link
WORM object
Multipart Upload
SDK Reference
Java
Pre-signing Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Upload and Download Speed Limit
Copy Object
List Object
Data Processing
Image Processing
Video Processing
Audio Processing
Log Configuration
Hierarchical Namespace
Bucket base interface
Introduction to Typical Scenes
Python
Data processing
Pre-signature mechanism
Bucket interface
Basic bucket interface
Manage bucket configuration
Object interface
Upload object
Download object
Copy object
List objects
Introduction to typical scenes
C
Object Interface
Upload object
C++
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Data Processing
Go
Object Interface
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Bucket Interface
Manage Bucket Configuration
Data Processing
Log Configuration
Introduction to Typical Scenes
Pre-signature Mechanism
Node.js
Pre-signature Mechanism
Bucket Interface
Basic Bucket Interface
Manage Bucket Configuration
Object Interface
Upload object
Download object
Copy object
List objects
Data Processing
Image Processing
Introduction to Typical Scenes
Browser.js
Pre-signature Mechanism
Upload Object
Download Object
Copy Object
List Objects
Manage Objects
Data Processing
.NET
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects
Introduction to Typical Scenes
PHP
Bucket Interface
Object Interface
Download Object
Data Processing
Exception Handling
Android
Bucket Interface
Object Interface
Upload Object
Download Object
Exception Handling
iOS
Bucket Interface
Object Interface
Upload Object
Download Object
Harmony
Pre-signature Mechanism
Bucket Interface
Object Interface
Upload Object
Download Object
List Objects v2
Introduction to Typical Scenarios
Rust
Pre-signing mechanism
Bucket interface
Basic bucket operations
Lifecycle rules
Bucket ACLs
Cross-origin resource sharing
Bucket inventory
Back-to-origin rules
Bucket replication
Static website hosting
Event notification
Custom domain
Server-side encryption
Object interface
Basic operation
Access control
Error Codes
01-SERVICE_ERROR
02-AUTH
03-ACCESS_CONTROL
04-QOS
05-REQUEST
06-BUCKET
07-LIFECYCLE
08-ACL
09-CORS
10-POLICY
11-MIRROR
12-REPLICATION
13-WEBSITE
14-NOTIFICATION
15-ENCRYPT
16-CUSTOMDOMAIN
17-OBJECT
18-TAG
19-MULTIPART
20-STORAGE_CLASS
21-CALLBACK
22-DATA_PROCESS
23-BATCH
24-STORAGE_LENS
25-INVENTORY
26-DIRECTORY_BUCKET
FAQ
Best Practices
AIGC-Doubao LLMs
Accessing TOS with third-party tools
Granting temporary access
Data migration
Data processing
Checking data integrity
Performance optimization
- Documentation
Torch Object StorageSDK ReferenceNode.jsObject InterfaceDownload objectResumable Download (Node.js SDK)
Copy page
Download PDFDownload object
Resumable Download (Node.js SDK)
Copy pageDownload PDFResumable Download (Node.js SDK)
When downloading large files, you can use the downloadFilebreakpoint to resume the download interface. Breakpoint continuation downloads use sharding downloads to write content to temporary files. If partial sharding succeeds and partial failure occurs, the information of successful download and failed sharding can be recorded and saved to the Checkpoint file. When downloading the same object again, only the failed sharding is downloaded. When all shards succeed, rename the temporary file to the official file and delete the checkpoint file.
Precautions
- Before downloading an object, you must have tos: GetObjectpermission, see Overview.
- To open multiple versions of a bucket, you must have the tos: GetObjectVersionpermission when downloading the specified version object. For details, see Overview.
- If an application downloads a large number of the same object at the same time, your access speed will be limited by TOS bandwidth and geography. It is recommended that you use CDN products to improve performance while also reducing your costs.
- The SDK will record the uploaded status information in the Checkpointfile, so the program needs to have write permission to the Checkpointfile.
- When using breakpoint to resume downloading, the progress information of the file download will be recorded in the Checkpointfile. If a sharding download fails during the download process, it will continue to download from the point recorded in the Checkpointfile when downloading again, so as to achieve the effect of breakpoint renewal. Once the download is complete, the Checkpointfile is deleted.
- If the file's ETag changes, Parts are lost, or modified during the download process, the file is downloaded again.
Sample code
Breakpoint Resume Download
The following code is used to download a file using a breakpoint continuation method.
// Import SDK, if the TOS Node.JS SDK version is less than 2.5.2, please change "TosClient" to "TOS" in the code below.
import { TosClient, TosClientError, TosServerError } from '@volcengine/tos-sdk';
import fsp from 'fs/promises';
// Create a client.
const client = new TosClient({
accessKeyId: process.env['TOS_ACCESS_KEY'],
accessKeySecret: process.env['TOS_SECRET_KEY'],
region: "Provide your region", // Fill in the region where the Bucket is located. For example, if it is in China North 2 (Beijing), fill in "Provide your region" as cn-beijing.
endpoint: "Provide your endpoint", // Enter the domain address.
});
function handleError(error) {
if (error instanceof TosClientError) {
console.log('Client Err Msg:', error.message);
console.log('Client Err Stack:', error.stack);
} else if (error instanceof TosServerError) {
console.log('Request ID:', error.requestId);
console.log('Response Status Code:', error.statusCode);
console.log('Response Header:', error.headers);
console.log('Response Err Code:', error.code);
console.log('Response Err Msg:', error.message);
} else {
console.log('unexpected exception, message: ', error);
}
}
async function main() {
try {
const bucketName = 'node-sdk-test-bucket'; // Target bucket
const objectName = 'example_dir/example.txt'; // Target object
// Downloaded file path
const filePath = './example_dir/example.txt';
await client.downloadFile({
bucket: bucketName,
key: objectName,
// Downloaded file path
filePath,
// Number of concurrent tasks for segmented downloading
taskNum: 5,
// Specify the temporary file path for resumable downloads
checkpoint: './example_dir/example.checkpoint',
});
const fileStat = await fsp.stat(filePath);
console.log('fileSize: ', fileStat.size);
} catch (error) {
handleError(error);
}
}
main();
Configure progress bar
When the breakpoint resumes the download, the download progress can be received through the dataTransferStatusChangeparameter. The code example is as follows.
// Import SDK, if the TOS Node.JS SDK version is less than 2.5.2, please change "TosClient" to "TOS" in the code below.
import { DataTransferType, TosClient, TosClientError, TosServerError } from '@volcengine/tos-sdk';
import fsp from 'fs/promises';
// Create a client.
const client = new TosClient({
accessKeyId: process.env['TOS_ACCESS_KEY'],
accessKeySecret: process.env['TOS_SECRET_KEY'],
region: "Provide your region", // Fill in the region where the Bucket is located. For example, if it is in China North 2 (Beijing), fill in "Provide your region" as cn-beijing.
endpoint: "Provide your endpoint", // Enter the domain address.
});
function handleError(error) {
if (error instanceof TosClientError) {
console.log('Client Err Msg:', error.message);
console.log('Client Err Stack:', error.stack);
} else if (error instanceof TosServerError) {
console.log('Request ID:', error.requestId);
console.log('Response Status Code:', error.statusCode);
console.log('Response Header:', error.headers);
console.log('Response Err Code:', error.code);
console.log('Response Err Msg:', error.message);
} else {
console.log('unexpected exception, message: ', error);
}
}
async function main() {
try {
const bucketName = 'node-sdk-test-bucket'; // Target bucket
const objectName = 'example_dir/example.txt'; // Target object
// Downloaded file path
const filePath = './example_dir/example.txt';
await client.downloadFile({
bucket: bucketName,
key: objectName,
// Downloaded file path
filePath,
// Specify the size of each part during download (10M)
partSize: 10 * 1024 * 1024,
// Number of concurrent part download tasks
taskNum: 5,
// Specify the temporary file path for resumable downloads
checkpoint: './example_dir/example.checkpoint',
// Set a custom callback function to monitor the download progress
dataTransferStatusChange: (event) => {
if (event.type === DataTransferType.Started) {
console.log('Data Transfer Started');
} else if (event.type === DataTransferType.Rw) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Once Read:${event.rwOnceBytes},ConsumerBytes/TotalBytes: ${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Succeed) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Data Transfer Succeed, ConsumerBytes/TotalBytes:${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Failed) {
console.log('Data Transfer Failed');
}
},
});
const fileStat = await fsp.stat(filePath);
console.log('fileSize: ', fileStat.size);
} catch (error) {
handleError(error);
}
}
main();
Configure event callback
The following code is used to customize the breakpoint continuation download callback function.
// Import SDK, if the TOS Node.JS SDK version is less than 2.5.2, please change "TosClient" to "TOS" in the code below.
import { DataTransferType, DownloadEventType, TosClient, TosClientError, TosServerError } from '@volcengine/tos-sdk';
import fsp from 'fs/promises';
// Create a client.
const client = new TosClient({
accessKeyId: process.env['TOS_ACCESS_KEY'],
accessKeySecret: process.env['TOS_SECRET_KEY'],
region: "Provide your region", // Fill in the region where the Bucket is located. For example, if it is in China North 2 (Beijing), fill in "Provide your region" as cn-beijing.
endpoint: "Provide your endpoint", // Enter the domain address.
});
function handleError(error) {
if (error instanceof TosClientError) {
console.log('Client Err Msg:', error.message);
console.log('Client Err Stack:', error.stack);
} else if (error instanceof TosServerError) {
console.log('Request ID:', error.requestId);
console.log('Response Status Code:', error.statusCode);
console.log('Response Header:', error.headers);
console.log('Response Err Code:', error.code);
console.log('Response Err Msg:', error.message);
} else {
console.log('unexpected exception, message: ', error);
}
}
async function main() {
try {
const bucketName = 'node-sdk-test-bucket'; // Target bucket
const objectName = 'example_dir/example.txt'; // Target object
// Downloaded file path
const filePath = './example_dir/example.txt';
await client.downloadFile({
bucket: bucketName,
key: objectName,
// Downloaded file path
filePath,
// Specify the size of each part during download (10M)
partSize: 10 * 1024 * 1024,
// Number of concurrent part download tasks
taskNum: 5,
// Specify the temporary file path for resumable downloads
checkpoint: './example_dir/example.checkpoint',
// Set a callback function to view the download progress using a custom method
dataTransferStatusChange: (event) => {
if (event.type === DataTransferType.Started) {
console.log('Data Transfer Started');
} else if (event.type === DataTransferType.Rw) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Once Read:${event.rwOnceBytes},ConsumerBytes/TotalBytes: ${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Succeed) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Data Transfer Succeed, ConsumerBytes/TotalBytes:${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Failed) {
console.log('Data Transfer Failed');
}
},
// Event listener callback
downloadEventChange: (event) => {
switch (event.type) {
case DownloadEventType.CreateTempFileSucceed: {
console.log('Download File Create Temp File Success.');
break;
}
case DownloadEventType.CreateTempFileFailed: {
console.log('Download File Create Temp File Fail.');
break;
}
case DownloadEventType.DownloadPartSucceed: {
console.log(`Download ${event.bucket} ${event.key} part success, DownloadPartInfo:%o`, event.downloadPartInfo);
break;
}
case DownloadEventType.DownloadPartAborted: {
console.log(`Download ${event.bucket} ${event.key} part aborted.`);
break;
}
case DownloadEventType.DownloadPartFailed: {
console.log(`Download ${event.bucket} ${event.key} part fail, err:%o`, event.err);
break;
}
case DownloadEventType.RenameTempFileSucceed: {
console.log(`Download ${event.bucket} ${event.key} success.`);
break;
}
case DownloadEventType.CreateTempFileFailed: {
console.log(`Download ${event.bucket} ${event.key} fail, err:%o`, event.err);
break;
}
}
},
});
const fileStat = await fsp.stat(filePath);
console.log('fileSize: ', fileStat.size);
} catch (error) {
handleError(error);
}
}
main();
Configure client side speed limit
When the breakpoint resumes the download, the client side can use the rateLimiterparameter to limit the bandwidth occupied by the downloaded data. The code is as follows.
// Import SDK, if the TOS Node.JS SDK version is less than 2.5.2, please change "TosClient" to "TOS" in the code below.
import { DataTransferType, createDefaultRateLimiter, TosClient, TosClientError, TosServerError } from '@volcengine/tos-sdk';
import fsp from 'fs/promises';
// Create a client.
const client = new TosClient({
accessKeyId: process.env['TOS_ACCESS_KEY'],
accessKeySecret: process.env['TOS_SECRET_KEY'],
region: "Provide your region", // Fill in the region where the Bucket is located. For example, if it is in China North 2 (Beijing), fill in "Provide your region" as cn-beijing.
endpoint: "Provide your endpoint", // Enter the domain address.
});
function handleError(error) {
if (error instanceof TosClientError) {
console.log('Client Err Msg:', error.message);
console.log('Client Err Stack:', error.stack);
} else if (error instanceof TosServerError) {
console.log('Request ID:', error.requestId);
console.log('Response Status Code:', error.statusCode);
console.log('Response Header:', error.headers);
console.log('Response Err Code:', error.code);
console.log('Response Err Msg:', error.message);
} else {
console.log('unexpected exception, message: ', error);
}
}
async function main() {
try {
const bucketName = 'node-sdk-test-bucket'; // Target bucket
const objectName = 'example_dir/example.txt'; // Target object
// The downloaded file path.
const filePath = './example_dir/example.txt';
const rateLimit1M = 1024 * 1024;
await client.downloadFile({
bucket: bucketName,
key: objectName,
// Download file path
filePath,
// Specify the chunk size during download to be 10M
partSize: 10 * 1024 * 1024,
// Number of concurrent chunk download tasks
taskNum: 5,
// Specify the temporary file path for resumable downloads
checkpoint: './example_dir/example.checkpoint',
// Set callback function to view download progress through custom methods
dataTransferStatusChange: (event) => {
if (event.type === DataTransferType.Started) {
console.log('Data Transfer Started');
} else if (event.type === DataTransferType.Rw) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Once Read:${event.rwOnceBytes},ConsumerBytes/TotalBytes: ${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Succeed) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Data Transfer Succeed, ConsumerBytes/TotalBytes:${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Failed) {
console.log('Data Transfer Failed');
}
},
// Limit the download speed to 1M/s on the client side while downloading the object.
rateLimiter: createDefaultRateLimiter(rateLimit1M, rateLimit1M),
});
const fileStat = await fsp.stat(filePath);
console.log('fileSize: ', fileStat.size);
} catch (error) {
handleError(error);
}
}
main();
Cancellation mechanism
The following code is used to cancel the ongoing breakpoint continuation download task at runtime.
// Import SDK, if the TOS Node.JS SDK version is less than 2.5.2, please change "TosClient" to "TOS" in the code below.
import { DataTransferType, CancelToken, TosClient, TosClientError, TosServerError, isCancel } from '@volcengine/tos-sdk';
import fsp from 'fs/promises';
// Create a client.
const client = new TosClient({
accessKeyId: process.env['TOS_ACCESS_KEY'],
accessKeySecret: process.env['TOS_SECRET_KEY'],
region: "Provide your region", // Fill in the region where the Bucket is located. For example, if it is in China North 2 (Beijing), fill in "Provide your region" as cn-beijing.
endpoint: "Provide your endpoint", // Enter the domain address.
});
function handleError(error) {
if (error instanceof TosClientError) {
console.log('Client Err Msg:', error.message);
console.log('Client Err Stack:', error.stack);
} else if (error instanceof TosServerError) {
console.log('Request ID:', error.requestId);
console.log('Response Status Code:', error.statusCode);
console.log('Response Header:', error.headers);
console.log('Response Err Code:', error.code);
console.log('Response Err Msg:', error.message);
} else {
console.log('unexpected exception, message: ', error);
}
}
async function main() {
// checkpoint file path
const checkpointFilePath = './example_dir/example.checkpoint';
try {
const bucketName = 'node-sdk-test-bucket'; // Target bucket
const objectName = 'example_dir/example.txt'; // Target object
// Downloaded file path
const filePath = './example_dir/example.txt';
try {
// Generate cancelTokenSource.
const cancelTokenSource = CancelToken.source();
// Cancel task after 1 second.
setTimeout(() => {
cancelTokenSource.cancel();
console.log('cancel downloadFile');
}, 1_000);
await client.downloadFile({
bucket: bucketName,
key: objectName,
// Downloaded file path
filePath,
// Specify the chunk size during download (1M)
partSize: 1 * 1024 * 1024,
// Number of concurrent chunk download tasks
taskNum: 5,
// Specify the temporary file path for resumable download
checkpoint: './example_dir/example.checkpoint',
// Set a custom callback function to monitor the download progress
dataTransferStatusChange: (event) => {
if (event.type === DataTransferType.Started) {
console.log('Data Transfer Started');
} else if (event.type === DataTransferType.Rw) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Once Read:${event.rwOnceBytes},ConsumerBytes/TotalBytes: ${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Succeed) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Data Transfer Succeed, ConsumerBytes/TotalBytes:${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Failed) {
console.log('Data Transfer Failed');
}
},
cancelToken: cancelTokenSource.token,
});
} catch (error) {
if (!isCancel(error)) {
throw error;
}
const checkpointFileContent = await fsp.readFile(checkpointFilePath, 'utf-8');
console.log(`checkpoint file's content after cancel: `, checkpointFileContent);
console.log('Continue to download');
// You can resume downloading through the checkpoint file.
await client.downloadFile({
bucket: bucketName,
key: objectName,
// Downloaded file path
filePath,
// Specify the chunk size during download (1M)
partSize: 1 * 1024 * 1024,
// Number of concurrent chunk download tasks
taskNum: 5,
// Specify the temporary file path for resumable downloads
checkpoint: './example_dir/example.checkpoint',
// Set a custom callback function to monitor download progress
dataTransferStatusChange: (event) => {
if (event.type === DataTransferType.Started) {
console.log('Data Transfer Started');
} else if (event.type === DataTransferType.Rw) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Once Read:${event.rwOnceBytes},ConsumerBytes/TotalBytes: ${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Succeed) {
const percent = ((event.consumedBytes / event.totalBytes) * 100).toFixed(2);
console.log(`Data Transfer Succeed, ConsumerBytes/TotalBytes:${event.consumedBytes}/${event.totalBytes},${percent}%`);
} else if (event.type === DataTransferType.Failed) {
console.log('Data Transfer Failed');
}
},
});
const fileStat = await fsp.stat(filePath);
console.log('fileSize: ', fileStat.size);
}
} catch (error) {
handleError(error);
}
}
main();
Last updated: 2025.09.22 18:35:48
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
