Docs

Torch Object Storage

TOS PyTorch Connector

Configuring TOS PyTorch Connector

TOS PyTorch Connector requires valid access credentials in order to authenticate requests when interacting with TOS. This topic explains how to configure the connector's authentication information and parameters.

Prerequisites

You have installed TOS PyTorch Connector. For instructions, see Installing TOS PyTorch Connector.

Authentication

To securely access TOS, you must provide valid access credentials: your access key ID (AK) and secret access key (SK). Regarding TOS PyTorch Connector, we recommend passing the credentials as parameters in the code explicitly. Create a CredentialProvider(ak=ak, sk=sk) object and pass it directly via the cred parameter when instantiating TosMapDataset, TosIterableDataset, or TosCheckpoint. This approach offers strong code portability and a clear credential scope, and it does not pollute the global environment.

Configuration

The easiest and recommended way is to pass the access credentials directly to TOS PyTorch Connector via CredentialProvider.

Obtaining an AK/SK pair

Your AK and SK are your security credentials for accessing TOS. Keep them safe.

To obtain them: Log in to the Volcano Engine console. Then, go to IAM > Access Key Management to create and manage your AK/SK pair.

Authorization: For data security, strictly adhere to the principle of least privilege. Assign specific permission policies to your AK and SK. If your application only needs to read training data, grant it the read-only permission (TosReadOnlyAccess). This effectively prevents risks of data tampering or deletion caused by key leakage.

Passing credentials

After getting your AK and SK, you can pass them through the cred parameter when initializing TOS PyTorch Connector classes.

The table below explains the basic configuration parameters.

Parameter	Required	Type	Example value	Description
region	Yes	str	cn-beijing	TOS region ID.
endpoint	No	str	http://tos-cn-beijing.ivolces.com	TOS endpoint.
cred	No	CredentialProvider	CredentialProvider('ak', 'sk')	Used to pass credentials.
ak	Yes	str	AKLT***********	Access key ID. Parent node: cred.
sk	Yes	str	WIRk***********	Secret access key. Parent node: cred.
client_conf	No	TosClientConfig	-	Used to configure the connector.
part_size	No	int	8388608	The size of each part during a multipart upload or download. Parent node: client_conf.
max_retry_count	No	int	3	The maximum number of retries for a single request. Parent node: client_conf.
shared_prefetch_tasks	No	int	32	The number of concurrent range reads for sequential access. Parent node: client_conf.
shared_upload_part_tasks	No	int	32	The number of concurrent multipart uploads. Parent node: client_conf.
enable_crc	No	bool	True	Checks data integrity for uploads and sequential reads.

Code examples

TOS PyTorch Connector offers out-of-the-box datasets and checkpoints. With simple configuration, you can load datasets and access checkpoints. Replace the parameters, such as region, endpoint, AK, and SK, with your actual values.

Using a dataset

from tostorchconnector import CredentialProvider​
from tostorchconnector import TosMapDataset​
​
# Dataset transformation function​
def transform(item):​
    data = item.read()​
    return item.key, item.size​
​
# Dataset path​
dataset_uri = 'tos://bucket/prefix'​
region = 'cn-beijing'​
# TOS endpoint​
endpoint = 'http://tos-cn-beijing.ivolces.com'​
ak = '<your-access-key>'​
sk = '<your-secret-key>'​
​
with TosMapDataset.from_prefix(dataset_uri,​
                                    region=region, endpoint=endpoint,​
                                    transform=transform,​
                                    cred=CredentialProvider(​
                                        ak=ak,​
                                        sk=sk)) as dataset:​
​
    # Retrieve the size of the dataset​
    dataset_len = len(dataset)​
    # Randomly access a specific item​
    item = dataset[0]​
    bucket = item.bucket​
    key = item.key​
    # Read all data​
    content = item.read()​
    print(len(content))​

Using a checkpoint

import torch​
from tostorchconnector import TosCheckpoint, CredentialProvider​
​
checkpoint_uri = 'tos://bucket/key'​
region = 'cn-beijing'​
endpoint = 'http://tos-cn-beijing.ivolces.com'​
ak = '<your-access-key>'​
sk = '<your-secret-key>'​
​
with TosCheckpoint(region, endpoint, cred=CredentialProvider(ak, sk)) as checkpoint:​
    # Write data​
    with checkpoint.writer(checkpoint_uri) as writer:​
        writer.write(b'hello')​
        writer.write(b'world')​
    ​
    # Read data​
    with checkpoint.reader(checkpoint_uri) as reader:​
        data = reader.read()​
        assert data == b'helloworld'​

Last updated: 2026.03.23 11:37:41