This topic explains how to access and store TOS data in PyTorch jobs.

Prerequisites

Before you begin using TOS, make sure that you have created a Volcano Engine account and completed real name verification.

Make sure you have activated TOS and created a TOS bucket. For instructions, see Quick start.

Your operating system is Linux x86-64.

You have installed Python 3.8 or a later version.

Environment

Before connecting to TOS, install PyTorch in a Python 3.8+ environment. For more information, see Pytorch.

Use the example below to verify that PyTorch was installed successfully.

Integration methods

Integration method	Description
Using fsspec	You can encapsulate TOS as Python's standard read/write interfaces using tosfs, which implements the fsspec specification. Then, use the torchdata library to read from and write to the local path. For more information, see tosfs.
Using FSX	FSX supports mounting TOS buckets as a local file system on Linux. This approach is ideal for scenarios requiring file system semantics to access bucket data, such as when training model data in AI/machine learning applications that depend on file system operations. FSX is currently in closed beta. Please submit a ticket for access. For information on the operating environment and any restrictions, see FSX Overview.
Using S3 Connector	S3 Connector, provided by Amazon Web Services (AWS), is a connector for PyTorch. It enhances the capabilities of torchdata for accessing S3 and supports map-style and iterable-style datasets, as well as checkpointing interfaces. For more information, see S3-connector-for-pytorch. Currently, only FNS buckets are supported. For more information, see Bucket type.
Using tosutil	You can use tosutil to mount a TOS bucket as a local file system directory. Then, use the torchdata library to read from and write to the local path.

Steps

The steps for different integration methods are as follows.

fsspec

FSX

S3 Connector

tosutil

Run this command to install tosfs.

# Install tosfs for fsspec
pip install tosfs

Run this command to initialize the client.

from tosfs.core import TosFileSystem

# Configure access credentials
if __name__ == '__main__':
tosfs = TosFileSystem(
key='<Your Volces AK>',
secret='<Your Volces SK>',
endpoint_url='http://tos-cn-beijing.volces.com',
region='cn-beijing',
)

Run this code to construct an iterable-style dataset.

tip

In this example, we use the bucketname bucket in the Beijing region. Modify the code according to your workloads.

from torchdata.datapipes.iter import FSSpecFileLister, FSSpecFileOpener

if __name__ == '__main__':
kwargs = {
'key' : '<Your Volces AK>',
'secret' : '<Your Volces SK>',
'endpoint_url' : 'https://tos-cn-beijing.volces.com',
'region' :'cn-beijing'
}

bucket = 'bucketname'
print('test iterable-style datasets')
file_lister = FSSpecFileLister(root='tos://'+ bucket+'/shard-data/', **kwargs)
iterable_dataset = FSSpecFileOpener(file_lister, mode='rb', **kwargs)
# Iterate and read objects in the dataset
for _, item in iterable_dataset:
print(item.bucket, item.key)
data = item.read()
print(len(data))

Run this code to save or load the checkpoint.

tip

In this example, we use the bucketname bucket in the Beijing region. Modify the code according to your workloads.

import torch
import torchvision
from tosfs.core import TosFileSystem

if __name__ == '__main__':
tosfs = TosFileSystem(
key='<Your Volces AK>',
secret='<Your Volces SK>',
endpoint_url='http://tos-cn-beijing.volces.com',
region='cn-beijing',
)

bucket = 'bucketname'
print('test checkpointing')
model = torchvision.models.resnet18()

# Encapsulate TOS as Python's standard read/write interfaces
with tosfs.open(bucket+ '/checkpoint/epoch1.ckpt', 'wb') as writer:
torch.save(model.state_dict(), writer)

with tosfs.open(bucket+'/checkpoint/epoch1.ckpt', 'rb') as reader:
stat_dict = torch.load(reader)

model.load_state_dict(stat_dict)
print(model)

This section uses veLinux to show how to install FSX and mount a bucket via FSX. For more information on environments and parameters, see the following documents:

Downloading and installing FSX

Configuration and mounting

Mount parameters

Install an FSX client.

For operating system-specific installation packages, submit a ticket to contact the technical team for the latest builds.

Run this command to install the client.

dpkg -i fsx_amd64.deb

Run this command to check the client version.

fsx -v

The following version information indicates a successful installation.

Version: 1.1.0.400

Configure the necessary permissions.

To mount a bucket, the relevant IAM user must have ListBucket and HeadBucket permissions. To read objects within the mounted bucket, the user needs GetObject permission. To write objects, PutObject and AbortMultipartUpload permissions are required.

The example below shows a minimal set of permissions. For more information, see Permissions.

{
"Statement": [{
"Effect": "Allow",
"Action": [
"tos:GetObject",
"tos:PutObject",
"tos:DeleteObject",
"tos:AbortMultipartUpload",
"tos:ListBucket",
"tos:HeadBucket"
],
"Resource": [
"*"
]
}]
}

Mount the bucket.

For more information on mount parameters, see Configuration and mounting.

Type this command into your Linux terminal to create a mount point.

mkdir /mnt/point

Run this command to mount the bucket.

mount -t fsx /tos/<bucket name> <mount point> -o region="<region ID>",endpoint="<endpoint>",access_key_id="AK",secret_access_key="SK",no_writeback_cache

Example:

mount -t fsx /tos/bucketname /mnt/point -o region="cn-beijing",endpoint="http://tos-cn-beijing.volces.com",access_key_id="AKAKAK****",secret_access_key="SKSKSK****",no_writeback_cache

Run this command to verify that the mount was successful.

df -h

If successful, the command returns output containing the mount point and bucket name.

Filesystem Size Used Avail Use% Mounted on
BFA281845F:tos-fuse-test 8.0E 0 8.0E 0% /mnt/fuse_shm

Run this code to construct an iterable-style dataset.

tip

In this example, we use the shard-data/ directory. Modify the code according to your workloads.

from torchdata.datapipes.iter import FileLister, FileOpener

if __name__ == '__main__':
dataset_uri = './root/shard-data/'

print('test iterable-style datasets')
file_lister = FileLister(root=dataset_uri)
iterable_dataset = FileOpener(file_lister, mode='rb')
# Iterate and read objects in the dataset
for _, item in iterable_dataset:
print(item.bucket, item.key)
data = item.read()
print(len(data))

Run this code to save or load the checkpoint.

tip

In this example, we use the checkpoint/ directory. Modify the code according to your workloads.

import torch
import torchvision

if __name__ == '__main__':
print('test checkpointing')
model = torchvision.models.resnet18()

# Treat TOS directly as a local file
with open('./root/checkpoint/epoch0.ckpt', 'wb') as writer:
torch.save(model.state_dict(), writer)

with open('./root/checkpoint/epoch0.ckpt', 'rb') as reader:
stat_dict = torch.load(reader)

model.load_state_dict(stat_dict)
print(model)

Install S3 Connector.

Run this command to install the s3 connector base library.

pip install s3torchconnector

(Optional) Run this command to install the s3 connector for lighting.

pip install s3torchconnector[lightning]

Configure the environment.

Run this command to edit the credentials configuration file.

vim ~/.aws/credentials

Run this command to provide your credentials.

[default]
aws_access_key_id = <Your Volces AK>
aws_secret_access_key = <Your Volces SK>

Run this command to save the configurations.

:wq

Run this code to construct an iterable-style dataset.

tip

In this example, we use the shard-data/ directory of the bucketname bucket in the Beijing region. Modify the code according to your workloads.

from s3torchconnector import S3IterableDataset, S3ClientConfig

if __name__ == '__main__':
dataset_uri = 's3://bucketname/shard-data/'
region = 'cn-beijing'
endpoint = 'http://tos-s3-cn-beijing.volces.com'
s3_config = S3ClientConfig(force_path_style=False)

print('test iterable-style datasets')
iterable_dataset = S3IterableDataset.from_prefix(dataset_uri,
region=region,
endpoint=endpoint,
s3client_config=s3_config)
# Iterate and read objects in the dataset
for item in iterable_dataset:
print(item.bucket, item.key)
data = item.read()
print(len(data))

Run this code to construct a map-style dataset.

tip

In this example, we use the shard-data/ directory of the bucketname bucket in the Beijing region. Modify the code according to your workloads.

from s3torchconnector import S3IterableDataset, S3ClientConfig

if __name__ == '__main__':
dataset_uri = 's3://bucketname/shard-data/'
region = 'cn-beijing'
endpoint = 'http://tos-s3-cn-beijing.volces.com'
s3_config = S3ClientConfig(force_path_style=False)

print('test map-style datasets')
map_dataset = S3MapDataset.from_prefix(dataset_uri,
region=region,
endpoint=endpoint,
s3client_config=s3_config)
# Iterate and read objects in the dataset
for item in map_dataset:
print(item.bucket, item.key)
data = item.read()
print(len(data))

# Read objects from the dataset by index.
item = map_dataset[0]
bucket = item.bucket
key = item.key
print(bucket, key)
data = item.read()
print(len(data))

Run this code to save or load the checkpoint.

tip

In this example, we use the checkpoint/ directory of the bucketname bucket in the Beijing region. Modify the code according to your workloads.

import torch
import torchvision
from s3torchconnector import S3Checkpoint, S3ClientConfig

if __name__ == '__main__':
checkpoint_uri = 's3://bucketname/checkpoint/'
region = 'cn-beijing'
endpoint = 'http://tos-s3-cn-beijing.volces.com'
s3_config = S3ClientConfig(force_path_style=False)

print('test checkpointing')
checkpoint = S3Checkpoint(region=region,
endpoint=endpoint,
s3client_config=s3_config)

model = torchvision.models.resnet18()

with checkpoint.writer(checkpoint_uri + 'epoch0.ckpt') as writer:
torch.save(model.state_dict(), writer)

with checkpoint.reader(checkpoint_uri + 'epoch0.ckpt') as reader:
state_dict = torch.load(reader)

model.load_state_dict(state_dict)

This section uses tosutil to show how to mount a bucket using Filesystem in Userspace (FUSE). However, you can select any alternative tool that best fits your needs.

Install tosutil. For instructions, see Downloading and installing tosutil.

Configure tosutil and check its version and connectivity. For instructions, see Quick start.

Use the mount command to mount the bucket and verify the mount result. For instructions, see Mount a bucket as a local file system directory (mount).

This section demonstrates how to mount the bucketname bucket to the ./root directory.

./tosutil mount tos://bucketname ./root -aow -ar -ao

Run this command to verify that the mount was successful.

ls ./root

Run this code to construct an iterable-style dataset.

tip

In this example, we use the shard-data/ directory. Modify the code according to your workloads.

from torchdata.datapipes.iter import FileLister, FileOpener

if __name__ == '__main__':
dataset_uri = './root/shard-data/'

print('test iterable-style datasets')
file_lister = FileLister(root=dataset_uri)
iterable_dataset = FileOpener(file_lister, mode='rb')
# Iterate and read objects in the dataset
for _, item in iterable_dataset:
print(item.bucket, item.key)
data = item.read()
print(len(data))

Run this code to save or load the checkpoint.

tip

In this example, we use the checkpoint/ directory. Modify the code according to your workloads.

import torch
import torchvision

if __name__ == '__main__':
print('test checkpointing')
model = torchvision.models.resnet18()

# Treat TOS directly as a local file
with open('./root/checkpoint/epoch0.ckpt', 'wb') as writer:
torch.save(model.state_dict(), writer)

with open('./root/checkpoint/epoch0.ckpt', 'rb') as reader:
stat_dict = torch.load(reader)

model.load_state_dict(stat_dict)
print(model)

Torch Object Storage