Docs

Torch Object Storage

TOS PyTorch Connector

Overview

Designed for AI training workloads, TOS PyTorch Connector delivers high-performance data access to TOS data. By enabling efficient reads of large-scale training data, it dramatically accelerates data loading for machine learning and deep learning projects using the PyTorch framework.

Benefits

Item	TOS PyTorch Connector	Other tools (e.g., Python SDK and FSX)
Performance	TOS PyTorch Connector delivers high performance, with optimizations specifically designed for loading datasets and reading and writing checkpoints.	Alternatives lack these targeted optimizations, resulting in lower performance.
Data loading method	TOS PyTorch Connector supports streaming loading, which effectively reduces costs and overhead.	You need to pre-download or pre-warm data, which adds extra costs and management burden.
Data access	TOS PyTorch Connector directly reads from and writes to TOS via HTTP, offering the fastest access.	FSX, for instance, adds a layer of complexity by requiring an extra protocol translation via FUSE.
Ease of use	TOS PyTorch Connector provides simple configuration for dataset and checkpoint support with the PyTorch framework.	You need to write custom code to encapsulate datasets and checkpoints, which increases development complexity.

Features

TOS PyTorch Connector offers the following three key features for the PyTorch framework:

Feature	Description	Related class	Method
Map-style dataset	Enables random data access, allowing you to quickly retrieve specific samples during training.	TosMapDataset	Both TosMapDataset and TosIterableDataset support the same methods. from_prefix() Constructs a dataset using a TOS_URI prefix. This is ideal for scenarios where TOS paths follow a consistent pattern. from_urls() Constructs a dataset from a list of TOS URIs. This is ideal for scenarios where TOS paths are specific but scattered.
Iterable-style dataset	Supports streaming sequential access, efficiently processing large, continuous data streams.	TosIterableDataset
Checkpointing interface	Loads checkpoints from TOS during model training and saves checkpoints directly to TOS after each training cycle.	TosCheckpoint	TosCheckpoint() Initializes a TosCheckpoint object to load checkpoints from TOS during model training and save checkpoints directly to TOS after each training cycle. TosObjectReader() Reads a checkpoint from TOS. TosObjectWriter() Writes a checkpoint to TOS.

How it works

The diagram below illustrates how to use TOS PyTorch Connector to train PyTorch models with TOS data.

Steps

Install TOS PyTorch Connector.

Configure TOS PyTorch Connector.

Use TOS PyTorch Connector.

Use TosMapDataset to construct a map-style dataset for random access.

Use TosIterableDataset to construct an iterable-style dataset for streaming sequential access.

Use TosCheckpoint to save and load checkpoints.