This article mainly introduces common methods and parameter summaries of the two core classes: TosFileSystem and TosFile.

tosfsspec.TosFileSystem

TosFileSystem is the main entry point for interacting with TOS, providing all top-level file system operations.

Constructor

Common methods

Constructor parameter table

Parameter	Item	Default value	Description
region	str	Yes	Region where the target Bucket is located, for example, 'cn-beijing'.
endpoint	str	''	TOS external access domain name. If empty, the default Endpoint will be used based on region; specify explicitly here for intranet access or custom domain.
key	str	''	Access Key ID for accessing TOS. Explicit configuration is recommended.
secret	str	''	Secret Access Key for accessing TOS.
part_size	int	8 * 1024 * 1024	Base part size (bytes) for multipart upload and multipart copy. tip part_size must be greater than or equal to 4 * 1024 * 1024, or a ValueError will be raised.
max_retry_count	int	3	Maximum number of retries for a failed request.
shared_prefetch_tasks	int	32	Number of concurrent prefetch tasks during read.
shared_upload_part_tasks	int	32	Number of concurrent tasks during multipart upload.
shared_upload_part_copy_tasks	int	32	Number of concurrent tasks during multipart copy.
enable_crc	bool	True	Enable CRC64 checksum. Enabled for data integrity verification after upload and download.

Common methods

The following lists common methods of TosFileSystem, key parameters, and corresponding simple examples. All examples assume instance initialization is complete:

open(self, path, mode='rb', block_size=None, autocommit=True, **kwargs)

Purpose: open an object in TOS and return a TosFile file object.

Key parameters:

path: can be in the form of 'tos://bucket/key' or 'bucket/key'; using the protocol form is recommended.

mode: supports binary modes only: 'rb', 'wb', 'ab'.

block_size: block size (bytes) used for reading and writing. Must be at least 4MB in write mode; if not explicitly set, it falls back to part_size.

Other common keyword parameters: In write mode, you can explicitly specify the object's ContentType using Content-Type.

Code examples:

ls(path, detail=True, sort_names=True)

Purpose: List files and directories under the specified path.

Key parameters:

path: Lists all Buckets when '/' or '' is used; lists the prefixes under the Bucket when 'tos://bucket/prefix/' is used.

detail: Returns a list of dictionaries with metadata if True; otherwise returns a list of string paths.

Code examples:

info(path, **kwargs) / stat(path)

Purpose: Retrieve detailed metadata of a single file or directory.

exists(path, **kwargs)

Purpose: determine whether the given path exists.

checksum(path)

Purpose: get the checksum of the current object version. File types use the object's ETag, and directory types use the directory information hash.

Code example:

isdir(path) / isfile(path)

Purpose: determine whether the path is a directory or a file.

Code example:

glob(path, maxdepth=None, **kwargs)

Purpose: Use wildcard patterns to match paths and return a list of matched files.

find(path, maxdepth=None, withdirs=False, detail=False, **kwargs)

Purpose: List all files (and optionally directories) under a path, usually to implement custom traversal logic.

walk(path, maxdepth=None, detail=False, **kwargs)

Purpose: Recursively traverse the directory tree, returning a (root, dirs, files) generator.

Code examples:

put_file(lpath, rpath, callback=None, **kwargs)

Purpose: Upload a local file to the specified path in TOS.

Key parameters:

lpath: Local file path.

rpath: target TOS path, usually 'tos://bucket/key'.

callback: optional progress callback implementing the set_size() and relative_update() interfaces.

Other common keyword arguments: explicitly specify the object's Content-Type via ContentType. If not specified, it will be automatically inferred based on the local file extension.

Code examples:

get_file(rpath, lpath, callback=None, **kwargs)

Purpose: download remote objects to a local file.

Key parameters:

rpath: source TOS path.

lpath: local save path; intermediate directories will be automatically created if not existing.

callback: optional progress callback.

Code examples:

cp_file(path1, path2, reserve_metadata=False, **kwargs)

Purpose: copy a single object within TOS.

Key parameters:

path1: source object path.

path2: destination object path.

reserve_metadata: if True, preserves source object's metadata as much as possible.

copy(path1, path2, recursive=False, maxdepth=None, on_error=None, reserve_metadata=False, **kwargs)

Purpose: copy one or more paths within TOS, supporting recursive directory copying and wildcards.

mv(path1, path2, recursive=False, maxdepth=None, reserve_metadata=False, **kwargs)

Purpose: Move (rename) objects or directories within TOS. For HNS Buckets, directory moves are usually atomic; for FNS Buckets, the operation may involve copy and delete.

rm(path, recursive=False, maxdepth=None)

Purpose: Delete a single path or a group of paths. path can be a string or a list of strings.

rmdir(path, **kwargs)

Purpose: Delete an empty directory; when delete_bucket=True is passed, the entire Bucket can be deleted.

mkdir(path, create_parents=True, create_bucket=False, **kwargs)

Purpose: Create a directory; when create_bucket=True and the path contains only the Bucket name, a Bucket creation is attempted.

makedirs(path, exist_ok=False)

Purpose: recursively create directory structure; no error if directory exists when exist_ok=True.

touch(path, truncate=True, **kwargs)

Purpose: create a zero-byte object or truncate an existing object.

Key parameters:

truncate: raises an exception if set to False and the object exists.

content_type: optional; explicitly specify Content-Type. If not specified, inferred automatically based on the key suffix.

Code examples:

close()

Purpose: close the underlying client connection and release resources. Usually called automatically when the process exits; call explicitly to release resources early.

Code examples:

tosfsspec.TosFile

TosFile is a file class object returned by fs.open(), providing standard I/O methods.

Constructor

Common methods

Method/Attribute	Description
read(size=-1)	Read the specified number of bytes from the current position. size=-1 means read until the end of the file.
readinto(buf)	Read data directly into the writable buffer object buf and return the actual number of bytes read, avoiding extra memory copying.
readuntil(char=b'\n', blocks=None)	Continuously read data until the specified delimiter char (including the delimiter) or the end of the file is reached.
write(data)	Write the byte data data into the file buffer.
seek(loc, whence=0)	Move the file pointer to the specified position. whence can be 0 (beginning), 1 (current position), or 2 (end of file).
tell()	Return the current position of the file pointer.
flush(force=False)	Flushes the accumulated data in the buffer to the underlying write stream. Usually called automatically by commit() or when closing the file.
commit()	Commits all data in the current buffer. Explicit calls are generally unnecessary for write modes with auto-commit enabled.
discard()	Abandons the current write session, terminates the underlying shard upload, and discards uncommitted data.
close()	Closes the file and ensures all data is written. Automatically called when exiting the with statement block.
readable() / writable() / seekable()	Return boolean values indicating whether the file supports reading, writing, or random access.
closed	Indicates whether the file is closed.
bucket	Name of the bucket to which the current file belongs.
key	Object key of the current file within the bucket.

Code examples

Torch Object Storage