Docs

Torch Object Storage

Overview

What is hierarchical namespace

A hierarchical namespace (HNS) bucket is a new type of bucket introduced by TOS. It uses a hierarchical structure to organize metadata. Besides its hierarchical namespace features, an HNS bucket can scale as much as a flat namespace (FNS) bucket. It also delivers seamless interoperability between object and file semantics. This allows data to be accessed through multiple protocols, so you can process data more efficiently.

HNS buckets can do everything FNS buckets can do, plus more. They support directory-level mv and rename operations. They also improve common read operations, like List and Head for directories, to make data processing more efficient and faster. This makes them well-suited for use cases in big data, data lakes, and AI.

Supported regions

The following regions currently support hierarchical namespace:

China (Beijing)

China (Guangzhou)

China (Shanghai)

Asia Pacific (Johor)

tip

If you need to use this feature in a different region, submit a ticket or contact your account manager for assistance.

Supported features

Currently, HNS buckets can be accessed via the TOS console, APIs, and SDKs. For details on the features supported by each method of access, see the following documents:

Supported console features (HNS buckets)

Supported APIs (HNS buckets)

Supported SDKs (HNS buckets)

Supported tools (HNS buckets)

How it works

The following diagram shows how HNS buckets work.

The following diagram shows an example of an HNS bucket, in which objects are organized into a folder hierarchy.

Benefits

Hierarchical namespace provides the following benefits:

Rapid integration: Use TOS HNS buckets with your existing big data analytics applications, such as Hadoop and Spark, without making any modifications. With the proper configuration, TOS operates similarly to native HDFS, allowing you to leverage its extensive storage capacity, elastic scalability, security, reliability, and availability.

Protocol interoperability: Data written to an HNS bucket via object semantics, such as the S3 API, can be read directly through HDFS-compatible methods. Likewise, data written in HDFS can be read using object semantics. Besides its hierarchical namespace features, an HNS bucket can scale as much as a flat namespace (FNS) bucket. It also delivers seamless interoperability between object and file semantics. This allows data to be accessed through multiple protocols, so you can process data more efficiently.

Performance optimization: HNS buckets offer high-performance, atomic directory and file operations. They significantly improve directory rename performance, completing the task in milliseconds for both large directories and files. Compared to FNS buckets, latency is reduced by over 99%.

Ecosystem compatibility: HNS buckets easily integrate with various open-source tools within the Volcano Engine ecosystem. This allows users to create flexible big data and AI pipelines based on TOS. They are compatible with the Volcano Engine big data platform and its components, including E-MapReduce (EMR), Lake AI Service (LAS), Cloud File System (CloudFS), and Machine Learning Platform (AML), as well as open-source tools such as Hadoop and Spark. They also support integration with leading cache solutions, such as JuiceFS and Alluxio.

Use cases

Below are some common scenarios in which you should consider using hierarchical namespace:

Cloud-native data lakehouse

Offline/batch data analysis

Real-time data warehouse

Data migration

Engine support

Supported engines

Engine/Platform	Related documentation
Spark	Using Proton with Spark
Flink	Using Proton with Flink
Hive	Using Proton with Hive
Impala	Using Proton with Impala
Hadoop	Using Proton with Hadoop