Docs

File Storage vePFS

Best Practices

DeepSeek 3FS and vePFS architecture analysis and comparison

This article extracts representative cases from AI and other scenarios, and conducts a practical comparison between 3FS and vePFS in both metadata and data dimensions.

Comparative analysis

Mainstream solutions for AI workloads

In AI scenarios, the workflow transforms raw data through several stages to meet AI requirements:

Data collection stage: Data is collected using sampling, crawling, and other tools based on business needs. Invalid or abnormal data is cleaned according to specific rules to generate raw data for training. The input and output data at this stage consist of vast, unstructured data.

Data preprocessing stage: Big data processing suites and related toolchains are used to label and categorize raw data, creating foundational training data. The input and output data at this stage consist of large volumes, unstructured data, and mixed file sizes.

Training phase: After segmenting or recognizing data, such as text and images, convert them into numerical symbol sequences recognizable by neural networks. Through single or multi-node parallel training, model vector data is generated. This phase features input data that is massive, unstructured, and mixed with files of various sizes. The output is a batch of model files.

Inference phase: Using the trained model files, new content is generated through single or multi-node inference. This phase accepts model files and user requests as input and outputs results produced by the large model’s inference.

Research into domestic large model solutions reveals that, except for a few companies building infrastructure in dedicated IDCs, most opt for elastic computing and storage resources from public cloud services. Examining storage solutions offered by domestic and global public cloud vendors for AI scenarios, object storage primarily addresses the need to store vast amounts of files during data collection and preprocessing. File storage mainly supports high-performance concurrent processing in training and inference phases. Both work together as the foundational storage for large models. Comparing storage solutions used by large model vendors, file storage products mainly follow three technical approaches:

Self-developed distributed file storage: Vendors use fully proprietary file storage products or ones adapted from open-source software to support internal training and inference tasks. Examples include the Tectonic file system developed by Meta, the 3FS file system developed by DeepSeek, or custom systems based on HDFS and Ceph. They also leverage robust internal toolkits to integrate the storage and business systems.

Parallel file storage: Vendors use mature parallel file storage products from the public cloud for training and inference. Examples include Volcano Engine vePFS, Alibaba CPFS, AWS FSx for Lustre, and Tencent CFS Turbo. Data transfer modules bridge the flow between parallel file storage and object storage.

File cache acceleration layer: Vendors deploy JuiceFS or Alluxio on virtual machines, or use similar products from the public cloud, such as Volcano Engine CloudFS, Baidu RapidFS, and Tencent GooseFS. The file storage cache layer delivers high-speed access. Cache mapping resolves data transfer between file storage and object storage.

Based on the summary above, 3FS is a typical example of a self-developed distributed file system, while Volcano Engine file storage vePFS represents the parallel file storage category. The following sections briefly introduce the architecture of these two products, and then use case designs for the metadata and data planes to conduct scenario-based comparative tests.

Introduction to comparative system architecture

vePFS

The client uses a private protocol:

The architecture is similar to FUSE and uses kernel space and user space collaboration. The kernel handles integration with the VFS framework and implements the standard POSIX interface, while user space manages interactions with backend metadata and storage services. Communication between kernel space and user space is handled via shared memory.

Distributed locking ensures data consistency across multiple clients, while clients leverage various metadata and data caches to accelerate concurrent reading efficiency.

The metadata service adopts a distributed symmetric architecture:

Storage nodes and metadata nodes are unified, with each storage node also serving as a metadata node.

Metadata for the file system is maintained jointly by client nodes and metadata nodes. The former provides entry points for metadata updates, while the latter handles metadata read/write requests and tracks file open states.

The directory tree is organized using a Hash structure, distributed across different metadata nodes. When a Hash bucket is full, it automatically splits.

Fileset is supported, allowing the system to create independent metadata spaces with separate inode spaces and management capabilities. This feature enables users to manage different business scenarios with greater precision.

The storage service processes user read and write IO requests:

Both replica and EC modes are supported.

Files are stored in blocks, and different strategies are applied based on the size of the write IO. Large IO allocates a complete chunk for the written data and then updates the mapping information above. Data from small IO writes is first saved to the WAL. Once the log is flushed to disk, the system returns to the upper layer. The background attempts to merge IO operations and eventually writes the data to the chunk.

Similar to 3FS, one-sided communication is supported when using RDMA.

Volcano Engine further integrates a range of enterprise-grade features into its products.

The performance of vePFS scales linearly with capacity. With RDMA protocol, it achieves throughput bandwidth up to TiB/s, tens of millions of IOPS, and sub-millisecond latency performance indicators. It also supports access via TCP protocol, allowing flexible adaptation to various integration methods.

Provides data insight capabilities to help users effectively identify data hotspots and maximize the value of hot data.

Offers bidirectional synchronization with object storage through flexible data movement strategies, reducing storage costs.

For data security, it supports audit logs at both resource management and file access levels and allows directory-level access control for IAM sub-users.

Supports Fileset-based directory-level quota management and QoS control, effectively isolating resource access.

Ensures service stability by adopting a multi-level reliability architecture to maintain efficient and stable operation of the file system.

Deeply integrates with Volcano Engine's machine learning platform MLP and container service VKE, providing complete training and inference capabilities.

Comparison with the 3FS architecture

vePFS and 3FS provide a brief comparison of architectural differences across the three main components: client, metadata service, and storage service.

Client:

The vePFS proprietary protocol client uses a collaborative design between kernel mode and user mode, similar in principle to the implementation of the 3FS FUSE Client.

Metadata service:

vePFS adopts a classic randomized directory distribution design, a strategy widely used by systems such as Lustre DNE, BeeGFS, and WekaFS. In this design, the entire directory tree is distributed across all metadata nodes according to directory partitioning, with each metadata node responsible for part of the directory. The correctness of cross-node actions is ensured by distributed locking.

3FS adopts a classic design of separating metadata serving and computation, with similar approaches widely used by Tectonic, ADSL, and HopsFS. In this design, directory tree data is abstracted into key-value pairs or database records, persisted in an underlying distributed KV or database, while a metadata service layer at the top provides file system semantics. The correctness of cross-node actions is ensured by distributed transactions.

Storage Service:

vePFS supports two storage modes: replication and EC. These are the industry’s most mainstream storage solutions. Typically, the replication mode delivers higher performance, while the EC mode offers a higher disk utilization rate.

Currently, 3FS only supports the replication storage mode. The code includes an option for EC, which has not been implemented yet.

Test Environment Configuration

Both vePFS and 3FS servers use the following server configuration during deployment (hereafter referred to as Server A):

Key server components	Component Description
CPU	64core
Memory	256GiB
Network card	200Gbps *2
NVMe SSD	7.68TiB *16
RDMA	RoCE

The specific deployment scheme is as follows:

System	Roles and Server Description
vePFS	Shared metadata and data: Server A*3
3FS	Metadata: Server A1 Data: Server A3

To ensure a fair comparison, vePFS uses the same three-replica configuration as 3FS during testing. The default vePFS provided by Volcano Engine uses an EC configuration. If a three-replica setup is required, please submit a ticket to contact the Volcano Engine support team.

We designed and tested 29 use cases for both metadata and data layers. Due to space constraints, this article only presents 9 key test cases and result analyses.

Metadata layer testing and analysis

Technical point analysis and comparison

Before conducting the tests, we first analyze key technical aspects of the metadata layer for vePFS and 3FS based on the architecture.

Request pathway

On the file system layer, vePFS distributes metadata based on directories and subtrees, addressing consistency and reliability issues with a more complex system design.

3FS moves distributed capabilities down to the underlying KV storage and leverages FoundationDB's serializable strong ACID transaction semantics to simplify overall metadata design and directory semantics implementation. However, this architecture is difficult to achieve optimal performance, with issues such as longer request pathways and performance degradation in scenarios involving transaction conflicts.

POSIX semantics

vePFS provides comprehensive support for POSIX semantics and is compatible with private clients as well as NFS and SMB protocols.

3FS does not fully support POSIX semantics but supports FUSE and USRBIO.

Value-added features

vePFS offers robust value-added features, including file locks, Fileset-level quotas, and QoS, enabling broader application scenarios For example, directory quotas are essential for many enterprise users.

3FS does not support value-added features.

Scenario-based analysis and testing

We used typical AI training metadata behaviors as input to comparative testing of vePFS and 3FS, covering the following scenario-based tests:

Scenario 1: During the dataset preparation phase in multimodal training, a large number of small files are written concurrently to the file system. Datasets typically contain millions to billions of sample data items, with file sizes ranging from KiB to MiB. The storage system must deliver high throughput for creating and writing small files. To address the metadata requirements in this scenario, we designed Test Case 1: Small File Creation and Deletion Test.

Scenario 2: During the training phase in a multimodal scenario, the system loads the cleaned dataset and concurrently reads massive amounts of small files from the storage system. The computing cluster can reach a scale of over a thousand GPUs. The data volume is an order of magnitude smaller than before cleaning, but the storage system must offer high throughput for reading small files. The scenario requires a certain level of metadata read capability from the file system. We designed Test Case 2: Concurrent Small File Read Test.

Scenario 3: Extended to general file system scenarios, such as typical HPC settings—covering industries like genomics and meteorology, with more diverse and complex business needs. This requires high performance, semantic integrity, consistent caching, value-added features, rich access protocols, and a comprehensive system capability. Limited by time and testing resources, we selected POSIX semantic integrity for verification in general scenarios and designed Test Case 3: POSIX Compatibility Test.

Test Case 1: Small File Creation and Deletion Test

Case: Multiple clients use mdtest to concurrently create and delete small files in separate directories, generating 1 million files per directory.

Test Results: The following data shows the performance of a single metadata node.

Analysis of Results:

The large gap in metadata write performance between 3FS and vePFS is mainly due to the underlying construction based on a generic distributed KV system, which results in a longer data path and higher transaction overhead.

Test Case 2: Concurrent Small File Read Test

Test case: Multiple clients run mdtest concurrently in separate directories, with 1 million files in each directory.

Result analysis:

3FS has a much larger gap compared to vePFS, mainly due to a longer overall architecture path, multiple request hops, and transaction overhead.

Test Case 3: POSIX Compatibility Test

Test case: Use the general POSIX interface compatibility test suite pjdfstest to run all test cases, and compare the case pass rate across products.

Result analysis:

vePFS has few failed cases, only failing in some specific scenarios. It offers relatively complete POSIX semantic support.

3FS is less comprehensive. It does not support file types such as pipe files, block device files, character device files, or socket files; all test cases for these types failed. These semantics are not essential requirements for AI training and inference scenarios. With its current pass rate, 3FS can adequately support these scenarios.

Conclusion

Based on the above technical architecture comparison and analysis, combined with test validation, the following observations are made:

At the architecture level, 3FS effectively implements the system layering concept. By leveraging distributed KV storage, it simplifies its own architecture design and the implementation of directory semantics. This architecture also amplifies metadata path requests, making it difficult to achieve optimal performance.

In terms of semantic features, 3FS abandons some POSIX semantic support and enhancements for value-added features of general file systems.

Overall, the metadata design of 3FS is not friendly to small files. Upper layers can use packaging formats like FFRecord to merge small files, shifting metadata performance issues to data-side IO performance.

Data-side testing and analysis

Technical point analysis and comparison

Before conducting comparison tests, we first analyze the key technical points of vePFS and 3FS on the data side.

Client

vePFS uses a proprietary protocol client, implementing optimizations such as asynchronous I/O, RPC aggregation, caching and prefetching, shared memory between user space and kernel space within the client.

3FS provides two access methods: FUSE Client and USRBIO. USRBIO operates fully in user space, employs asynchronous processing, zero-copy, and shared memory, delivering superior endpoint access performance compared to FUSE Client.

Read/write routing

Common read/write routing modes include Main Routing Mode and Star Routing Mode:

Main Routing Mode: The client first sends data to the storage primary node, which then forwards it to other nodes.

Star Routing Mode: The client directly distributes data to storage node shards.

vePFS uses Star Routing Mode for multi-replica storage and Main Routing Mode for EC storage.

3FS uses chained replication in Main Routing Mode with the multi-replica approach, starting from the chain head and propagating to the chain tail.

Both vePFS and 3FS support arbitrary replica reads.

Redundancy and network amplification

Replica mode: Assume the configuration uses three replicas

Write scenario: 3FS uses the primary routing mode, where compute nodes send a single copy of the write data, resulting in 1x amplification. In contrast, vePFS uses the star routing mode, causing compute node network amplification to reach 3x. Thus, 3FS offers superior compute node amplification compared to vePFS. However, in 3FS, chained replica copying among storage nodes creates an additional 2x amplification on storage network output.

Read scenario: Both 3FS and vePFS read from backend storage nodes via the client, resulting in identical network amplification for compute and storage nodes.

	Compute node	Storage Node
vePFS (Star Routing Mode)	Network magnification: 3x	Network Output Amplification: None Network Input Amplification: 3x
3FS (Primary Routing Mode)	Network Out Amplification: 1x	Network Out Amplification: 2x Network In Amplification: 3x

EC mode: 3FS does not support EC mode. Assuming vePFS uses an EC configuration of 8+2 and main routing for write scenarios, the primary node network output amplification is 1.25x. Compared to replication mode, EC mode significantly improves overall network output amplification. However, EC also introduces some drawbacks, such as CPU overhead from EC encoding and decoding, and extra data copy costs associated with EC data slicing.

Space management

vePFS uses fixed-size chunk management, such as a 2MiB configuration. Different files can reuse chunks, but file deletion requires fragmentation clean-up or GC operations, making software implementation more complex.

The 3FS chunk size ranges from 64KiB to 64MiB, increases in powers of 2 for a total of 11 options. The Allocator adapts to the maximum chunksize set by the user and selects the physical block closest to the actual space after splitting according to the maximum chunksize. Different files cannot reuse chunks. Deleted files reclaim their corresponding chunks, simplifying the software implementation.

Scenario-based analysis and testing

We used the IO patterns from large model training and inference scenarios as input to compare three storage systems: vePFS and 3FS, all configured with three-way redundancy. The comparison tests covered the following scenario-based tests:

Scenario 1: In the process of collecting model datasets, a large number of datasets are gathered for training. These training datasets come from different regions and fields. After collection is complete, data is written to storage using multiple files and concurrent writes. Therefore, we constructed a data collection scenario using multiple clients and files in parallel, writing files with 1MiB IO size to test backend storage write performance. See Test Case 4: Storage Node Write Throughput Test for details.

Scenario 2: During large model training, thousands of cards are needed for parallel distributed training, with many machines and a higher fault rate. To ensure training tasks can recover after failures, CheckPoints must be saved regularly during training. CheckPoints are written to backend storage using multiple files, high concurrency, and high throughput. Therefore, we used a single client writing multiple files concurrently with 4MiB IO size to test single client write performance. See Test Case 5: Single Client Write Throughput Test for details.

Scenario 3: In model distribution scenarios, multiple clients concurrently pull models from backend storage. The backend storage bandwidth directly determines the length of time to pull models for multiple clients. Therefore, we used multiple clients in parallel with 1MiB IO size to read files and tested backend storage read performance. See Test Case 6: Storage Node Read Throughput Test for details.

Scenario 4: In inference scenarios, the read performance of a single client is especially important and directly affects the length of time for a single client to pull a model. Therefore, we used a single client reading multiple files concurrently with 4MiB IO size to test single client read performance. See Test Case 7: Single Client Read Throughput Test for details.

Scenario 5: Other scenarios. Besides backend storage read and write for training and inference, storage space amplification and disk read-write amplification are also important metrics for evaluating storage systems. Storage space amplification directly impacts storage space utilization, while disk read-write amplification affects disk utilization and cluster bandwidth performance. Therefore, in Test Case 8: Space Amplification Test, we covered space amplification testing. In Test Case 9: Disk Read-Write Amplification from Overwrite Write Test, we covered the impact of overwrite writes on disk read-write amplification.

Test Case 4: Storage Node Write Throughput Test

Test case: Create 9 directories, each containing 90,000 files with a size of 2 MiB per file. Each IO appends 1 MiB, and multiple clients send write requests simultaneously. Storage is configured with three replicas.

Result analysis: vePFS uses a star mode for writing, which delivers superior performance and high single-node throughput. 3FS uses a main routing mode to write serially in a chain across replicas. On the storage side, this design adds twice the serial writing latency. Under equal concurrency, actual testing showed low pressure on storage nodes and idle CPUs, so system write throughput could not be fully loaded, meaning backend storage performance was not fully utilized. This value does not represent the upper limit of write throughput for 3FS storage nodes.

Test case 5: Single client write throughput test

Test case: With a single client, create 10,000 files, each sized 500 MiB. Multiple files are written concurrently with a 4 MiB IO size per write. Storage is configured with three replicas.

Result analysis: According to the data, vePFS and 3FS offer comparable single-client write capabilities.

Test case 6: Storage node read throughput test

Test case: Create one directory containing 200 files, each 100 GiB in size. Embed data using 1 MiB IOs, then perform sequential reads of 1 MiB. Multiple clients send identical concurrent requests, and storage is configured with three replicas.

Result analysis: At first, the average read capacity of a single 3FS node was 33.67 GiB/s. By replacing the CPU and memory with better storage models and using optimizations such as core binding and parameter tuning, the average read capacity per node was finally optimized to 38 GiB/s, which is close to the officially claimed performance. Overall, 3FS still provides slightly lower read bandwidth than vePFS.

Test case 7: Single client read throughput test

Case: Single client builds 2,000 files, each file is 500 MiB, reads multiple files concurrently with each I/O size of 4 MiB, and uses a three-replica storage configuration.

Result analysis: vePFS is limited by CRC overhead and cross-NUMA memory access, causing CPU bottlenecks; 3FS USRBIO also faces CPU bottlenecks, and the performance of both is basically equal.

Test case 8: Space usage amplification test

3FS configures 1 MiB chunksize; the Chunk Engine allocates within 1 MiB based on the size of written data. If the data written exceeds 1 MiB, a new chunk is allocated, repeating until the entire file is stored. This section tests space usage amplification under data alignment and misalignment scenarios by using different data patterns.

Case:

Pattern1 File Size 128KiB: Configure 100,000 files in a single directory, each file is 128KiB. Pre-embedded IO writes 128KiB, and the storage configuration uses three replicas.

Pattern2 File Size 128KiB+4KiB: Configure 100,000 files in a single directory, each file is 128KiB+4KiB. Pre-embedded IO writes 128KiB+4KiB, and the storage configuration uses three replicas.

Pattern3 File Size 1MiB+8KiB: Configure 100,000 files in a single directory, each file is 1MiB+8KiB. Pre-embedded IO writes 1MiB+8KiB, and the storage configuration uses three replicas.

Test results:

Result analysis:

In the Pattern1 scenario, the amplification of both products is similar.

In the Pattern2 scenario, small files (128KiB+4KiB) are not chunk size aligned. Besides triple replica amplification, 3FS expands chunk size according to the power of 2 starting at 64KiB, doubling chunk space. Maximum space waste can approach 6 times.

In the Pattern3 scenario, small files (1MiB+8KiB), 3FS uses a 1MiB chunk and a 64KiB chunk, pushing space amplification to 3.1. The test uses vePFS with a chunk size of 2MiB; in this pattern, files take over half a chunk, and the leftover chunk space becomes a fragment, causing amplification. If smaller files are written later, the fragment space can still be reused.

Test Case 9: Disk read/write amplification caused by overwrite writes

This scenario only includes vePFS and 3FS tests.

Use case:

Pattern1 128KiB rwrite: Create 100,000 files in a single directory, each file is 10MiB. Data is pre-written with 128KiB IO, then overwritten randomly with 128KiB after pre-writing.

Pattern2 128KiB+4KiB rwrite: Create 100,000 files in a single directory, each file is 10MiB. Data is pre-written with 128KiB IO, then overwritten randomly with 128KiB+4KiB after pre-writing.

Pattern3 4KiB rwrite: Create 100,000 files in a single directory, each file is 10MiB. Data is pre-written with 1MiB IO, then overwritten randomly with 4KiB after pre-writing.

Test results:

Result analysis:

After 3FS splits requests by 1MiB, the maximum chunksize for Chunk Engine is 1MiB. Because 3FS requires the entire 1MiB chunk to be read for modification, updates the required data, and then writes the new chunk in 1MiB. The system is configured with three replicas. Each shard includes a full chunk read and a full chunk write.

Theoretical amplification for Pattern1: (1MiB read + 1MiB write) ÷ 128KiB × 3 = 48; measured value is 50.

Theoretical amplification for Pattern2: (1MiB read + 1MiB write) ÷ (128KiB + 4KiB) × 3 = 46.5; measured value is 48.

The theoretical amplification value for Pattern3: (1MiB read + 1MiB write) ÷ (4KiB) * 3 = 1536; actual measurement is 1500.

vePFS does not have read amplification; the amplification comes only from triple-replica writes, which is around 3.

Conclusion

Based on the above technical architecture comparison and analysis, along with test verification, we observe:

3FS uses chained replication with multiple replicas, performing multi-hop serial writes to backend storage. This sacrifices space utilization and write latency for better read performance. For reading, 3FS enables one-hop arbitrary replica reads, which is especially suitable for large-block read scenarios and provides exceptional read performance.

Compared to file systems that support EC, 3FS uses triple replicas, resulting in higher space consumption. When writing small files, data sharing between chunks is not possible, which further increases chunk space waste.

3FS is not friendly to small file overwrite writes; you need to perform read-modify-write based on chunk granularity, causing significant disk read/write amplification.

Overall, 3FS performs well within closed ecosystems like DeepSeek model distribution. It has balanced scenario-based applications with architectural trade-offs, optimizing for reading scenarios and simplifying software complexity, resulting in good end-to-end performance.

3FS Design Philosophy

The philosophy behind 3FS throughout the entire architecture design can be summed up in two keywords: collaborative design and trade-offs.

Collaborative design

In computer architecture, layered system architecture is considered the golden rule for managing complexity For a complex system, using well-structured hierarchical abstraction to break down complex issues into smaller, more focused problems enables targeted solutions. This dramatically reduces the mental load for developers, significantly boosts development efficiency, and makes the entire system easier to build.

In large-scale AI training and inference engineering, layered system architecture establishes clear boundaries of responsibility for roles such as algorithm engineers, training platform engineers, and storage engineers. This allows everyone to focus on their areas of expertise and effectively shortens the cycle from design to implementation and scaling However, there are natural cognitive barriers between different fields. Even if these barriers are as thin as a veil, failing to break through them can lead to efficiency traps and blind spots in collaboration Different roles on both sides of the barrier might not understand each other's best practices, resulting in the system running less efficiently These efficiency traps and collaboration blind spots are likely invisible and go unnoticed.

The DeepSeek team seems to be aware of this issue, proactively breaking through cognitive barriers and using USRBIO as a bridge to bring 3FS business parties and 3FS to a shared understanding:

The business side recognizes that large files and high IO are more compatible with 3FS, so they are willing to adapt and aggregate small files within the business and framework to fully leverage the throughput advantage of 3FS.

3FS optimizes end-to-end performance for the scenarios above and addresses the high learning curve of USRBIO by providing foundational components like FFRecord packaging format and DataLoader to simplify usage complexity.

Through this collaborative design across layers, 3FS effectively meets the performance requirements of DeepSeek’s operations.

Trade-offs

The DeepSeek team consists of about one hundred members and is responsible for all preliminary research, development, and maintenance of algorithms and infrastructure. With such staffing, expecting an in-house file system to reach industry-leading levels in most technical metrics is clearly unrealistic and unnecessary. The 3FS developers are fully aware of this, and their system design reflects the art of making trade-offs by focusing their limited resources on a few key technical aspects rather than pursuing perfection everywhere.

3FS makes trade-offs in system design on the following points:

Focuses on optimizing high-throughput read/write workloads; performance for other workload types is not deeply tuned.

USRBIO, chain replication, and chunk-level writing are all designs that favor large files and large I/O. However, these approaches have inherent drawbacks when handling small files and small I/O scenarios. For example, the chain replication protocol results in higher latency, and chunk-level writing leads to greater space amplification.

Metadata is stored using FoundationDB and has not been deeply optimized for transaction execution efficiency, resulting in low metadata performance limits. This is not critical for large files, but it is unsuitable for scenarios involving massive numbers of small files. 3FS does not intend to address this issue, but instead lets upper-layer applications avoid it.

Only supports DeepSeek infrastructure; compatibility with other environments is not a design goal:

All DeepSeek internal environments use RDMA network interconnection, so 3FS has abandoned support for non-RDMA environments.

SSD disks have two sector configurations, 512 bytes and 4096 bytes. 3FS only supports 512-byte sectors in its code, which makes storage node crashes likely in environments with 4096-byte sectors. This issue caused some trouble in our testing of 3FS. Of course, this problem is not difficult to fix. However, it reveals that further code adaptation and testing are needed if 3FS is to be extended to more production environments outside DeepSeek.

The system's stability design is quite basic, placing high demands on environment stability and maintenance capabilities:

The system uses a Master-centric design pattern. If the Master fails, it affects the entire system's service. Systems with strict SLA commitments should avoid this design and adopt a separation between the service plane and the control plane to prevent SLA violations caused by prolonged recovery times after Master failures.

Metadata and data nodes only rely on leases with the Master to determine whether nodes can provide services. There are no refined safeguards against issues like network partitions, unidirectional networks, or switch black holes, so network fluctuations can easily impact service quality Of course, similar network issues are very rare with DeepSeek’s high-quality InfiniBand network, so 3FS has not focused heavily on them.

These trade-offs make the 3FS system seem imperfect, with short shortcomings and long strengths But from DeepSeek’s perspective, everything is just right.

Outlook

As we continue to explore the design philosophy of 3FS, we see that every approach consistently answers a fundamental question: How should a product be designed to best serve its target customers For 3FS, its initial target customer was only DeepSeek, so meeting DeepSeek’s needs is all the product has to do.

However, for us, the answer to this question is different Unlike DeepSeek’s highly focused business scenarios, cloud file storage serves users from all industries with diverse and complex business needs, including AI training and inference similar to DeepSeek, as well as autonomous driving, rendering, finance, and office scenarios These scenarios demonstrate drastically different workload characteristics, including large files, small files, random IO, sequential IO, and sensitivity to metadata. Admittedly, we could develop a customized system like 3FS for each business scenario, but this approach would require us to create many different systems to cover all these cases, resulting in an overwhelming maintenance workload and making it extremely difficult for users to choose. Therefore, we believe that encapsulating complexity within the system while simplifying the product for customers aligns even better with the design philosophy of 3FS.

Although vePFS currently matches 3FS’s performance in training scenarios, looking ahead, user data volumes are growing rapidly and business complexity continues to increase. The file system field still faces enormous challenges, and the industry has a long way to go. From these challenges, we have distilled the following key issues:

Issue 1: Bidirectional data flow between file systems and object storage

Object storage offers exceptional scalability, extremely low costs, high reliability, and a robust ecosystem, making it the first choice for cloud data storage. Many business processes revolve around object storage. However, object storage cannot cover all business scenarios, and in many cases, file systems remain the preferred option for users:

Many software and frameworks have not integrated native object storage interfaces and rely heavily on file system semantics.

Object storage’s performance in metadata handling and IO operations for massive numbers of small files lags behind file storage in certain respects.

Some users are more familiar with using the file system and prefer it over other options.

If file storage can enable two-way data movement with object storage, achieving one set of data with multiple presentations, flowing as needed and reducing the operational cost for users moving data between systems, this would significantly improve the efficiency and experience of data handling While establishing the basic capability is not difficult, creating intelligent and user-friendly product logic with extremely efficient and stable data flows poses significant challenges for both product and architecture design.

Issue 2: Metadata scalability for a trillion-file scale

Traditionally, it is believed that file system scale generally reaches tens of billions, while object storage can reach the trillion level The emergence of data movement capability breaks this assumption Once a file storage directory establishes a data flow relationship with an object bucket, it cannot and should not limit the number of objects in the object bucket As a result, the maximum scale for the file system directly matches the trillion-level scale of object storage.

Scaling from tens of billions to trillions—expanding tenfold—is far more challenging in engineering than simply adding ten times more nodes to the metadata system

Increasing the number of nodes greatly raises the likelihood of hardware and network failures, demanding much more stringent requirements for system stability design

The performance requirements for metadata must match the scale of file count; a tenfold increase in the number of files means a tenfold increase in performance requirements Achieving this goal requires at least two major technical breakthroughs:

A breakthrough in cross-node operation performance The correctness of cross-node operations, whether ensured through transactions or distributed locks, faces performance barriers due to fundamental design principles. For example, transaction-based solutions relying on a single clock node to generate transaction timestamps will cap system performance at one million TPS because of this single point Breaking through this performance barrier requires fundamental innovation in system architecture

A breakthrough in single-node engine performance Flat object storage buckets have no directory hierarchy, and many users tend to use them in ways that align with this design, resulting in mega directories containing tens of millions or even hundreds of millions of files. The main performance metrics of mega directories must be consistent with those of ordinary directories. Data movement capability in metadata systems generates considerable background traffic, which consumes a portion of metadata performance These new developments require metadata nodes to further push the limits of single-node engine performance. The stronger the single-node engine, the fewer compromises are needed in architecture design and product capability

Question 3: Decoupling data plane performance and capacity elasticity when measured by throughput

Capacity and performance are coupled in the traditional sales logic of cloud file storage. The performance a file system can provide is entirely determined by its capacity

This sales model is usually not the most cost-effective for users, as the required ratio of performance to capacity varies at different stages. Especially in the early stages of a business, file system capacity is relatively small and performance is the main concern. Users want performance that exceeds the specification (performance per unit capacity) to accelerate business growth.

This trend has intensified even further recently. DeepSeek's explosive popularity is driving the adoption of AI large models, and the demand for inference is growing at an astonishing rate. When deploying inference, a file system may store only hundreds of GiB to several TiB of model files, but hundreds or thousands of inference servers simultaneously loading them require throughput at the TiB/s level to meet demand.

Therefore, decoupling sales of performance and capacity in cloud file storage products is an inevitable trend. Behind this product capability, the file system's data plane requires solid engineering to ensure stable service when facing sudden surges in traffic.

Issue 4: Extreme performance issues at the client side, measured by business outcomes

As the direct entry point for business access, the performance of the client directly affects the user experience of the service. In the past, file systems often used kernel-mode clients, SDKs, or system call interception for client implementations. These approaches typically lead to high complexity, high operational cost, and poor compatibility. As various capabilities have gradually improved, FUSE has become the mainstream client solution for emerging file systems in recent years

Based on FUSE, several optimization paths on the client side will be combined to jointly support the core goal of performance optimization:

Optimization in FUSE engineering will continue, constantly achieving higher performance limits and narrowing the gap with other solutions This solution offers the best compatibility, though it is quite challenging. Many file system vendors are working on it and will remain committed

3FS USRBIO provides an innovative technical model: in specific scenarios, it integrates the advantages of FUSE and SDK, using FUSE to ensure broad compatibility and SDK to accelerate the most critical IO paths This approach requires business-side adaptation, but after DeepSeek's pilot, it is projected to gain wide acceptance and may eventually lead to an industry-wide standard The standard may not necessarily be USRBIO, but will use a similar concept

Near-compute caching further boosts client performance When the client and compute node are on the same high-speed network, leveraging idle memory or local disk resources on the compute node to provide local or distributed caching enables the fastest path for loading hot data

Problem 5: Near real-time data analysis and insights with massive file scale

With the rapid growth of AI and big data services, users are accumulating massive amounts of data in file systems The attributes and access behaviors of this data hold significant value. Through comprehensive analysis and understanding, insights gained can help users better understand business workloads and further optimize costs, performance, and security Typical use cases include:

Monitoring file access frequency to identify infrequently accessed files and move them to object storage, reducing overall data storage costs

Providing information such as I/O access patterns, file size distribution, and hot directories to help analyze access bottlenecks. This guides optimizations like packaging small files or splitting large hot directories to improve access performance

Supporting multidimensional statistical queries on the file system to help administrators improve data management efficiency and plan file system resource usage effectively

Native support for intelligent data analysis and insights is critical in file systems, as only native support ensures real-time information at massive scale On this basis, integrating large model capabilities is also important, offering a more efficient and natural user interface compared to SQL and other methods

We have been committed to addressing these challenges, and our solutions are gradually being implemented in our products. vePFS, together with other cloud storage offerings, delivers high-quality Volcano Engine AI Cloud Native services for you.

Last updated: 2026.03.03 10:39:10