What is Lance #

Lance is a modern columnar data format designed specifically for large-scale AI datasets. It is similar to Parquet, but deeply optimized for the storage and access of multimodal data, such as images, videos, audio, point clouds, and traditional tabular data. Lance supports POSIX-compatible Linux file systems as well as mainstream cloud object storage (Volcano Engine TOS), making it an ideal foundation for building a unified AI data lake.

The core of the Lance format lies in its efficient file structure, which consists of three logical layers:

1. File format layer: A self-developed underlying storage format that can efficiently store and randomly access large-scale, multimodal data.
2. Data lake format layer: Similar to Apache® Iceberg, it provides powerful data lake management features, such as atomic transactions and multi-version control. It is mainly intended for offline engine access, does not support writes from streaming engines, and is currently not applicable to AI scenarios.

   Apache Gravitino, the names of other Apache projects, and the ASF logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

3. Application layer: Lance supports multiple integration methods. You can operate with a standalone SDK, such as PyLance, or read and write Lance data through distributed engines, such as Spark and Ray. You can even use the lancedb interface to use Lance as a vector database.

What can Lance do #

Main features #

The main features of Lance include:

• High-performance random access: For individual records, the performance of the take operation is more than 10 times higher than Parquet.
• Vector search: Indexes support millisecond-level nearest neighbor search and combine scalar queries with vector search.
• Zero-copy and automatic version control: Automatically manages data versions and uses built-in zero-copy logic to reduce redundancy.
• Ecosystem integration: Apache® Arrow, DuckDB, and more are under development.

  Apache Arrow, the names of other Apache projects, and the ASF logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Core problems solved (challenges of AI datasets) #

In traditional AI/ML workflows, dataset management faces many challenges, such as:

• Version confusion: Changes to training data are difficult to track, resulting in non-reproducible model training results.
• Inefficient access: For large-scale datasets, especially those containing vectors or multimedia files, random access and query performance are poor.
• Fragmented ecosystem: Data preprocessing, model training, and vector search often require different tools and data formats, increasing system complexity and data redundancy.

Lance addresses these issues by providing a unified platform, with core capabilities including:

• Unified management of multimodal data: Stores unstructured data (such as images) and structured data (such as metadata and labels) in the same table.
• Efficient data version control: Automatically records every data change to ensure experiment reproducibility.
• High-performance query and search: Supports data filtering and vector similarity search.

Core features #

• Zero-copy, index-free data access: Uses zero-copy technology when reading data to avoid unnecessary data duplication, significantly improving I/O efficiency. Its unique format design enables random access performance that is 100 times faster than Parquet, without relying on additional indexes.
• Dataset version control and reproducibility: Every data write, update, or delete operation automatically generates a new, immutable dataset version.
• Seamless integration with the ecosystem: Lance is fully compatible with the Apache Arrow memory format, enabling easy data exchange with mainstream data science tools such as Pandas, DuckDB, and Polars, and also supports integration with large-scale distributed computing engines like Spark and Ray.

Architecture overview #

Internal structure of the Lance format #

A Lance dataset is stored as a directory, which mainly contains:

• data/ directory: Stores actual data files (.lance files).
• _versions/ directory: Stores the metadata manifest for each version (manifest files), recording which data files are included in that version.
• _transactions/ directory: Used to ensure atomicity of write operations.
• _deletions/ directory: Stores deletion files (deletion vectors), used to track deleted rows without rewriting data files.
• The _indices/ directory: After creating an index for Lance data, the index files are stored in this directory, with each index stored in a separate UUID subdirectory.

Data read/write flow #

Usage #

For instructions, see Quick start.

#