Introduction to Daft #

DAFT is the world's first distributed computing engine with native support for multimodal data processing, breaking through the traditional ETL paradigm and providing:
Unified multimodal data abstraction + native Python/Rust API + distributed execution engine integrated solution, enabling seamless connection from raw data to AI features.
Daft is positioned to provide end-to-end processing capabilities for enterprise-grade multimodal data lakes. It eliminates traditional tools' reliance on single data types, seamlessly integrating with dozens of data formats such as text, images, video, audio, JSON, Parquet, and HuggingFace Dataset, enabling automation of the entire process from data ingestion, cleaning, and transformation to analysis and modeling. Whether it is product images and user reviews on e-commerce platforms or video and point cloud data in intelligent driving, Daft can aggregate scattered multimodal information into structured "data assets," providing a consistent data processing foundation for subsequent AI training and business analytics.

Daft technical advantages #

Daft is designed with a focus on multimodal data processing, supports native multimodal data types, enables users to process petabyte-scale multimodal data, seamlessly integrates with existing machine learning systems, and delivers top-tier data processing performance.

1. Native multimodal processing: Within a single unified framework, by supporting native multimodal data types such as image, embedding, and tensor, Daft provides built-in operations like image decoding and vector similarity calculation, avoiding the lengthy workflows of traditional solutions.
2. Heterogeneous resource scheduling: Daft can automatically allocate optimal heterogeneous resources based on the computational characteristics of tasks, achieving seamless coordination of CPU and GPU computing tasks within the same workflow, thereby maximizing resource utilization and significantly improving overall computational performance.
3. Rust-powered performance: Leveraging a Rust core, Daft provides vectorized execution and asynchronous I/O capabilities, processes the same queries with less memory, and supports petabyte-scale data processing.
4. Seamless ML ecosystem integration: Seamlessly integrates with existing machine learning (ML) workloads, providing compatible interfaces for workloads such as PyTorch, NumPy, Pandas, and HuggingFace, making it easy to connect with existing systems.
5. Deep data lake collaboration: Supports multiple cloud storage media (such as S3, Volcano Engine TOS, and more), seamlessly connects with modern data lake formats (such as Apache® Iceberg, Hudi, Delta Lake, and more), offers deeply optimized integration with the Lance format for AI data lakes, and supports enterprise-grade catalogs (such as Unity, AWS Glue, and more).

   Apache Iceberg, the names of other Apache projects, and the ASF logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

6. Python UDF data processing capability: Leverages zero-copy capability powered by Apache Arrow, enabling direct multimodal data processing within UDFs, thereby eliminating data movement overhead and improving processing speed.

@daft.udf(return_dtype=daft.DataType.python())
def crop_images(images, crops, padding=0):
    cropped = []
    for img, crop in zip(images, crops):
        x1, x2, y1, y2 = crop
        cropped_img = img[x1:x2 + padding, y1:y2 + padding]
        cropped.append(cropped_img)
    return cropped

df = df.with_column(
    "cropped",
    crop_images(df["image"], df["crop"], padding=1),
)
df.show(2)

7. Out-of-the-box reliability: Intelligent memory management prevents OOM errors, and reasonable default values avoid complex configurations, allowing users to focus on their business tasks and providing an out-of-the-box experience.

Daft use cases #

Text generation #

Traditional engines (such as Pandas) are inefficient when processing large-scale text data combined with structured data, making data preparation for applications such as RAG more complex. Spark cannot perform heterogeneous scheduling of CPU/GPU resources, resulting in uneven resource utilization efficiency for text processing tasks. Daft natively supports multimodal data processing, and by leveraging Daft's built-in embedding types and embedding functions, efficiently schedules CPU operators and GPU operators to the appropriate resources, flexibly configures the resources required for different types of operators, and achieves efficient GPU utilization. Its built-in Rust kernel delivers efficient data processing performance, enabling business users to focus on business process development without expending significant effort on complex data processing pipelines.

Image processing #

In scenarios such as image classification, style transfer, or AIGC (such as Stable Diffusion), Daft can manage distributed preprocessing and model inference pipelines. Daft's heterogeneous scheduling capability can schedule different types of CPU/GPU operators, such as resolution adjustment, feature extraction, and label generation, to the appropriate resources. With streaming computation capability, it improves GPU utilization and significantly increases image processing and generation throughput. Built-in image types and image processing functions enable efficient image processing.

Video processing #

Conventional video processing requires splitting videos into frames for individual processing, resulting in a cumbersome workflow and making it difficult to maintain temporal consistency. Daft, based on a native multimodal AI data lake architecture, directly supports processing multimodal data such as audio and video without additional conversion steps. As a distributed engine with Rust as its core, Daft efficiently handles the parallel computing requirements of video data, maintaining data integrity while ensuring processing speed, making it especially suitable for preprocessing large-scale video datasets for AI training.

Autonomous driving #

Autonomous driving systems face challenges in integrating multi-source heterogeneous data (such as point clouds, images, and sensor data), and traditional engines like Spark struggle to provide unified processing. Daft supports seamless integration of multimodal data including images, text, audio, video, and point clouds, and is specifically designed to address such complex data processing scenarios. This engine originates from Lyft's practical needs in data processing, enabling efficient handling of real-time data streams in autonomous driving, achieving elastic scaling and engineering implementation of multimodal data, while balancing cost control and high-performance output. Daft supports seamless integration with PyTorch, allowing processed data to be directly incorporated into autonomous driving model training, improving the efficiency of model training.

Embodied intelligence #

Embodied intelligence requires processing multimodal data streams such as video, tactile, and motion data, and traditional data processing tools struggle to meet the complex requirements of multimodal data. As an AI-native multimodal data processing platform, Daft can interface with various embodied scenario data sources such as HDF5, LeRobot, and mcap, providing a unified data processing middleware for embodied intelligence. Its distributed architecture supports complex data preprocessing and feature engineering, enabling robotic systems to more efficiently integrate various sensory data including vision, touch, and position, thereby improving the accuracy and response speed of environmental interactions.

Daftquick start #

The following demonstrates how to use Daft to query image files in a TOS path, filter out images that do not meet the requirements, and finally use a UDF to generate a sample with red marks for images with red occlusion.

import daft
import numpy as np
import PIL.Image
from PIL import ImageFilter 

IO_CONFIG = daft.io.IOConfig(s3=daft.io.S3Config(anonymous=True))  # Use anonymous S3 access
# Load image data
df = daft.from_glob_path(
    "s3://daft-public-data/open-images/validation-images/*",
    io_config=IO_CONFIG,
)

# Filter out oversized images
df = df.where(df["size"] < 300000)

df = df.select("path")
df = df.with_column("image", df["path"].url.download(io_config=IO_CONFIG))
df = df.with_column("image", df["image"].image.decode())

def magic_red_detector(img: np.ndarray) -> PIL.Image.Image:
    """Gets a new image which is a mask covering all 'red' areas in the image."""
    img = PIL.Image.fromarray(img)
    lower = np.array([245, 100, 100])
    upper = np.array([10, 255, 255])
    lower_hue, upper_hue = lower[0, np.newaxis, np.newaxis], upper[0, np.newaxis, np.newaxis]
    lower_saturation_intensity, upper_saturation_intensity = (
        lower[1:, np.newaxis, np.newaxis],
        upper[1:, np.newaxis, np.newaxis],
    )
    hsv = img.convert("HSV")
    hsv = np.asarray(hsv).T
    mask = np.all(
        (hsv[1:, ...] >= lower_saturation_intensity) & (hsv[1:, ...] <= upper_saturation_intensity), axis=0
    ) & ((hsv[0, ...] >= lower_hue) | (hsv[0, ...] <= upper_hue))
    img = PIL.Image.fromarray(mask.T)
    img = img.filter(ImageFilter.ModeFilter(size=5))
    return img

df = df.with_column(
    "red_mask",
    df["image"].apply(magic_red_detector, return_dtype=daft.DataType.python()),
)
df.show()