Operator ID: daft.las.functions.video.video_first_frame.VideoFirstFrame

Description #

The video first frame identification processor extracts the first frame from a video as the cover image, supports output in multiple image formats, and allows skipping black screen frames.

Key features #

• Efficient extraction:
  • Reads only the header information of the video to quickly extract the first frame and improve processing efficiency
  • Supports GPU acceleration to further enhance extraction speed
• Multiple format support:
  • Output: JPG, PNG, BMP, and other common image formats
• Intelligent black screen detection:
  • Allows skipping black screen frames and extracting the first non-black screen frame to improve cover quality
• Multiple input and output methods:
  • Input: Supports video path input and binary data input
  • Output: Supports saving results to TOS or local directory
• Distributed processing support:
  • Distributed processing based on Daft and Ray, supports large-scale video processing

Format support #

• Input: Mainstream video formats such as MP4, AVI, MOV
• Output: Common image formats such as JPG, PNG, BMP

Operator parameters #

Input #

Output #

Column for the extracted first frame image path

Parameters #

If a parameter does not have a default value, it is required

Examples #

The following code demonstrates how to use Daft (for distributed scenarios) to run the operator for audio format conversion. Supports conversion to multiple formats including MP3, WAV, FLAC, AAC, and OGG.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.udf import las_udf
from daft.las.functions.video import VideoFirstFrame

if __name__ == "__main__":
    # The extracted initial frame image will be saved to the specified TOS path. Therefore, you need to set environment variables to ensure you have permission to write to TOS, including: ACCESS_KEY, SECRET_KEY, TOS_ENDPOINT, TOS_REGION, TOS_TEST_DIR
    TOS_TEST_DIR = os.getenv("TOS_TEST_DIR", "your-bucket")
    output_tos_dir = f"tos://{TOS_TEST_DIR}/video_firstframe"

    if os.getenv("DAFT_RUNNER", "native") == "ray":
        import logging

        import ray

        def configure_logging():
            logging.basicConfig(
                level=logging.INFO,
                format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
                datefmt="%Y-%m-%d %H:%M:%S",
            )
            logging.getLogger("tracing.span").setLevel(logging.WARNING)
            logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
            logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
            logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

        ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
        daft.set_runner_ray()
    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    # Use environment variables to construct the URL
    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "video_path": [
            f"https://{tos_dir_url}/public/shared_video_dataset/test_video_with_black_start.mp4"
        ]
    }

    ds = daft.from_pydict(samples)

    constructor_kwargs = {
        "output_tos_dir": output_tos_dir,
        "output_format": "png",
        "skip_black_frames": True,
        "black_threshold": 0.1,
        "rank": None,
    }

    ds = ds.with_column(
        "firstframe_path",
        las_udf(VideoFirstFrame, construct_args=constructor_kwargs, num_gpus=1, batch_size=1, concurrency=1)(
            col("video_path")
        ),
    )

    ds.show()
    # ╭────────────────────────────────┬──────────────────────────────────╮
    # │ video_path                     ┆ firstframe_path                 │
    # │ ---                            ┆ ---                              │
    # │ Utf8                           ┆ Utf8                             │
    # ╞════════════════════════════════╪══════════════════════════════════╡
    # │ tos://your-bucket/video_first… ┆ tos://your-bucket/video_first…   │
    # ╰────────────────────────────────┴──────────────────────────────────╯