Description #

The Qwen2.5-VL multimodal video understanding model supports temporal semantic parsing and natural language description generation, making it suitable for a wide range of video analysis scenarios.

Main features #

• Multimodal temporal modeling: Supports three video input formats (URL, Base64 encoding, binary stream), flexibly adapting to different data sources.
• Spatiotemporal joint modeling: Automatically captures the spatiotemporal features and semantic associations of videos, enabling more accurate video content understanding.
• Conversational prompt support: Customize the generation direction via the prompt parameter to meet diverse business requirements.
• Efficient inference: Integrates the VLLM inference engine, supporting bfloat16, float16, and float32 precision, fully utilizing GPU performance.
• It is recommended to use GPUs with 48GB or more GPU memory.

Applicable scenarios #

• Video content understanding and summarization
• Intelligent monitoring and event analysis
• Multimodal dialogue and interaction
• Other AI applications requiring video semantic parsing

Caution #

• Only GPU environments are supported. It is recommended to set parameters appropriately based on GPU memory and business requirements.
• For best results, standardize video size and frame rate before inference.

Operator parameters #

Input #

Output #

An array of processed results, with each element representing the visual understanding result for each video.

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use daft to run the operator for video content understanding and generate descriptions as instructed.

from __future__ import annotations

import logging
import os

import ray

import daft
from daft import col
from daft.las.functions.multimodal.qwen_vl_video_understanding_vllm import QwenVLVideoUnderstandingVLLM
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    os.environ["DAFT_RUNNER"] = "ray"

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()
    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "video_path": [
            f"https://{tos_dir_url}/public/shared_video_dataset/eating_56.mp4"
        ],
        "prompt": [""]  
    }

    video_src_type = "video_url"
    model_path = os.getenv("MODEL_PATH", "/opt/las/models")
    model_name = os.getenv("MODEL_NAME", "Qwen/Qwen2.5-VL-7B-Instruct")
    dtype = "bfloat16"
    default_prompt = None
    max_caption_length = 256
    min_pixels = 320 * 160
    max_pixels = 320 * 160
    fps = 1
    batch_size = 2
    seed = 42
    max_model_len = 12800
    max_num_seqs = 128
    tensor_parallel_size = 1
    enable_prefix_caching = True
    gpu_memory_utilization = 0.95
    enforce_eager = True

    ds = daft.from_pydict(samples)
    ds = ds.with_column(
        "caption",
        las_udf(
            QwenVLVideoUnderstandingVLLM,
            construct_args={
                "video_src_type": video_src_type,
                "model_path": model_path,
                "model_name": model_name,
                "dtype": dtype,
                "prompt": default_prompt,
                "max_caption_length": max_caption_length,
                "min_pixels": min_pixels,
                "max_pixels": max_pixels,
                "fps": fps,
                "batch_size": batch_size,
                "seed": seed,
                "max_model_len": max_model_len,
                "max_num_seqs": max_num_seqs,
                "tensor_parallel_size": tensor_parallel_size,
                "enable_prefix_caching": enable_prefix_caching,
                "gpu_memory_utilization": gpu_memory_utilization,
                "enforce_eager": enforce_eager,
            },
            num_gpus=tensor_parallel_size,
            batch_size=batch_size,
            concurrency=1,
        )(col("video_path"), col("prompt")),
    )

    ds.show()

    # ╭────────────────────────────────┬─────────────────────────┬─────────────────────────────────────────────────────────────╮
    # │ video_path                     ┆ prompt                  ┆ caption                                                     │
    # │ ---                            ┆ ---                     ┆ ---                                                         │
    # │ Utf8                           ┆ Utf8                    ┆ Utf8                                                        │
    # ╞════════════════════════════════╪═════════════════════════╪═════════════════════════════════════════════════════════════╡
    # │ https://las-cn-beijing-public… ┆ Please provide a detailed description of the video. …    ┆ This is an animated clip featuring a cartoon character, which is designed as a…           │
    # ╰────────────────────────────────┴─────────────────────────┴─────────────────────────────────────────────────────────────╯