Description #

Qwen2.5-VL multimodal video understanding model – temporal semantic parsing and natural language description generation

Key features #

• Multimodal temporal modeling
• Supports three video formats: URL, Base64 encoding, and binary stream
• Spatiotemporal joint modeling
• Captures spatiotemporal features and semantic associations of videos
• Dialog-style prompt support
• Guides generation direction via the prompt parameter
• Resource usage
• It is recommended to use a GPU with 48GB or more VRAM

Optimization features #

• Optimized for mixed Chinese and English scenarios: enhanced Chinese semantic processing
• Supports first resizing videos to a consistent size; it is recommended to set the size according to the video and GPU specifications

Operator parameters #

Input #

Output #

The processed array, where each element is the visual understanding result for each video.

Parameters #

If a parameter does not have a default value, it is required

Examples #

The following code demonstrates how to use daft to run the operator for video content understanding and generate descriptions according to instructions.

from __future__ import annotations

import logging
import os

import ray

import daft
from daft import col
from daft.las.functions.multimodal.qwen_vl_video_understanding import QwenVLVideoUnderstanding
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    os.environ["DAFT_RUNNER"] = "ray"

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()
    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "video_path": [
            f"https://{tos_dir_url}/public/shared_video_dataset/eating_56.mp4"
        ],
        "prompt": [""]  
    }

    video_src_type = "video_url"
    model_path = os.getenv("MODEL_PATH", "/opt/las/models")
    model_name = os.getenv("MODEL_NAME", "Qwen/Qwen2.5-VL-7B-Instruct")
    dtype = "bfloat16"
    use_flash_attention_2 = True
    default_prompt = None
    max_caption_length = 256
    min_pixels = 320 * 160
    max_pixels = 320 * 160
    fps = 1
    batch_size = 2
    rank = 0

    ds = daft.from_pydict(samples)
    ds = ds.with_column(
        "caption",
        las_udf(
            QwenVLVideoUnderstanding,
            construct_args={
                "video_src_type": video_src_type,
                "model_path": model_path,
                "model_name": model_name,
                "dtype": dtype,
                "use_flash_attention_2": use_flash_attention_2,
                "prompt": default_prompt,
                "max_caption_length": max_caption_length,
                "min_pixels": min_pixels,
                "max_pixels": max_pixels,
                "batch_size": batch_size,
                "rank": rank,
            },
            num_gpus=1,
            batch_size=2,
            concurrency=1,
        )(col("video_path"), col("prompt")),
    )

    ds.show()

    # ╭────────────────────────────────┬─────────────────────────┬─────────────────────────────────────────────────────────────╮
    # │ video_path                     ┆ prompt                  ┆ caption                                                     │
    # │ ---                            ┆ ---                     ┆ ---                                                         │
    # │ Utf8                           ┆ Utf8                    ┆ Utf8                                                        │
    # ╞════════════════════════════════╪═════════════════════════╪═════════════════════════════════════════════════════════════╡
    # │ tos://las-cn-beijing-public-o… ┆ Please provide a detailed description of the video. …     ┆ This is an animation clip. A cartoon character appears in the scene, and it looks like a…          │
    # ╰────────────────────────────────┴─────────────────────────┴─────────────────────────────────────────────────────────────╯