Description #

The Qwen2.5-VL multimodal image understanding model supports visual semantic parsing and natural language description generation, making it suitable for a variety of image analysis scenarios.

Main features #

• Multimodal temporal modeling: Supports three image input formats (URL, Base64 encoding, binary stream), allowing flexible adaptation to different data sources.
• Conversational prompt support: Customize the generation direction via the prompt parameter to meet diverse business requirements.
• Efficient inference: Integrates the VLLM inference engine and supports bfloat16, float16, and float32 precision, fully utilizing GPU computing power.
• It is recommended to use GPUs with 48GB or more GPU memory.

Applicable scenarios #

• Image content understanding and summarization
• Intelligent monitoring and event analysis
• Multimodal dialogue and interaction
• Other AI applications requiring image semantic parsing

Caution #

• Only GPU environments are supported. It is recommended to set parameters appropriately based on GPU memory and business requirements.
• For best results, standardize images before inference.

Operator parameters #

Input #

Output #

The processed array, with each element representing the understanding result for each image.

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use daft to run the operator for image content understanding and generate descriptions according to instructions.

from __future__ import annotations

import logging
import os

import ray

import daft
from daft import col
from daft.las.functions.multimodal.qwen_vl_image_understanding_vllm import QwenVLImageUnderstandingVLLM
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    os.environ["DAFT_RUNNER"] = "ray"

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()
    daft.set_execution_config(actor_udf_ready_timeout=600)
    daft.set_execution_config(min_cpu_per_task=0)

    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "image_path": [
            f"https://{tos_dir_url}/public/shared_image_dataset/cat_ip_adapter.jpeg"
        ],
        "prompt": [""],
    }

    image_src_type = "image_url"
    model_path = os.getenv("MODEL_PATH", "/opt/las/models")
    model_name = os.getenv("MODEL_NAME", "Qwen/Qwen2.5-VL-7B-Instruct")
    dtype = "bfloat16"
    default_prompt = None
    max_caption_length = 256
    resized_height = 256
    resized_width = 256
    batch_size = 2
    seed = 42
    max_model_len = 12800
    max_num_seqs = 128
    tensor_parallel_size = 1
    enable_prefix_caching = False
    gpu_memory_utilization = 0.95
    enforce_eager = True

    ds = daft.from_pydict(samples)
    ds = ds.with_column(
        "caption",
        las_udf(
            QwenVLImageUnderstandingVLLM,
            construct_args={
                "image_src_type": image_src_type,
                "model_path": model_path,
                "model_name": model_name,
                "dtype": dtype,
                "prompt": default_prompt,
                "max_caption_length": max_caption_length,
                "resized_height": resized_height,
                "resized_width": resized_width,
                "batch_size": batch_size,
                "seed": seed,
                "max_model_len": max_model_len,
                "max_num_seqs": max_num_seqs,
                "tensor_parallel_size": tensor_parallel_size,
                "enable_prefix_caching": enable_prefix_caching,
                "gpu_memory_utilization": gpu_memory_utilization,
                "enforce_eager": enforce_eager,
            },
            num_gpus=tensor_parallel_size,
            batch_size=batch_size,
            concurrency=1,
        )(col("image_path"), col("prompt")),
    )

    ds.show()

    # ╭────────────────────────────────┬─────────────────────────┬─────────────────────────────────────────────────────────────╮
    # │ image_path                     ┆ prompt                  ┆ caption                                                     │
    # │ ---                            ┆ ---                     ┆ ---                                                         │
    # │ Utf8                           ┆ Utf8                    ┆ Utf8                                                        │
    # ╞════════════════════════════════╪═════════════════════════╪═════════════════════════════════════════════════════════════╡
    # │ https://las-cn-beijing-public… ┆ Please provide a detailed description of the image. …     ┆ This image shows an anthropomorphic cat dressed in vintage-style clothing, including a blue garment… │
    # ╰────────────────────────────────┴─────────────────────────┴─────────────────────────────────────────────────────────────╯