Description #

Large model multimodal understanding processor
Uses VeArk's large model to analyze and understand images, videos, or text, and returns text output.

Key features #

• Multimodal scenario support: Supports any combination of images, videos, and text as input. The operator automatically constructs a message structure that complies with multimodal model specifications.
• Rich input sources: Supports URLs using http/https/tos/s3 protocols, Base64 encoding, and raw binary data. TOS/S3 addresses are automatically pre-signed.
• Flexible data formats: Image, video, and text fields all support string (single) or list (multiple) input forms.

Input and output specifications #

• Input format:
  • images (images): string or list type.
  • videos (videos): string or list type.
  • texts (text): string or list type, used as input prompts.
• Output format:
  • Default mode: Returns a model-generated result of string type.
  • Diagnostic mode: When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, returns struct type, including:
    • llm_result: The text result generated by the model.
    • finish_reason: The reason for model output termination. Possible values include stop (normal termination), length (reached maximum length), and content_filter (content filtering).

Version compatibility notes #

This operator (ArkLLMVisionUnderstanding) is used differently in Daft 0.6.5 (and earlier versions) compared to version 0.6.14 (and later versions).

• In Daft 0.6.14 and later versions, you can specify the images, texts, and videos fields as input, corresponding to the model's image, text, and video inputs, and simultaneous input is supported. The following is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMVisionUnderstanding,
  
              construct_args={
  
                  "model": "Doubao-1.5-vision-pro",
  
                  "system_text": "",
  
              },
  
          )(videos=col("video_urls"), texts=col("prompts")),
  
      )
  

• Only video or image can be input; simultaneous input of video and image is not supported. The input type is specified via multimodal_type, which supports "video", "image", and "text". By specifying the prompt parameter, you can also input text information. The following is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMVisionUnderstanding,
  
              construct_args={
  
                  "model": "Doubao-1.5-vision-pro",
  
                  "multimodal_type": "video",
  
                  "prompt": "",
  
                  "inference_type": "online",
  
              },
  
          )(col("videos")),
  
      )
  

Operator parameters #

Input #

Output #

By default, the returned field type is string, and the content is the model output result.
When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, the returned field type is struct, including the following fields:

• llm_result: Model output result
• finish_reason: Reason for model output termination. Common values include stop (normal termination), length (reached length limit), and content_filter (triggered content filtering policy).

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use this operator to interpret videos.

from __future__ import annotations
import os
import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_vision_understanding import ArkLLMVisionUnderstanding
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)
  
if __name__ == "__main__":
    # Environment variable LAS_API_KEY must be configured: LAS_API_KEY is obtained by creating it on the LAS service page
    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "videos": [
            f"https://{tos_dir_url}/public/shared_video_dataset/eating_56.mp4",
        ],
        "prompts": [""],
    }
    
    df = daft.from_pydict(samples)
    df = df.with_column(
        "llm_result",
        las_udf(
            ArkLLMVisionUnderstanding,
            construct_args={
                "model": "Doubao-1.5-vision-pro",
                "system_text": "",
                "inference_type": "online",
            },
        )(videos=col("videos"), texts=col("prompts")),
    )
    df.show()

    # Output (the result of each large model inference may vary)
    # ╭────────────────────────────────┬────────────────┬─────────────────────────────────────────────────────────────╮
    # │ videos                         ┆ prompts        ┆ llm_result                                                  │
    # │ ---                            ┆ ---            ┆ ---                                                         │
    # │ String                         ┆ String         ┆ String                                                      │
    # ╞════════════════════════════════╪════════════════╪═════════════════════════════════════════════════════════════╡
    # │ https://las-public-data-qa.to… ┆ What is in the video? ┆ The video showcases a cartoon-style scene, with the main element being a two-story red egg… │
    # ╰────────────────────────────────┴────────────────┴─────────────────────────────────────────────────────────────╯