Description #

Large model multimodal understanding processor
Uses VolcanoVeArk's large model to analyze and understand images, videos, or text, and returns a text output.

Key features #

• Multimodal scenario support: Supports any combination of images, videos, and text as input. The operator automatically constructs a message structure that complies with multimodal model specifications.
• Rich input sources: Supports URLs using http/https/tos/s3 protocols, Base64 encoding, and raw binary data. TOS/S3 addresses are automatically pre-signed.
• Flexible data formats: The image, video, and text columns all support string (single) or list (multiple) formats as input.

Input and output specifications #

• Input format:
  • images (images): string or list type.
  • videos (videos): string or list type.
  • texts (text): string or list type, used as user prompts.
• Output format:
  • Default mode: Returns the model-generated result as a string type.
  • Diagnostic mode: When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, returns a struct type containing:
    • llm_result: The text result generated by the model.
    • finish_reason: The reason for the model output termination. Possible values include stop (normal end), length (maximum length reached), and content_filter (content filtering).

Version compatibility notes #

This operator (ArkLLMVisionUnderstanding) is used differently in Daft 0.6.5 (and earlier versions) compared to Daft 0.6.14 (and later versions).

• In Daft 0.6.14 and later versions, you can specify the images, texts, and videos fields as input, corresponding to the model's image, text, and video inputs, and simultaneous input is supported. The following is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMVisionUnderstanding,
  
              construct_args={
  
                  "model": "Doubao-1.5-vision-pro",
  
                  "system_text": "",
  
              },
  
          )(videos=col("video_urls"), texts=col("prompts")),
  
      )
  

• In Daft 0.6.5 and earlier versions, only video or image can be input, and simultaneous input of video and image is not supported. The input type is specified via multimodal_type, which supports "video", "image", and "text". It also supports passing text information by specifying the prompt. The following is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMVisionUnderstanding,
  
              construct_args={
  
                  "model": "Doubao-1.5-vision-pro",
  
                  "multimodal_type": "video",
  
                  "prompt": "",
  
                  "inference_type": "online",
  
              },
  
          )(col("videos")),
  
      )
  

Operator parameters #

Input #

Output #

By default, the returned field type is string, and the content is the model output result.
When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, the returned field type is struct, containing the following fields:

• llm_result: Model output result
• finish_reason: Reason for model output termination. Common values include stop (normal termination), length (reached maximum length), and content_filter (triggered content filtering policy).

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use the ArkLLMVisionUnderstanding operator to understand videos.

from __future__ import annotations
import os
import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_vision_understanding import ArkLLMVisionUnderstanding
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)
  
if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is obtained by creating it on the LAS service page
    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "videos": [
            f"https://{tos_dir_url}/public/shared_video_dataset/eating_56.mp4",
        ],
        "prompts": [""],
    }
    
    df = daft.from_pydict(samples)
    df = df.with_column(
        "llm_result",
        las_udf(
            ArkLLMVisionUnderstanding,
            construct_args={
                "model": "Doubao-1.5-vision-pro",
                "system_text": "",
                "inference_type": "online",
            },
        )(videos=col("videos"), texts=col("prompts")),
    )
    df.show()

    # Output (the results of large model inference may differ each time)
    # ╭────────────────────────────────┬────────────────┬─────────────────────────────────────────────────────────────╮
    # │ videos                         ┆ prompts        ┆ llm_result                                                  │
    # │ ---                            ┆ ---            ┆ ---                                                         │
    # │ String                         ┆ String         ┆ String                                                      │
    # ╞════════════════════════════════╪════════════════╪═════════════════════════════════════════════════════════════╡
    # │ https://las-public-data-qa.to… ┆ What is in the video? ┆ The video shows a cartoon-style scene, with the main element being a two-story red egg-shaped building. │
    # ╰────────────────────────────────┴────────────────┴─────────────────────────────────────────────────────────────╯