Description #

Large model multimodal understanding processor
Use VeArk's large model to analyze and interpret images, videos, or text, and return the output as text.

Key features #

• Multimodal scenario support: Supports any combination of images, videos, and text as input. The operator automatically constructs a message structure that complies with multimodal model specifications.
• Rich input sources: Supports http/https/tos/s3 protocol URLs, Base64 encoding, and raw binary data. TOS/S3 addresses are automatically pre-signed.
• Flexible data formats: The image, video, and text columns all accept string (single) or list (multiple) input formats.

Input and output specifications #

• Input format:
  • images (image): string or list type.
  • videos (video): string or list type.
  • texts (text): string or list type, used as user prompts.
• Output format:
  • Default mode: Returns a model-generated result of string type.
  • Diagnostic mode: When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, returns a struct type containing:
    • llm_result: The text result generated by the model.
    • finish_reason: The reason for model output termination. Possible values include stop (normal completion), length (maximum length reached), and content_filter (content filtering).

Version compatibility notes #

This operator (ArkLLMVisionUnderstanding) is used differently in Daft 0.6.5 (and earlier versions) compared to Daft 0.6.14 (and later versions).

• In Daft 0.6.14 and later versions, you can specify the images, texts, and videos fields as input, corresponding to the model's image, text, and video inputs. Simultaneous input is supported. Here is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMVisionUnderstanding,
  
              construct_args={
  
                  "model": "doubao-1.5-vision-pro-32k",
  
                  "system_text": "",
  
                  "inference_type": "online",
  
              },
  
          )(videos=col("video_urls"), texts=col("prompts")),
  
      )
  

• In Daft 0.6.5 and earlier versions, only video or image can be provided as input, and simultaneous input of video and image is not supported. The input type is specified via multimodal_type, which supports "video", "image", and "text". It also supports providing text information by specifying the prompt parameter. Here is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMVisionUnderstanding,
  
              construct_args={
  
                  "model": "doubao-1.5-vision-pro-32k",
  
                  "multimodal_type": "video",
  
                  "prompt": "",
  
                  "inference_type": "online",
  
              },
  
          )(col("videos")),
  
      )
  

Operator parameters #

Input #

Output #

By default, the return field type is string, and the returned content is the model output result.
When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, the return field type is struct, containing the following fields:

• llm_result: Model output result
• finish_reason: Reason for model output termination. Common values include stop (normal termination), length (maximum length reached), and content_filter (content filtering policy triggered).

Parameters #

If a parameter does not have a default value, it is a required parameter.

Examples #

The following code demonstrates how to use this operator to interpret images.

from __future__ import annotations
import os
import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_vision_understanding import ArkLLMVisionUnderstanding
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)
  
if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY can be created and obtained on the LAS service page
    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "image_urls": [
            f"https://{tos_dir_url}/public/shared_image_dataset/cat_ip_adapter.jpeg"
        ],
        "prompts": [""],
    }

    df = daft.from_pydict(samples)
    df = df.with_column(
        "llm_result",
        las_udf(
            ArkLLMVisionUnderstanding,
            construct_args={
                "model": "doubao-1.5-vision-pro-32k",
                "system_text": "",
                "inference_type": "online", # This model only supports online inference
            },
        )(images=col("image_urls"), texts=col("prompts")),
    )
    df.show()

    # Output (Each inference result may differ)
    # ╭────────────────────────────────┬────────────────┬─────────────────────────────────────────────────────────────╮
    # │ image_urls                     ┆ prompts        ┆ llm_result                                                  │
    # │ ---                            ┆ ---            ┆ ---                                                         │
    # │ String                         ┆ String         ┆ String                                                      │
    # ╞════════════════════════════════╪════════════════╪═════════════════════════════════════════════════════════════╡
    # │ https://las-public-data-qa.to… ┆ What is in the picture? ┆ The image features an anthropomorphic cat with cream and light brown fur, and has a pair of… │
    # ╰────────────────────────────────┴────────────────┴─────────────────────────────────────────────────────────────╯