Description #

Provides deep reasoning capabilities of large models in multimodal scenarios
Uses models with deep reasoning capabilities to analyze and understand images, videos, or text, and returns structured text output. The operator automatically constructs a message structure compliant with multimodal model specifications. Users only need to provide image, video, or text data as specified to complete inference.

Key features #

• Deep reasoning mechanism: The model automatically decomposes questions and performs logical reasoning before answering, generating a reasoning chain (reasoning_content)
• Multimodal scenario support: Supports simultaneous input of images, videos, and text, and automatically assembles multimodal messages
• Input simplification mechanism: Supports multiple data sources, including local files, HTTP/HTTPS URLs, TOS/S3 object storage, and more. Visual understanding capabilities can be achieved through simple configuration.
• Flexible reasoning mode: Supports controlling the deep reasoning mode (enabled / disabled / auto) via the thinking_type parameter, allowing flexible trade-offs between answer quality and performance

Input and output specifications #

• Input format:
  • Images (images): Supports string or list formats
    • Supported data source types:
      • url: Network resource address, supports http/https/tos/s3. TOS/S3 will automatically generate a pre-signed URL
      • base64: Base64 encoded data
      • binary: Binary data (internally converted to Base64 encoding)
  • Videos (videos): Supports string or list formats. Data source types are consistent with images
  • Text (texts): Supports string or list formats, used as user prompt input

(String and list types cannot be mixed within the same field)

• Output format:
  • Default mode: Returns struct type, containing the following fields:
    • llm_result: Final answer content from the model
    • reasoning_content: The model's reasoning chain / inference process
  • When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, struct will additionally include:
    • finish_reason: Reason for model result termination, possible values include:
      • stop: Normal termination
      • length: Exceeds token limit
      • content_filter: Content filtering policy triggered
  • When the environment variable LAS_LLM_BOTS_REFERENCES=true is set, struct will additionally include:
    • references: Retrieval results or external knowledge information referenced by the model, facilitating downstream transmission and parsing

Model capability enhancement #

• Reasoning chain visualization: The reasoning_content field outputs the model's reasoning process, making it easier to observe and review.
• Result reliability control: The finish_reason field identifies abnormal termination cases, such as length truncation or content filtering.
• Multimodal understanding: Supports mixed input parsing of images, videos, and text, suitable for complex multimodal scenarios.

Version compatibility description #

This operator (ArkLLMThinkingVision) is used differently in Daft 0.6.5 (and earlier versions) compared to Daft 0.6.14 (and later versions).

• In Daft 0.6.14 and later versions, you can specify the images, texts, and videos fields when providing input, which correspond to the model's image, text, and video inputs, and you can provide them simultaneously. Here is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMThinkingVision,
  
              construct_args={
  
                  "model": "doubao-seed-1.6",
  
                  "system_text": "",
  
                  "inference_type": "online",
  
              },
  
          )(videos=col("videos"), texts=""),
  
      )
  

• In Daft 0.6.5 and earlier versions, input only supports passing in video or image, not both at the same time. The input type is specified using multimodal_type, which accepts "video", "image", and "text". It also supports providing text information by specifying the prompt. Here is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMThinkingVision,
  
              construct_args={
  
                  "model": "doubao-seed-1.6",
  
                  "multimodal_type": "video",
  
                  "system_text": "",
  
                  "prompt": "",
  
                  "inference_type": "online",
  
              },
  
          )(col("videos")),
  
      )
  

Operator parameters #

Input #

Output #

By default, the returned field type is struct and includes the following fields:

• llm_result: Model output result
• reasoning_content: Reasoning chain content output by the model (reasoning process)

When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, the returned struct will additionally include the following field:

• finish_reason: The reason for model output termination. Common values include:
• stop: Normal termination
• length: Generated length reaches the upper limit
• content_filter: Content safety/filtering policy triggered

When the environment variable LAS_LLM_BOTS_REFERENCES=true is set, the returned struct will additionally include the following field:

• references: Reference information in JSON string format, used to carry external knowledge or citation results retrieved by the model, facilitating downstream parsing and display.

Both environment variables can be enabled simultaneously, in which case the output struct will include all of the above fields.

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use Daft to access the ModelArk multimodal deep reasoning model (Doubao series) for batch inference. Caution: The results of each inference by the large model may vary.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_thinking_vision import ArkLLMThinkingVision
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)
 
if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is obtained by creating it on the LAS service page
    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "videos": [
            f"https://{tos_dir_url}/public/shared_video_dataset/eating_56.mp4",
        ]
    }

    df = daft.from_pydict(samples)
    df = df.with_column(
        "llm_result",
        las_udf(
            ArkLLMThinkingVision,
            construct_args={
                "model": "doubao-seed-1.6-thinking",
                "system_text": "",
                "inference_type": "online",
            },
        )(videos=col("videos")),
    )

    df = df.with_column("reasoning_content", col("llm_result")["reasoning_content"])
    df = df.with_column("llm_result", col("llm_result")["llm_result"])
    df.show()

    #  Output (the result of each large model inference may vary)
    # ╭────────────────────────────────┬──────────────────────────────────────┬─────────────────────────────────────────────────────────────╮
    # │ videos                         ┆ llm_result                           ┆ reasoning_content                                           │
    # │ ---                            ┆ ---                                  ┆ ---                                                         │
    # │ String                         ┆ String                               ┆ String                                                      │
    # ╞════════════════════════════════╪══════════════════════════════════════╪═════════════════════════════════════════════════════════════╡
    # │ https://las-public-data-qa.to… ┆ The video presents a whimsical animated scene:   ┆ The user now needs to analyze the content of this animated video. First, observe the visuals. At the beginning, it is a… │
    # │                                ┆                                      ┆                                                             │
    # │                                ┆ The initial scene focuses on*…                     ┆                                                             │
    # ╰────────────────────────────────┴──────────────────────────────────────┴─────────────────────────────────────────────────────────────╯