Description #

Provides deep reasoning capabilities of large models in multimodal scenarios
Uses models with deep reasoning capabilities to analyze and understand images, videos, or text, and returns structured text output. The operator automatically constructs a message structure compliant with multimodal model specifications. Users only need to provide image, video, or text data as specified to complete inference.

Key features #

• Deep reasoning mechanism: The model automatically decomposes questions and performs logical reasoning before answering, generating a reasoning chain (reasoning_content)
• Multimodal scenario support: Supports simultaneous input of images, videos, and text, and automatically assembles multimodal messages
• Input simplification mechanism: Supports multiple data sources including local files, HTTP/HTTPS URLs, TOS/S3 object storage, and more. Visual understanding capabilities can be enabled through simple configuration.
• Flexible reasoning mode: Supports controlling the deep reasoning mode via the thinking_type parameter (enabled / disabled / auto), allowing flexible trade-offs between answer quality and performance

Input and output specifications #

• Input format:
  • Images (images): Supports string or list formats
    • Supported data source types:
      • url: Network resource address, supports http/https/tos/s3. TOS/S3 will automatically generate a pre-signed URL
      • base64: Base64 encoded data
      • binary: Binary data (internally converted to Base64 encoding)
  • Videos (videos): Supports string or list formats. Data source types are consistent with images
  • Text (texts): Supports string or list formats, used as user prompt input

(Mixing string and list types in the same field is not allowed)

• Output format:
  • Default mode: Returns struct type, including the following fields:
    • llm_result: The model's final answer content
    • reasoning_content: The model's reasoning chain / reasoning process
  • When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, the struct will additionally include:
    • finish_reason: The reason for the model result ending. Possible values include:
      • stop: Normal termination
      • length: Exceeds token limit
      • content_filter: Matches content filtering policy
  • When the environment variable LAS_LLM_BOTS_REFERENCES=true is set, the struct will additionally include:
    • references: Retrieval results or external knowledge information referenced by the model, facilitating downstream transmission and parsing

Model capability enhancement #

• Chain-of-thought visualization: The reasoning process of the model is output via the reasoning_content field, making it easier to observe and review.
• Result reliability control: The finish_reason field is used to identify abnormal termination cases, such as length truncation or content filtering.
• Multimodal understanding: Supports mixed input parsing of images, videos, and text, suitable for complex multimodal scenarios.

Version compatibility note #

This operator (ArkLLMThinkingVision) is used differently in Daft 0.6.5 (and earlier versions) compared to Daft 0.6.14 (and later versions).

• In Daft 0.6.14 and later versions, you can specify the images, texts, and videos fields as input, corresponding to the model's image, text, and video inputs, and simultaneous input is supported. Here is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMThinkingVision,
  
              construct_args={
  
                  "model": "doubao-seed-1.6",
  
                  "system_text": "",
  
                  "inference_type": "online",
  
              },
  
          )(videos=col("videos"), texts=""),
  
      )
  

• In Daft 0.6.5 and earlier versions, only video or image can be provided as input, and simultaneous input of video and image is not supported. The input type is specified via multimodal_type, which accepts "video", "image", and "text". It also supports providing text information by specifying the prompt parameter. Here is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMThinkingVision,
  
              construct_args={
  
                  "model": "doubao-seed-1.6",
  
                  "multimodal_type": "video",
  
                  "system_text": "",
  
                  "prompt": "",
  
                  "inference_type": "online",
  
              },
  
          )(col("videos")),
  
      )
  

Operator parameters #

Input #

Output #

By default, the returned field type is struct and includes the following fields:

• llm_result: Model output result
• reasoning_content: Chain-of-thought content output by the model (reasoning process)

When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, the struct will additionally include the following field:

• finish_reason: The reason for model output termination. Common values include:
• stop: Normal termination
• length: Generated length has reached the upper limit
• content_filter: Content safety/filtering policy triggered

When the environment variable LAS_LLM_BOTS_REFERENCES=true is set, the struct will additionally include the following field:

• references: Reference information in JSON string format, used to carry external knowledge or citation results retrieved by the model, facilitating downstream parsing and display.

Both environment variables can be enabled simultaneously, in which case the struct will include all of the above fields.

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use daft to access the ModelArk multimodal deep reasoning model (Doubao series) for batch inference. Note that the results of each large model inference may vary.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_thinking_vision import ArkLLMThinkingVision
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)

if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is obtained by creating it on the LAS service page
    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "videos": [
            f"https://{tos_dir_url}/public/shared_video_dataset/eating_56.mp4",
        ]
    }

    df = daft.from_pydict(samples)
    df = df.with_column(
        "llm_result",
        las_udf(
            ArkLLMThinkingVision,
            construct_args={
                "model": "doubao-1.5-thinking-vision-pro",
                "system_text": "",
                "inference_type": "online",
            },
        )(videos=col("videos")),
    )

    df = df.with_column("reasoning_content", col("llm_result")["reasoning_content"])
    df = df.with_column("llm_result", col("llm_result")["llm_result"])
    df.show()

    #  Output (each time the result of large language model inference may differ)
    # ╭────────────────────────────────┬────────────────────────────────────────────┬────────────────────────────────────────────╮
    # │ videos                         ┆ llm_result                                 ┆ reasoning_content                          │
    # │ ---                            ┆ ---                                        ┆ ---                                        │
    # │ Utf8                           ┆ Utf8                                       ┆ Utf8                                       │
    # ╞════════════════════════════════╪════════════════════════════════════════════╪════════════════════════════════════════════╡
    # │ https://las-public-data-qa.to… ┆ The video presents an animated segment: initially, it shows a        ┆ The user now needs to describe the content in the video. First, let's examine the scene:        │
    # │                                ┆ a multi-layer cartoon style...                              ┆ the beginning features a multi-layer cake...                             │
    # ╰────────────────────────────────┴────────────────────────────────────────────┴────────────────────────────────────────────╯