Description #

Provides deep reasoning capabilities for large models in multimodal scenarios
Uses models with deep reasoning capabilities to analyze and understand images, videos, or text, and returns structured text output. The operator automatically constructs a message structure that complies with multimodal model specifications. Users only need to provide image, video, or text data according to the specification to complete inference.

Key features #

• Deep reasoning mechanism: The model automatically decomposes questions and performs logical reasoning before answering, generating a reasoning chain (reasoning_content)
• Multimodal scenario support: Supports simultaneous image, video, and text input, and automatically assembles messages for multimodal input
• Input simplification mechanism: Supports multiple data sources such as local files, HTTP/HTTPS URLs, and TOS/S3 object storage. Visual understanding capabilities can be achieved through simple configuration.
• Flexible reasoning mode: Supports controlling the deep reasoning mode (enabled / disabled / auto) via the thinking_type parameter, allowing flexible trade-offs between answer quality and performance

Input and output specifications #

• Input format:
  • Images (images): Supports string or list format
    • Supported data source types:
      • url: Network resource address, supports http/https/tos/s3. TOS/S3 will automatically generate a pre-signed URL
      • base64: Base64-encoded data
      • binary: Binary data (internally converted to Base64 encoding)
  • Videos (videos): Supports string or list format; data source types are the same as for images
  • Texts (texts): Supports string or list format, used as user prompts

(Mixing string and list types in the same field is not allowed)

• Output format:
  • Default mode: Returns struct type, containing the following fields:
    • llm_result: The model's final answer content
    • reasoning_content: The model's reasoning chain / inference process
  • When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, struct will additionally include:
    • finish_reason: The reason for the model result ending, with possible values including:
      • stop: Normal termination
      • length: Exceeds token limit
      • content_filter: Content filtering policy triggered
  • When the environment variable LAS_LLM_BOTS_REFERENCES=true is set, struct will additionally include:
    • references: Retrieval results or external knowledge information referenced by the model, facilitating downstream pass-through and parsing

Model capability enhancement #

• Chain-of-thought visualization: Outputs the model's reasoning process through the reasoning_content field, making it easy to observe and review
• Result reliability control: Identifies abnormal termination cases through the finish_reason field, such as length truncation or content filtering
• Multimodal understanding: Supports parsing mixed inputs of images, videos, and text, suitable for complex multimodal scenarios

Version compatibility notes #

This operator (ArkLLMThinkingVision) is used differently in Daft 0.6.5 (and earlier versions) compared to version 0.6.14 (and later versions).

• In Daft 0.6.14 and later versions, you can specify the images, texts, and videos fields as input, which correspond to the model's image, text, and video inputs, and simultaneous input is supported. The following is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMThinkingVision,
  
              construct_args={
  
                  "model": "doubao-seed-1.6",
  
                  "system_text": "",
  
                  "inference_type": "online",
  
              },
  
          )(videos=col("videos"), texts=""),
  
      )
  

• In Daft 0.6.5 and earlier versions, only video or image can be provided as input, and simultaneous input of both video and image is not supported. The input type is specified via multimodal_type, which supports "video", "image", and "text". It also supports providing text information by specifying the prompt. The following is an example:

  df = df.with_column(
  
          "llm_result",
  
          las_udf(
  
              ArkLLMThinkingVision,
  
              construct_args={
  
                  "model": "doubao-seed-1.6",
  
                  "multimodal_type": "video",
  
                  "system_text": "",
  
                  "prompt": "",
  
                  "inference_type": "online",
  
              },
  
          )(col("videos")),
  
      )
  

Operator parameters #

Input #

Output #

By default, the returned field type is struct, which includes the following fields:

• llm_result: Model output result
• reasoning_content: Chain-of-thought content output by the model (reasoning process)

When the environment variable LAS_LLM_FINISH_REASON_CHECK=true, the returned struct will additionally include the following field:

• finish_reason: The reason for the model output ending. Common values include:
• stop: Normal termination
• length: Generation length reached the upper limit
• content_filter: Hit content safety/filtering policy

When the environment variable LAS_LLM_BOTS_REFERENCES=true, the returned struct will additionally include the following field:

• references: Reference information in JSON string format, used to carry external knowledge or reference results retrieved by the model, facilitating downstream parsing and display.

Both environment variables can be enabled at the same time, in which case the output struct will include all of the above fields.

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use daft to access the ModelArk multimodal deep reasoning model (Doubao series) for batch inference. Caution: The results of each large model inference may vary.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_thinking_vision import ArkLLMThinkingVision
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)

if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is created on the LAS service page.
    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "videos": [
            f"https://{tos_dir_url}/public/shared_video_dataset/eating_56.mp4",
        ]
    }

    df = daft.from_pydict(samples)
    df = df.with_column(
        "llm_result",
        las_udf(
            ArkLLMThinkingVision,
            construct_args={
                "model": "doubao-seed-1.6-flash",
                "system_text": "",
                "inference_type": "online",
            },
        )(videos=col("videos")),
    )

    df = df.with_column("reasoning_content", col("llm_result")["reasoning_content"])
    df = df.with_column("llm_result", col("llm_result")["llm_result"])
    df.show()

    #  Output (the results of each large model inference may vary)
    # ╭────────────────────────────────┬────────────────────────┬───────────────────────────────────────────────────────────╮
    # │ videos                         ┆ llm_result             ┆ reasoning_content                                         │
    # │ ---                            ┆ ---                    ┆ ---                                                       │
    # │ String                         ┆ String                 ┆ String                                                    │
    # ╞════════════════════════════════╪════════════════════════╪═══════════════════════════════════════════════════════════╡
    # │ https://las-public-data-qa.to… ┆ ### Detailed description of the video content: ┆                                                           │
    # │                                ┆                        ┆ The user now needs me to analyze the content of this video and provide a detailed description. First, I... │
    # │                                ┆                        ┆                                                           │
    # │                                ┆ #### **Scenario and environment...**    ┆                                                           │
    # ╰────────────────────────────────┴────────────────────────┴───────────────────────────────────────────────────────────╯