Description #

Provides deep thinking capabilities for large models in multimodal scenarios
Uses the Doubao-1.5-thinking-pro model, which possesses deep thinking capabilities, to analyze and interpret text, and returns structured output. Before answering questions, the model analyzes and decomposes the question, and generates answers based on this analysis.

Key features #

• Deep thinking mechanism: Before answering questions, the model automatically decomposes the question and performs logical reasoning, generating a reasoning chain (reasoning_content)
• Input simplification mechanism: Supports multiple data sources, including local files, HTTP/HTTPS URLs, and TOS/S3 object storage. Visual understanding capabilities can be achieved through simple configuration.
• Flexible thinking mode: Supports controlling the deep thinking mode through the thinking_type parameter (enabled / disabled / auto), allowing flexible trade-offs between answer quality and performance.

Input and output specifications #

• Input format:
  • Images (images): Supports string or list format
    • Supported data source types:
      • url: Network resource address, supports http/https/tos/s3. TOS/S3 will automatically generate a pre-signed URL.
      • base64: Base64-encoded data
      • binary: Binary data (internally converted to Base64 encoding)
  • Videos (videos): Supports string or list format; data source types are the same as for images
  • Texts (texts): Supports string or list format, used as input for user prompts.

(Mixing string and list types in the same field is not allowed)

• Output format:
  • Default mode: Returns a struct containing the following fields:
    • llm_result: The model's final answer content
    • reasoning_content: The model's reasoning chain / reasoning process
  • When the environment variable LAS_LLM_FINISH_REASON_CHECK=true is set, struct will additionally include:
    • finish_reason: The reason for the model result termination, with possible values including:
      • stop: Normal termination
      • length: Exceeds token limit
      • content_filter: Hits content filtering policy
  • When the environment variable LAS_LLM_BOTS_REFERENCES=true is set, struct will additionally include:
    • references: Retrieval results or external knowledge referenced by the model, for downstream transmission and parsing.

Model capability enhancement #

• Reasoning chain visualization: Outputs the model's reasoning process through the reasoning_content field, facilitating observation and retrospective analysis.
• Result reliability control: Identifies abnormal termination situations, such as truncation due to length or content filtering, through the finish_reason field.
• Multimodal understanding: Supports parsing mixed inputs of images, videos, and text, suitable for complex multimodal scenarios

Version compatibility notes #

This operator (ArkLLMThinkingVision) is used differently in Daft 0.6.5 (and earlier versions) compared to version 0.6.14 (and later versions).

• In Daft 0.6.14 and later versions, you can specify the images, texts, and videos fields as input, which correspond to the model's image, text, and video inputs, and simultaneous input is supported. The following is an example:

df = df.with_column(

        "llm_result",

        las_udf(

            ArkLLMThinkingVision,

            construct_args={

                "model": "doubao-seed-1.6",

                "system_text": "",

                "inference_type": "online",

            },

        )(videos=col("videos"), texts=""),

    )

• In Daft 0.6.5 and earlier versions, only video or image can be provided as input, and simultaneous input of video and image is not supported. The input type is specified via multimodal_type, which supports values of "video", "image", and "text". You can also provide text information by specifying the prompt. The following is an example:

df = df.with_column(

        "llm_result",

        las_udf(

            ArkLLMThinkingVision,

            construct_args={

                "model": "doubao-seed-1.6",

                "multimodal_type": "video",

                "system_text": "",

                "prompt": "",

                "inference_type": "online",

            },

        )(col("videos")),

    )

Operator parameters #

Input #

Output #

By default, the returned field type is struct and contains the following fields:

• llm_result: Model output result
• reasoning_content: The chain-of-thought content output by the model (reasoning process)

When the environment variable LAS_LLM_FINISH_REASON_CHECK=true, the returned struct will additionally include the following field:

• finish_reason: The reason for the model output ending. Common values include:
• stop: Normal termination
• length: Generation length has reached the upper limit
• content_filter: Hit content safety/filtering policy

When the environment variable LAS_LLM_BOTS_REFERENCES=true, the returned struct will additionally include the following field:

• references: Reference information in JSON string format, used to carry external knowledge or reference results retrieved by the model, facilitating downstream parsing and display.

Both environment variables can be enabled at the same time, in which case the output struct will contain all of the above fields.

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use Daft to access the Volcengine ModelArk multimodal deep thinking model (Doubao series) for text reasoning. Note that each inference may yield different results.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_thinking_vision import ArkLLMThinkingVision
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)

if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is obtained by creating it on the LAS service page
    queries = {
        "query": [
            "",
        ]
    }

    df = daft.from_pydict(queries)

    df = df.with_column(
        "llm_result",
        las_udf(
            ArkLLMThinkingVision,
            construct_args={
                "model": "doubao-1.5-thinking-pro",
                "inference_type": "online",
            },
        )(texts=col("query")),
    )

    df = df.with_column("reasoning_content", col("llm_result")["reasoning_content"])
df = df.with_column("llm_result", col("llm_result")["llm_result"])
    df.show()

    #  Output (the result of each large model inference may vary)
    # ╭──────────────────────────────────┬──────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────╮
    # │ query                            ┆ llm_result                                           ┆ reasoning_content                                        │
    # │ ---                              ┆ ---                                                  ┆ ---                                                      │
    # │ Utf8                             ┆ Utf8                                                 ┆ Utf8                                                     │
    # ╞══════════════════════════════════╪══════════════════════════════════════════════════════╪══════════════════════════════════════════════════════════╡
    # │ Help me plan a 10-day trip to Xinjiang in May… ┆                                                      ┆ Okay, the user asked me to help plan a 10-day trip to Xinjiang in May. First… │
    # │                                  ┆                                                      ┆                                                          │
    # │                                  ┆ May is the "early summer golden period" for traveling in Xinjiang, when the grasslands are turning green and the flower fields are just beginning to bloom… ┆                                                          │
    # ╰──────────────────────────────────┴──────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────╯