This article explains how to access the models provided by the ModelArk large model service platform of Volcano Engine in LAS, along with usage examples:

• Deep reasoning
• Text generation
• Visual understanding
• Image-text embedding

Prerequisites #

• Enable AI Data Lake Service (LAS).
• Create an API KEY in the AI Data Lake Service and configure the environment variable: LAS_API_KEY.

• To use the development machine mode to call Doubao operators, create a development machine in advance. For details, see Create a development machine.
• To use Task Management to call Doubao operators, create a task in advance. For details, see Task Management.

Operation guide #

The AI Data Lake Service (LAS) allows you to register and access the large models provided by ModelArk of Volcano Engine through the API Key Management module. For the models supported in LAS, refer to Data Marketplace - Operator Marketplace - ModelArk Large Model Series. Supported scenarios include text generation, video understanding, deep reasoning, image-text vectorization, and more.
The following sections provide specific usage instructions, including basic examples and parameter descriptions.

Deep reasoning #

For the deep reasoning models available on ModelArk, you can use the operators provided by the AI Data Lake Service to easily perform batch computations on data. Supported model examples include:

• doubao-seed-1.6
• doubao-seed-1.6-thinking
• doubao-1.5-thinking-pro
• doubao-1.5-thinking-vision-pro
• doubao-seed-1.6-flash
• ......

Quick start

The following example uses the doubao-1.5-thinking-pro model. Log in to the development machine and run the following code:

from __future__ import annotations

import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_thinking_vision import ArkLLMThinkingVision
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is created and obtained on the AI Data Lake service page
    queries = {
        "query": [
            "帮我规划5月去新疆的10天旅行安排",
        ]
    }

    df = daft.from_pydict(queries)
    df = df.with_column(
        "llm_result",
        las_udf(
            ArkLLMThinkingVision,
            construct_args={
                "model": "doubao-1.5-thinking-pro",
                "multimodal_type": "text",
                "inference_type": "online",
            },
        )(col("query")),
    )

    df = df.with_column("reasoning_content", col("llm_result").struct.get("reasoning_content"))
    df = df.with_column("llm_result", col("llm_result").struct.get("llm_result"))

    output_pd_df = df.to_pandas()
    df.show()

Basic usage

The following demonstrates how to use the doubao-1.5-thinking-vision-pro model to process video data in TOS, output the video path, video understanding result (llm_result), reasoning chain (reasoning_content), and save the results to TOS.

from __future__ import annotations

import os

import ray

import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_thinking_vision import ArkLLMThinkingVision
from daft.las.functions.udf import las_udf
from daft.las.io.tos import TOSConfig

# The following two lines convert a standalone program into a distributed task submitted to the Ray cluster. The following two lines are required when executing in "Task Management". The following two lines are not required for debugging on the development machine.
ray.init()
daft.context.set_runner_ray("ray://127.0.0.1:10001")

def run_pipeline(input_tos_dir, output_tos_path):
    input_s3_dir = input_tos_dir.rstrip('/')  
    tos_config = TOSConfig.from_env()
    io_config = daft.io.IOConfig(s3=tos_config.to_s3_config())

    # Load all mp4 files under the "input_s3_dir" path on TOS
    df = daft.from_glob_path(
        f"{input_s3_dir}/*.mp4",
        io_config=io_config,
    )

    # Call the ArkLLMThinkingVision function to perform inference on each video file
    df = df.with_column(
        "response",
        las_udf(
            ArkLLMThinkingVision,
            construct_args={
                "model": "doubao-1.5-thinking-vision-pro",
                "multimodal_type": "video",
                "prompt": "视频里有什么？",
                "inference_type": "online",
            },
        )(col("path")),
    )

    # Extract the reasoning_content and llm_result fields from the inference results
    df = df.with_column("reasoning_content", col("response").struct.get("reasoning_content"))
    df = df.with_column("llm_result", col("response").struct.get("llm_result"))

    # Select the required fields
    df = df.select("path", "llm_result", "reasoning_content")

    # Write to the "output_tos_path" path on TOS
    df.write_csv(output_tos_path, io_config=io_config)

if __name__ == "__main__":
    input_tos_dir = os.getenv("input_tos_dir", "input_tos_dir").replace("tos://", "s3://", 1)
    output_tos_path = os.getenv("output_tos_path", "output_tos_path").replace("tos://", "s3://", 1)
    run_pipeline(input_tos_dir, output_tos_path)

Set environment variables:

# TOS configuration parameters: since the data is stored on TOS, TOS parameter information must be configured
 export TOS_ENDPOINT="https://tos-cn-beijing.ivolces.com"
 export LAS_TOS_ACCESS_KEY="xxx"
 export LAS_TOS_SECRET_KEY="xx="
 export LAS_API_KEY="xxx"
 
# Input file path on TOS
 export input_tos_dir="xx"
# Output file path on TOS
 export output_tos_path="xx"
 

Operation steps

• Debugging steps in the development machine:

  1. On the development machine, save the above code to a specific path, for example doubao_think_demo.py
  2. Store the video data required for this exercise in TOS and configure the file path in the code
  3. Set the above environment variables and configure input_tos_dir with the TOS address where the video data is stored
  4. Debug the above code on the development machine: python doubao_think_demo.py

• Perform distributed computation in Task Management

  1. Create a custom image for the above development machine task through LAS Console > Data Processing > Development Machine (Save Image). Configure parameters such as namespace, image name, version, and development machine, and store the image in the Volcano Image Repository.
  2. Create a task in Task Management, select Custom Image or locate the storage path of the above image in the Volcano Image Repository, configure the script execution command, and create the task after completing the configuration.
  3. Execute the task in Task Management, and view the task progress in Instance Management through logs and the Daft UI.

Common parameters

In the above example, the ArkLLMThinkingVision class is used to perform inference on models within ModelArk. The request body, usage method, and configuration of request parameters are described as follows:

las_udf(
            ArkLLMThinkingVision,
            construct_args={
                "model": "doubao-1.5-thinking-vision-pro",
                "multimodal_type": "video",
                "prompt": "视频里有什么？",
                "inference_type": "online",
            },
        )

For explanations of other parameters, refer to the descriptions in the Operator Plaza of AI Data Lake Operator Plaza.

Text generation #

For the text generation models available on ModelArk, you can use the operators provided by AI Data Lake to easily perform batch processing on data. Supported model examples include:

• doubao-1.5-pro-256K
• doubao-1.5-pro-32k
• ......

Quick start

from __future__ import annotations

import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_text_generate import ArkLLMTextGenerate
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is obtained by creating it on the LAS service page
    queries = {
        "query": [
            "中国的首都在哪里",
            "十字花科植物有哪些",
        ]
    }

    ds = daft.from_pydict(queries)
    ds = ds.with_column(
        "llm_result",
        las_udf(
            ArkLLMTextGenerate,
            construct_args={
                "model": "doubao-1.5-pro-32k",
                "inference_type": "online",
            },
        )(col("query")),
    )
    ds.show()

Common parameters

In the above example, the ArkLLMTextGenerate class is used to perform inference on models within ModelArk. The request body, usage method, and configuration of request parameters are described as follows:

las_udf(
            ArkLLMTextGenerate,
            construct_args={
                "model": "doubao-1.5-pro-32k",
                "inference_type": "online",
            },
        )

For explanations of other parameters, refer to the descriptions in the Operator Plaza of AI Data Lake Operator Plaza.

Visual understanding #

For the visual understanding models available on ModelArk, you can use the operators provided by AI Data Lake to easily perform batch processing on data. Supported model examples include:

• doubao-1.5-vision-pro-32k
• doubao-1.5-vision-pro
• ......

Quick start

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_vision_understanding import ArkLLMVisionUnderstanding
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is obtained by creating it on the LAS service page
    TOS_TEST_DIR = os.getenv("TOS_TEST_DIR", "tos_bucket")
    samples = {"videos": [f"tos://{TOS_TEST_DIR}/ark_llm_vision_understanding/eating_56.mp4"]}

    df = daft.from_pydict(samples)
    df = df.with_column(
        "llm_result",
        las_udf(
            ArkLLMVisionUnderstanding,
            construct_args={
                "model": "doubao-1.5-vision-pro",
                "multimodal_type": "video",
                "prompt": "视频里有什么？",
                "inference_type": "batch",
            },
        )(col("videos")),
    )
    df.show()

Common parameters

In the above example, the ArkLLMVisionUnderstanding class is used to perform inference on models within ModelArk. The request body, usage method, and configuration of request parameters are described as follows:

las_udf(
            ArkLLMVisionUnderstanding,
            construct_args={
                "model": "doubao-1.5-vision-pro",
                "multimodal_type": "video",
                "prompt": "视频里有什么？",
                "inference_type": "batch",
            },
        )

For explanations of other parameters, refer to the descriptions in the Operator Plaza of AI Data Lake Operator Plaza.

Image-text embedding #

For the image-text embedding models available on ModelArk, you can use the operators provided by AI Data Lake to easily perform batch processing on data. Supported model examples include:

• doubao-embedding-vision
• ......

Quick start

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.ark_llm.doubao_embedding_vision import DoubaoEmbeddingVision
from daft.las.functions.udf import las_udf

if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is obtained by creating LAS_API_KEY on the LAS service page
    TOS_TEST_DIR = os.getenv("TOS_TEST_DIR", "tos_bucket")
    samples = {
        "image_path": [f"tos://{TOS_TEST_DIR}/doubao_embedding_vision/cat_ip_adapter.jpeg"],
        "text": ["猫"],
    }

    df = daft.from_pydict(samples)
    # Calculate vectorized data for images and text
    df = df.with_column(
        "embeeding_for_image_text",
        las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "image_format": "jpeg",
            },
        )(col("image_path"), col("text")),
    )

    # Calculate vectorized data for images
    df = df.with_column(
        "embeeding_for_image",
        las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "image_format": "jpeg",
            },
        )(col("image_path")),
    )

    # Calculate vectorized data for text
    df = df.with_column(
        "embeeding_for_text",
        las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "multimodal_type": "text",
            },
        )(col("text")),
    )
    df.show()

Common parameters

In the above example, the DoubaoEmbeddingVision class is used to perform inference on models within ModelArk. The request body, usage, and request parameter configuration are described as follows:

las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "multimodal_type": "text",
            },
        )

For explanations of other parameters, refer to the introduction in the AI Data Lake Operator Plaza.