Description #

Dedicated processor for large model text generation (Doubao/DeepSeek)

Key features #

• Optimization for pure text scenarios: Automatically constructs message structures compliant with model specifications based on user input text data
• Input simplification mechanism: Natively supports str-type input and automatically encapsulates it in the format {role: user, content: text}
• Multi-task support: Translation, summarization, question answering, and other NLP scenarios available out of the box
• Dual prompt system:
  • system_content: System-level behavioral guidance (such as translation style control)
  • prompt: User-level instruction template (supports {query} placeholder replacement)

Input and output specifications #

• Input format: Pure text data
• Output format:
  • Default mode: str-type generated result
  • Diagnostic mode: Set the environment variable LAS_LLM_FINISH_REASON_CHECK=true to return the complete generated result and the reason for model result termination:
    • llm_result: str-type, generated result
    • finish_reason: str-type, reason for model result termination; possible values: stop, length, content_filter

Operator parameters #

Input #

Output #

(By default) When the environment variable LAS_LLM_FINISH_REASON_CHECK=false, the returned field type is str.
When the environment variable LAS_LLM_FINISH_REASON_CHECK=true, the returned field type is struct and includes the following fields:

• llm_result: Model output result
• finish_reason: Reason for model output termination

Parameters #

If a parameter does not have a default value, it is required

Examples #

The following code demonstrates how to use daft to access the VolcanoVeArk text generation model for batch inference. Note that the result of each large model inference may vary.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.ark_llm.ark_llm_text_generate import ArkLLMTextGenerate
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)

if __name__ == "__main__":
    # The environment variable LAS_API_KEY must be configured: LAS_API_KEY is obtained by creating it on the LAS service page
    queries = {
        "query": [
            "",
            "",
        ]
    }

    ds = daft.from_pydict(queries)
    ds = ds.with_column(
        "llm_result",
        las_udf(
            ArkLLMTextGenerate,
            construct_args={
                "model": "doubao-lite-32k",
                "inference_type": "online",
            },
        )(col("query")),
    )
    ds.show()

    #  Output (the result of each large model inference may differ)
    # ╭────────────────────┬───────────────────────────────────────╮
    # │ query              ┆ llm_result                            │
    # │ ---                ┆ ---                                   │
    # │ Utf8               ┆ Utf8                                  │
    # ╞════════════════════╪═══════════════════════════════════════╡
    # │ Where is the capital of China      ┆ The capital of China is Beijing.                        │
    # │                    ┆                                       │
    # │                    ┆ Beijing is China's political center, cultural center, and international...       │
    # ├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
    # │ What cruciferous plants are there?    ┆ There are many types of cruciferous plants, common ones include:           │
    # │                    ┆ 1. **Vegetable category**                          │
    # │                    ┆  …                                    │
    # ╰────────────────────┴───────────────────────────────────────╯