Description #

Multimodal vector generation processor

Key features #

• Multimodal vectorization support: Supports joint vector generation for images, videos, and text, enabling cross-modal retrieval capabilities.
• Input format adaptation:
  • Natively supports input formats such as base64 encoding, binary data, and URLs for images and videos
  • Automatically handles media format conversion (JPEG/PNG/MP4/AVI and more)
• Model name: doubao-embedding-vision

Input and output specifications #

• Input format:
  • Image/video data/text data: string type, supports base64 encoding/URLs
• Output format:
  • Default mode: vector representation as a float array

LAS provides you with the "Online Experience" capability and a certain amount of free trial quota. No configuration is required to experience the data processing effect of the LAS operator online.

POST https://operator.las.ap-southeast-1.volces.com/api/v1/embeddings/multimodal

API description #

Invoke this operator to perform image-text vectorization.

Request parameters #

Response parameters #

Examples #

Request example #

curl --location "https://operator.las.ap-southeast-1.volces.com/api/v1/embeddings/multimodal" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $LAS_API_KEY" \
--data '{
    "model": "doubao-embedding-vision-250615",
    "encoding_format": "float",
    "input": [
        {
            "type": "image_url",
            "image_url": {
                "url": "https://las-ai-cn-beijing-baseline.tos-cn-beijing.volces.com/operator_cards_serving/public/baseline/doubao_embedding_vision/cat_ip_adapter.jpeg"
            }
        },
        {
            "type": "text",
            "text": "图片里有什么"
        }
    ]
}'

Response example #

{
    "id": "021764041607868c64dcafb5669f61b8aa56083c290255748bdbb",
    "created": "1764041608",
    "model": "doubao-embedding-vision-250615",
    "object": "list",
    "data": {
        "object": "embedding",
        "embedding": [
            0.0002918243408203125,
            -0.0189208984375,
            -0.003814697265625,
            0.058349609375,
            -0.002838134765625,
            0.032958984375,
            "..."
        ]
    },
    "usage": {
        "prompt_tokens": 1335,
        "total_tokens": 1335,
        "prompt_tokens_details": {
            "text_tokens": 23,
            "image_tokens": 1312
        }
    }
}

Operator parameters #

Input #

Output #

Returns the vectorized array processed by the model. Type is list[float]

Parameters #

If a parameter does not have a default value, it is required.

Examples #

The following code demonstrates how to use daft to access the ModelArk image vectorization model for vector computation.

from __future__ import annotations

import os

import daft
from daft import col
from daft.las.functions.ark_llm.doubao_embedding_vision import DoubaoEmbeddingVision
from daft.las.functions.udf import las_udf

if os.getenv("DAFT_RUNNER", "native") == "ray":
    import logging

    import ray

    def configure_logging():
        logging.basicConfig(
            level=logging.INFO,
            format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            datefmt="%Y-%m-%d %H:%M:%S.%s".format(),
        )
        logging.getLogger("tracing.span").setLevel(logging.WARNING)
        logging.getLogger("daft_io.stats").setLevel(logging.WARNING)
        logging.getLogger("DaftStatisticsManager").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaScheduler").setLevel(logging.WARNING)
        logging.getLogger("DaftFlotillaDispatcher").setLevel(logging.WARNING)

    ray.init(dashboard_host="0.0.0.0", runtime_env={"worker_process_setup_hook": configure_logging})
    daft.set_runner_ray()

daft.set_execution_config(min_cpu_per_task=0)

if __name__ == "__main__":
    # You need to configure the environment variable LAS_API_KEY: LAS_API_KEY is obtained by creating it on the LAS service page
    tos_dir_url = os.getenv("TOS_DIR_URL", "las-cn-beijing-public-online.tos-cn-beijing.volces.com")
    samples = {
        "image_path": [
            f"https://{tos_dir_url}/public/shared_image_dataset/cat_ip_adapter.jpeg"
        ],
        "text": ["猫"],
    }

    df = daft.from_pydict(samples)
    # Calculate vectorized data for images and text
    df = df.with_column(
        "embeeding_for_image_text",
        las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "image_format": "jpeg",
            },
        )(col("image_path"), col("text")),
    )

    # Calculate vectorized data for images
    df = df.with_column(
        "embeeding_for_image",
        las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "image_format": "jpeg",
            },
        )(col("image_path")),
    )

    # Calculate vectorized data for text
    df = df.with_column(
        "embeeding_for_text",
        las_udf(
            DoubaoEmbeddingVision,
            construct_args={
                "multimodal_type": "text",
            },
        )(col("text")),
    )
    df.show()

    #  Output (the result of each large model inference may vary)
    # ╭────────────────────────────────┬────────┬────────────────────────────────┬────────────────────────────────┬────────────────────────────────╮
    # │ image_path                     ┆ text   ┆ embeeding_for_image_text       ┆ embeeding_for_image            ┆ embeeding_for_text             │
    # │ ---                            ┆ ---    ┆ ---                            ┆ ---                            ┆ ---                            │
    # │ String                         ┆ String ┆ List[Float32]                  ┆ List[Float32]                  ┆ List[Float32]                  │
    # ╞════════════════════════════════╪════════╪════════════════════════════════╪════════════════════════════════╪════════════════════════════════╡
    # │ https://las-public-data-qa.to… ┆ cat    ┆ [0.010437012, -0.009399414, -… ┆ [0.0076293945, -0.014404297, … ┆ [0.019165039, 0.007598877, -0… │
    # ╰────────────────────────────────┴────────┴────────────────────────────────┴────────────────────────────────┴────────────────────────────────╯