Description #

The video content understanding operator supports using the Doubao model to understand video files, including parsing video content and generating natural language descriptions.

Limitations #

• The storage space occupied by video files must be less than 1 GiB.
• Currently, audio information in video files is not supported for understanding.
• Video links must be accessible via the public network or Volcano Engine intranet.

Key features #

• Compress the video to within 50 MB, then use the Doubao model for video understanding.
• Supported video formats: mp4, wmv, webm, mkv, m4v, flv, avi, mov. Due to the wide variety of video file format variants, not all files can be guaranteed to be recognized. Please verify through testing that files can be properly recognized.

Performance description #

• In theory, the larger the video, the longer the processing time required.
• Actual processing time is affected by factors such as video quality and network environment. It is recommended to use in the Volcano Engine intranet environment.

• Billing standards

  Sub-item	Billing standard description
  Billing item	There are two billing items: Inference service (input) and Inference service (output). The total fee payable is the sum of all billing items.
  Billing type	Pay-as-you-go, unit: CNY per million tokens, billed hourly based on actual usage.
  Unit price	The unit price depends on the type of video link you input. The unit prices for public network links and VolcanoEngineTOS path links are different. For different input video link scenarios, the unit price of each billing item is tiered based on the model input length at the time of model invocation. The unit price varies depending on the input length tier.

  warning

  • When the input video link is a TOS path, the unit price is usually lower.

• Billing details
  Billing formula: Total cost = Total price of billing item 1 + Total price of billing item 2 = (Unit price of billing item 1 * usage) + (Unit price of billing item 2 * usage)

  Model type	Sub-scenario (Input video link type)	Condition (thousand tokens)	Unit price: Billing item 1 Inference service (input)	Unit price: Billing item 2 Inference service (output)
  doubao-seed-1.6 doubao-seed-1.8	Video link: Public network link	Input length [0, 128]	4.7 CNY per million tokens	20 CNY per million tokens
  		Input length (128, 256]	5.9 CNY per million tokens	30 CNY per million tokens
  	Video link: TOS path	Input length [0, 128]	2.6 CNY per million tokens	20 CNY per million tokens
  		Input length (128, 256]	3.8 CNY per million tokens	30 CNY per million tokens
  doubao-seed-2.0-pro	Video link: Public network link	Input length [0, 32]	8.6 CNY per million tokens	28.9 CNY per million tokens
  		Input length (32, 128]	11.3 CNY per million tokens	42.4 CNY per million tokens
  		Input length (128, 256]	19.3 CNY per million tokens	82.8 CNY per million tokens
  	Video link: TOS path	Input length [0, 32]	6.6 CNY per million tokens	27 CNY per million tokens
  		Input length (32, 128]	9.3 CNY per million Tokens	40.4 CNY per million Tokens
  		Input length (128, 256]	17.4 CNY per million Tokens	80.8 CNY per million Tokens

• Billing estimation

  • Based on the above billing details table, you can estimate approximately how many tokens the input video contains. Combined with the operator request code, you can estimate the overall model input length in tokens.
  • Caution: This estimation result cannot be used as the exact input length data and is for reference only. When processing tasks, there will also be system prompts and other inputs, but these other inputs are generally short.

LAS provides you with the capability for "online experience" and offers a certain amount of free usage quota. No configuration is required to experience the data processing effect of the LAS operator online.

Interface description #

Invoke the las_vlm_video operator for video understanding via the Chat interface.

Request parameters #

Response parameters #

Examples #

Request example #

curl --location "https://operator.las.ap-southeast-1.volces.com/api/v1/chat/completions" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $LAS_API_KEY" \
--data '{
    "operator_id": "las_vlm_video",
    "operator_version": "v1",
    "data": {
       "las_extra_params": {
             "operator_id": "las_vlm_video",
             "operator_version": "v1"
             },
        "model_name":"doubao-seed-1-8-251228",
        "messages": [
            {
                "content": [
                    {
                        "video_url": {
                            "url": "https://las-ai-cn-beijing-baseline.tos-cn-beijing.volces.com/operator_cards_serving/public/baseline/las_vlm_video/v1/eating_56.mp4"
                        },
                        "type": "video_url"
                    },
                    {
                        "text": "",
                        "type": "text"
                    }
                ],

                "role": "user"
            }
        ]
    }
}'

Response example #

{
    "metadata": {
        "task_status": "COMPLETED",
        "business_code": "0",
        "error_msg": "",
        "request_id": "7517a8d9fa003a570777e2a390977f73"
    },
    "data": {
        "vlm_result": {
            "id": "0217610362616819fc4dc47d27ebf7239e22958e08dac0b3c4dd9",
            "object": "chat.completion",
            "created": 1761036294,
            "model": "doubao-seed-1.8",
            "serviceTier": "default",
            "choices": [
                {
                    "index": 0,
                    "message": {
                        "role": "assistant",
                        "content": "The video unfolds with a brightly colored two-layer cake as the core, and the overall atmosphere is fantastical and with a sense of humor. Here is the specific analysis:\ n\ n\ n ###** 1. Scene and object details **\ n- ** Main cake **: The bottom layer is dark red, and the surface is covered with smooth red frosting; the top layer is light yellow, decorated with red wavy stripes, and the top is covered with thick red frosting. The frosting is stacked with beige cream flowers, and the top of the cream is inserted with a red cherry. The bottom edge of the cake is decorated with yellow stars, and the overall shape is exquisite and inviting.\ n- ** Background Environment **: The background is a blue carved window. The window frame lines are smooth, and the dark purple light is revealed, creating a mysterious atmosphere. The visible part on the left is a red bookshelf, with faint outlines of books; on the right hangs a painting of a hand holding a blue magic wand with pink stars twinkling at the end, suggesting that the scene may be related to "magic".\ n\ n ###** 2. Core characters and dynamics **\ n The key twist in the video is the green creature that suddenly appears at the top of the cake:\ n- ** Initial state **: The creature "emerges" from the cream of the top cake, initially showing only its head, eyes closed, mouth raised, revealing two sharp teeth, with a sly complacent expression (as if snickering or planning something). \ n- ** Expression Change **: Then the creature suddenly opens its eyes, the eyeballs instantly become big and round (highlighting surprise or excitement), the mouth opens, revealing more fangs, and the smile is exaggerated, changing from "sly" to "ecstasy" or "surprise", as if it has just finished a prank or discovered something interesting.\ n- ** Interactive Details **: When the creature appears, the cream flower is slightly pushed up but not scattered, and the red cherry at the top remains in place, suggesting that the creature is "drilled" from the inside of the cream, with light and fantastical movements.\ n\ n ###** 3. Ambience and underlying narrative **\ n- ** Fantastic tone **: The magic wand hanging on the right echoes the sudden appearance of the creature, suggesting that there may be a magical element in the scene - the cake may be enchanted, or the creature itself has magical powers that can be hidden in the food.\ n- ** Humor and mischievousness **: The expression of the green creature ranges from "bad laugh" to "exaggerated excitement", and the character image is mischievous and intelligent, contrasting with the delicate cake (the "sweetness" of the cake and the "cunning" of the creature), creating a relaxed and humorous atmosphere, possibly in the moment of a mischievous character's "surprise appearance". The video builds a short and vivid scene through vibrant color contrasts (red, yellow cakes and green creatures), fantastical elements (magic paintings, creatures that suddenly appear), and quick changes in the characters' expressions (sly → ecstatic): a mischievous green creature hidden in a cake suddenly appears in a magical environment, with the excitement of a successful mischief, and the overall delivery is relaxed and fantastical fun.",
                        "reasoning_content": "\ nThe user now needs me to analyze this video. First, I need to look carefully at the video content and describe all the elements I see. The video starts with a two-layer cake with a red bottom layer and a yellow top layer with red wavy stripes, topped with cream and cherries. The background is a blue window with a bookshelf on the left and a painting on the right with a magic wand and stars.\ n\ nThen, a green creature suddenly appears from the cream on the top of the cake. At first, it closes its eyes and shows a bad smile. Then the eyes open and become very large. The mouth opens and shows teeth. The expression changes from complacency to surprise or excitement? Need to pay attention to the process of biological change. \ n\ nNext, to analyze the atmosphere of the scene, the blue windows and bookshelves may be in a room, and the magic wand painting on the right may suggest that there are magic elements. The cake looks very tempting, and the green creature appears from the cake. It may appear suddenly, with a feeling of surprise or mischief.\ n\ nThe expression of the creature changes: first close the eyes and smile, then open the eyes, the eyes become larger, the mouth is open, and the teeth are exposed. It may be that you find yourself on the cake, or the reaction to the surrounding environment. You need to pay attention to details, such as the state of the cream, whether the cream changes when the creature appears, such as being pushed up.\ n\ nAlso consider the chronological order. The video is very short, about 3 seconds. The action ranges from static cake to creature appearance to expression change. The possible intention is to show a character who has suddenly appeared, with a humorous or fantastical color, because there is a background of magic painting, which may be related to magic, and the creature may have appeared through magic?\ n\ nSummary elements: cake (red bottom layer, yellow top layer, red frosting, cream, cherry), background (blue window, bookshelf, magic painting), green creature (expression change: closed eyes smirk → open eyes surprise/excitement), action (creature emerges from the top of the cake).\ n\ nPossible scene setting: a room with magic, the cake may be enchanted, or the creature is hidden in the cake and suddenly appears. The green of the creature contrasts with the bright color of the cake, highlighting the character. \ n\ nNeed to confirm if there are other details, such as the yellow decoration on the bottom of the cake, possibly stars or petals? There is a yellow shape on the bottom of the cake at the beginning of the video, which may be decoration. When the creature appears, the cherry on the top is still there, and the cream is jacked up, but the cherry is not dropped, indicating that the creature is drilled out of the cream.\ n\ nKey to the change of expression: When the eyes are closed, the corners of the mouth are raised, there is a smug smile, and it may be planning something; After opening the eyes, the eyes are round, the mouth is open, and the teeth are exposed. It may be surprised by their successful appearance, or the reaction after seeing the surrounding environment, they appear more lively.\ n\ nOverall atmosphere: Fantasy, humorous, with a little naughty feeling, because the creature's expression ranges from bad laugh to excitement, and may be a naugh"
                    },
                    "finish_reason": "stop"
                }
            ],
            "usage": {
                "prompt_tokens": 4903,
                "completion_tokens": 1133,
                "total_tokens": 6036,
                "prompt_tokens_details": {
                    "cached_tokens": 0
                },
                "completion_tokens_details": {
                    "reasoning_tokens": 554
                }
            }
        },
        "compress_result": [
            {
                "video_url": "https://las-ai-cn-beijing-baseline.tos-cn-beijing.volces.com/operator_cards_serving/public/qa/las_vlm_video/v1/eating_56.mp4",
                "compression_details": {
                    "original_info": {
                        "original_path": "https://las-ai-cn-beijing-baseline.tos-cn-beijing.volces.com/operator_cards_serving/public/qa/las_vlm_video/v1/eating_56.mp4",
                        "original_size_mb": 0.35,
                        "original_width": 640,
                        "original_height": 360,
                        "original_fps": 25,
                        "original_duration": 2.9,
                        "original_format": "mp4"
                    },
                    "compression_process": {
                        "step_0_format_conversion": {
                            "executed": false
                        },
                        "step_1_fps_reduction": {
                            "executed": false
                        },
                        "step_2_resolution_reduction": {
                            "executed": false
                        },
                        "step_3_bitrate_compression": {
                            "executed": false
                        }
                    },
                    "final_result": {
                        "final_width": 640,
                        "final_height": 360,
                        "final_fps": 25,
                        "final_duration": 2.9,
                        "final_size_mb": 0.35,
                        "final_format": "mp4"
                    },
                    "timing_info": {
                        "total_processing_time_seconds": 0.23
                    }
                }
            }
        ]
    }
}