Lake AI Service
Lake AI Service
Operator hub
- Documentation
Lake AI ServiceVideoVideo UnderstandingFine-grained Video Understanding
Copy page
Download PDFVideo Understanding
Fine-grained Video Understanding
Copy pageDownload PDFFine-grained Video Understanding
The Fine-grained Video Understanding operator is designed for general video content. It supports direct understanding and analysis of videos. Whether the content is a short video, film or TV clip, course screen recording, or surveillance footage, users can upload a video and provide a query or task_template. The operator can then describe the video content, answer questions, extract information, and locate related clips with timestamps. It supports hour-level video input, up to 3h and 10G. It also supports structured descriptions with time information based on events and action stages in the video, combining visuals, speech, music, and sound effects.
Operator introduction
Operator ID: las_video_understanding, las_long_video_understand (legacy ID)
Applicable scenarios
Replay
00:00
/
00:00
Live
Play
00:00
Fullscreen
Cssfullscreen
Click and hold to drag
支持一些视频内容理解类的任务：
- Cross-clip Q&A: Integrate information from multiple clips to answer complex questions.
- Highlight generation: Extract key clips, such as emotional peaks and plot twists.
- Character tracking: Continuously lock on to target entities and analyze their behavior.
- Chapter breakdown: Split long videos into chapters by logic.
- Comprehensive audio-video analysis: Combine visual changes, speech content, and sound effect cues to output comprehensive understanding results.
- Key quote extraction: Extract core opinions or memorable lines from videos.
Typical application scenarios are as follows.
Key capabilities
- Global fine-grained understanding: Supports hour-level videos, up to 3h and 10G, and can generate coherent timelines, chapter summaries, and more.
- Joint audio-video understanding: Supports comprehensive analysis by combining video frames, speech, background audio, and sound effects. It is suitable for scenarios that require both visual and auditory cues.
- Event and behavior recognition: Accurately detects key events, character actions, scene changes, and logical relationships.
- Video Q&A: Natural language Q&A based on video content, with quick answer and timestamp localization.
- Efficient summaries and tags: Can generate chapter summaries, topic tags, character relationships, and more, facilitating knowledge management.
- Structured output: Supports outputting timelines, event lists, and more, for secondary processing or knowledge base construction.
- Scenario-based task templates: Provides built-in templates for general video descriptions, fine-grained video descriptions, comprehensive audio-video understanding, embodied interaction object recognition, action timeline annotation, target detection, and more. This reduces Prompt design and parameter tuning costs. Supports specifying the template version in the 模板名@版本号 format. If no version number is provided, v1 is used by default.
Effect display
【预置任务模板：大模型预训练场景示例】
- Example 1: Input video
-
Replay Play00:00 / 00:00 Live 00:00FullscreenCssfullscreenClick and hold to drag
- Application scenario 1: Fine-grained description
- Task template: general_video_captioning, generates a general Chinese description.
- Applicable scenario: Suitable for video-level semantic labels, video-text pairing, and text supervision for pretraining data.
- Example output result
-
- Application scenario 2: Dense video captioning
- Task template: dense_video_captioning, splits by time and generates summaries of actions and scene content.
- Applicable scenario: Emphasizes unified structure and time boundaries. It is suitable for clip-level indexing, temporal supervision construction, automatic slicing, and batch quality inspection. It is not limited to a specific scenario category.
- Example output result:
-
【预置任务模板：具身智能应用示例】
- Application scenario 1 input video: Video after stitching the left and right robotic arm auxiliary-view videos
-
Replay Play00:00 / 01:10 Live 00:00FullscreenCssfullscreenClick and hold to drag
- Scenario 2 input video: Main-view video
-
Replay Play00:00 / 01:10 Live 00:00FullscreenCssfullscreenClick and hold to drag
- First capture clearer item details from the auxiliary-view video and identify the items. Then perform action timeline annotation based on the main-view video.
- Application scenario 1: Interactive object recognition
- Task template: embodied_active_object_detection@v2
- Example output result:
-
- Recommendations after operator processing results:
- Organize the item list in the returned result into brief text. Use it as task_context.prompt_context in subsequent timeline standard requests.
- Each line in the context starts with the generated stable instance ID (obj_001, obj_002 …). This allows the main-view caption to reuse the same stable ID for alignment when the same object is operated on multiple times, for example, "pick up the same yellow long-tail clip for the 2nd time".
- After the item list, append an [action category vocabulary] and [skill filling rules]. Clearly tell the model which standardized atomic action categories are available, and output the corresponding skill label for each action segment.
- Application scenario 2: Action timeline annotation
- Task template: embodied_action_captioning@v2
- Key parameters:
- Use the interactive object recognition result as the task_context.prompt_context input.
- Example output result:
-
【自定义Query任务：影视剧/体育赛事总结】
- Example 2: Input video
-
Replay Play00:00 / 00:00 Live 00:00FullscreenCssfullscreenClick and hold to drag
- Operator configuration example: You can customize the query and flexibly set requirements for video content extraction, summarization, and more
- "data": {"video_url": "tos://las-doctest/las-ai/input/video/sports/体育赛事.mov","query": "请生成一份详细的 比赛文字直播日志 。按时间顺序列出所有得分事件，格式为： [时间] 球队 - 事件 - 当前比分 。请确保不遗漏任何一个进球。" , "fps": 5,"model_name": "doubao-seed-2-1-pro-260628"}
- Example result
- [13.0-16.0] 蓝队 - 12号队友接7号传球后投篮，球颠进篮筐得分 - 2:0[26.8-28.4] 红队 - 14号接3号长传快攻上篮得分 - 2:2[42.2-43.4] 蓝队 - 33号在内线接到队友传球后扣篮得分 - 4:2[59.0-60.4] 红队 - 8号接队友传球后三分出手命中 - 4:5[74.0-75.2] 蓝队 - 12号球员接队友长传快攻上篮得分 - 6:5
Supported regions
- Beijing: cn-beijing
- Shanghai: cn-shanghai
- Guangzhou: cn-guangzhou
Operator performance
Subitem | Description of performance impact |
Maximum Submit RPM | 2000 |
Maximum Poll RPM | 4000 |
Maximum concurrency | 100 |
Time consumption |
|
Input and output requirements
Input requirements
Subdivision | Detailed description |
Supported input data modalities |
|
Input format: Video |
|
Input path requirements | Provide input data to the operator through the request parameter video_url. Currently, public URLs and TOS paths are supported.
|
Output requirements
Subdivision | Detailed description |
Output data modality |
|
Output path: API return | After operator processing is complete, the result is returned directly through the response parameters of the operator Poll API. |
Billing description
- Billing standard
- SubitemBilling standard descriptionBillable itemsIncludes 4 billable items: model input, model output, cache storage, and cache hit. The payable fee is the sum of all billable items.Billing typePay-as-you-go. Unit: 元/百万 Tokens. Bills are generated hourly based on actual billable usage.Unit priceThe unit price of each billable item follows tiered pricing based on the model input length when invoking the model. Unit prices differ across different input length ranges.
- Billing details
- Billing formula: 总费用 = 模型输入费用 + 模型输出费用 + 缓存存储费用 + 缓存命中费用
-
- Billing estimation
- Based on the billing details table above, you can estimate how many Tokens the input video is roughly equivalent to. Combine this with the operator request code to estimate the overall model input length.
- Note: This estimation result cannot be used as accurate input length data. It is for reference only. During actual task processing, other inputs such as the system Prompt are also included, but these inputs are usually not very long.
Note
Currently, when using the doubao-seed-2.1 model, the generated billing statement still displays doubao-seed-2.0-related impact factors. You can use details such as the unit price in the bill to identify the model type actually used for the current bill.
Notes and prerequisites
Subitem | Notes and prerequisites |
Enable LAS |
|
Fees | Before invoking the operator, you must first understand the model invocation fees when using the operator. For details, see Large model invocation billing. |
Authentication (API Key) | Before invoking the operator, you must generate an API Key for operator invocation. We recommend configuring the API Key as an environment variable to call the operator more securely. For details, see Obtain an API Key and configure it. |
BaseURL | Before invoking the operator, you must learn about the BaseURL for operator invocation based on the region where your current LAS service is located. This is used to configure operator invocation path parameter values. For details, see Obtain the Base URL. The Examples below are for reference only. Replace them with the path value for your region during actual invocation. |
API invocation
API invocation description
- The API of this operator is an asynchronous task API. You must submit a task through the Submit API to obtain the task task_id. Then use the Poll API to obtain the running status of the corresponding task (task_id) and the operator processing result.
- Note that after the task is submitted, the task_id of the task is valid for 3 days. After 3 days, you cannot use this task_id to obtain the operator processing result.
Submit
API description
Invoke the las_video_understanding operator for Fine-grained Video Understanding. The minimum available request can pass in video_url and query, or pass in video_url and task_template. Other fields can be configured as needed based on task complexity and analysis requirements.
Request parameters
operator_id string required Example value: las_video_understanding
Operator ID: las_video_understanding or las_long_video_understand (legacy)
operator_version string required Example value: v1
Operator version
data long_video_understand_info required
Operator parameters
video_url string required Example value: https://example.com/video.mp4
Accessible address of the video to be analyzed. Supports http/https, Volcengine TOS (tos://bucket/key), and more. Ensure that the server can fetch the resource. We recommend using a stable file link that can be downloaded directly. Maximum supported duration: 3 hours. Maximum supported size: 10 GB. Supported formats: mp4, mov, avi, mkv, flv, webm. For tasks that need to output many event and action details, we recommend processing videos within 30 minutes first to obtain a more stable result structure and time boundaries.
query string Example value: Summarize the key goals in the video in chronological order, and mark player numbers and timestamps
Analysis objective or question description for the video. Required if task_template is not passed in. When both query and task_template are passed in, the explicit query overrides the default query of the template. Other explicit parameters also override the corresponding template default values. We recommend making the output objective and result organization as clear as possible. The more specific the question, the more stable the result usually is.
task_template string Example value: general_video_captioning@v1
Task template. After it is passed in, the default configuration required by the task can be automatically filled in. When using a template, you usually only need to pass in video_url and task_template. Supports specifying the template version through template_name@version_number. If no version number is provided, v1 is used by default. For example, general_video_captioning is equivalent to general_video_captioning@v1.
Supports:
- general_video_captioning: Generates continuous, detailed, unstructured video-level descriptions.
- dense_video_captioning: Generates structured JSON descriptions containing time information for events, actions, and scenes. It is suitable for tasks that need to output many event and action details. We recommend using it first for videos within 30 minutes.
- video_audio_understanding: Combines video frames, speech, music, and sound effects to generate comprehensive audio-video understanding results.
- omni_video_audio_captioning: Comprehensively understands visuals and sounds, and generates structured descriptions with time information based on events and action stages in the video.
- speech_recognition: Transcribes speech in audio or video and distinguishes speakers.
- audio_understanding: Summarizes and analyzes pure audio content.
- embodied_active_object_detection: Identifies main interactive objects for embodied operation scenarios. Supports v2.
- embodied_action_captioning: Generates action timeline annotations for embodied operation scenarios. Supports v2.
- embodied_action_captioning_prior: Generates action timeline annotations with candidate action priors. Supports v2.
- object_detection: Detects specified targets and returns bbox.
task_context task_context Example value: {"prompt_context":"主要交互物品：黄色图钉、浅绿色便签。"}
Structured context for the task template. Supports prompt_context, target_object, and audio_url. See the table below for applicable templates.
prompt_context string Example value: Main interactive items: yellow pushpin, light green sticky note
Prior information for task context. It currently takes effect when task_template is embodied_action_captioning or embodied_action_captioning@v2.
target_object string Example value: pink_bear (pink plush bear)
Target detection category and description. It currently takes effect when task_template is object_detection. We recommend using the format "English category key (Chinese/natural language description)". The category in the output uses the category key before the parentheses.
audio_url string Example value: tos://bucket/audio.wav
Independent audio address. When used for video_audio_understanding or omni_video_audio_captioning, it takes precedence over the built-in audio track of the video.
start float Example value: 30.0
Video analysis start time, in seconds. The range is [0, video duration], and start < end.
end float Example value: 120.0
Video analysis end time, in seconds. The range is [0, video duration], and start < end.
fps float Example value: 1.0
Video frame extraction frame rate. Increasing fps improves analysis accuracy, but also increases processing latency, Token consumption, and cost. The default value is 1.0. The value range is [1/60, 10]. The default value is sufficient for general scenarios. When the task requires recognizing fast actions or finer-grained events, you can try increasing the value appropriately.
media_resolution string Example value: medium
Video processing resolution tier. It affects visual detail retention and processing latency. The default value is medium. Supported values: low, medium, high, xhigh. The default value is sufficient for general scenarios. When the task has high requirements for recognizing visual details such as subtitles, small targets, and jersey numbers, you can try a higher tier.
model_name string Example value: doubao-seed-2-0-lite-260215
Specifies the video understanding model. The default value is doubao-seed-2-0-lite-260215. If you have no special requirements, we recommend using the default value.
Supports:
- Seed 2.0 Mini: doubao-seed-2-0-mini-260215, doubao-seed-2-0-mini-260428.
- Seed 2.0 Lite: doubao-seed-2-0-lite-260215, doubao-seed-2-0-lite-260428.
- Seed 2.0 Pro: doubao-seed-2-0-pro-260215.
- Seed 2.1 Pro: doubao-seed-2-1-pro-260628, doubao-seed-2-1-pro-260915.
- Seed 2.1 Turbo: doubao-seed-2-1-turbo-260628.
reasoning_effort string Example value: minimal
Controls model reasoning intensity. The default value is minimal, which means deep thinking is disabled. Common values: minimal, low, medium, high. The default value is sufficient for general scenarios. When the task involves cross-clip induction, complex causal relationships, or multi-character analysis, you can try using a higher tier.
clip_context string Example value: medium
Controls the context range of a single video segment. The default value is medium. Supported values: medium, long. The default value is sufficient for general scenarios. When the task depends more on context continuity, you can try the higher tier long.
use_responses_api boolean Example value: false
Whether to enable Responses API capabilities. After it is enabled, video content can be cached. The cache is valid for 1 hour and is suitable for repeated analysis requests on the same video.
previous_response_ids list of string Example value: ["cache_resp_test"]
After passing in the list of response identifiers returned from the previous response, cached video content can be reused. This takes effect only when use_responses_api=true.
Response parameters
metadata object
Request metadata
task_id string Example value: task-xxx
Task ID in asynchronous mode, used for job status queries.
task_status string Example value: PENDING
Task status in asynchronous mode.
- PENDING, task submitted and queued
- RUNNING, running
- COMPLETED, completed
- FAILED, failed
- TIMEOUT, timed out
business_code string Example value: 0
Business code. Usually returns 0 when the submission is successful.
error_msg string
If an exception occurs, detailed exception information is returned.
Recommended configuration
Scenario | Recommended parameters |
General summary, basic Q&A | Use the default parameters |
Sports events, action detail analysis | You can appropriately increase fps. When the recognition requirements for details such as jersey numbers and small targets are high, try media_resolution=high |
Meeting minutes, course breakdown, plot sorting | You can try clip_context=long. When the task involves complex induction, you can further try increasing reasoning_effort |
Repeated analysis of the same video, reuse cached video | use_responses_api=true, and pass in previous_response_ids. The video cache is valid for 1 hour |
Example
Request example
curl --location "https://operator.las.cn-beijing.volces.com/api/v1/submit" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $LAS_API_KEY" \
--data '{
"operator_id": "las_video_understanding",
"operator_version": "v1",
"data": {
"video_url": "https://example.com/video.mp4",
"task_template": "general_video_captioning@v1"
}
}'
Response example
{
"metadata": {
"task_id": "task-20251125163544-abc123",
"task_status": "PENDING",
"business_code": "0",
"error_msg": ""
}
}
Poll
API description
Query the execution status and result of a video understanding task.
Request parameters
Name | Type | Required | Example value | Description |
operator_id | string | Yes | las_video_understanding | Operator ID |
operator_version | string | Yes | v1 | Operator version |
task_id | string | Yes | task-xxx | Task ID |
Response parameters
metadata metadata
Request metadata
task_id string Example value: task-xxx
Task ID in asynchronous mode.
task_status string Example value: COMPLETED
Task status in asynchronous mode.
Asynchronous tasks usually require both task_status=COMPLETED and business_code=0 to determine that the task is truly completed.
- PENDING, task submitted and queued
- RUNNING, running
- COMPLETED, completed
- FAILED, failed
- TIMEOUT, timed out
submit_time string Example value: 2026-07-21T16:04:06+08:00
Task submission time in ISO 8601 timestamp format.
end_time string Example value: 2026-07-21T16:04:06+08:00
Task end time in ISO 8601 timestamp format.
business_code string Example value: 0
Business code. Usually returns 0 when business processing succeeds.
error_msg string
If an exception occurs, detailed exception information is returned.
data long_video_understand_result
Returned data
token_usages list of token_usage_info Example value: [{"model_name":"doubao-seed-2-0-lite-260215","token_usage":{"prompt_tokens":4453444,"completion_tokens":10929}}]
token usage
model_name string Example value: "doubao-seed-2-0-lite-260215"
Model name
token_usage object Example value: {"prompt_tokens": 4453444, "completion_tokens": 10929, "cached_tokens": 10032, "cached_store_tokens": 10032, "total_tokens": 4464373}
Token usage
video_duration float Example value: 847.296
Original video duration
resolution string Example value: 1920x1080
Original video resolution
final_summary string Example value: The video as a whole describes...
Final summary
response_ids list of string Example value: ["cache_resp_test"]
List of response identifiers returned after the Responses API is enabled. They can be passed in through previous_response_ids in subsequent Submit requests to reuse cached video content
Example
Request example
curl --location "https://operator.las.cn-beijing.volces.com/api/v1/poll" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer $LAS_API_KEY" \
--data '{
"operator_id": "las_video_understanding",
"operator_version": "v1",
"task_id": "task-20251125163544-abc123"
}'
Response example
{
"metadata": {
"task_id": "task-20251125163544-abc123",
"task_status": "COMPLETED",
"submit_time": "2026-07-21T16:04:06+08:00",
"end_time": "2026-07-21T16:04:06+08:00",
"business_code": "0",
"error_msg": ""
},
"data": {
"token_usages": [
{
"model_name": "doubao-seed-2-0-lite-260215",
"token_usage": {
"prompt_tokens": 4453444,
"completion_tokens": 10929,
"cached_tokens": 2612,
"cached_store_tokens": 2612,
"total_tokens": 4464373
}
}
],
"video_duration": 847.296,
"resolution": "1920x1080",
"final_summary": "视频整体讲述了...",
"response_ids": [
"cache_resp_test"
]
}
}
Error codes
HttpCode | Error code | Error message | Description |
400 | Model.InvalidName | The model name is invalid. | The model name is invalid |
401 | Authorization.Missing | Missing Authorization. | Authentication is missing |
401 | ApiKey.Invalid | The api key is invalid. | The API is invalid |
Last updated: 2026.09.24 11:25:40
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
