Lake AI Service
Lake AI Service
Operator hub
- Documentation
Lake AI ServiceEmbodied IntelligenceSemantic AnnotationEmbodied Semantic Annotation
Copy page
Download PDFSemantic Annotation
Embodied Semantic Annotation
Copy pageDownload PDFEmbodied Semantic Annotation
The Embodied Semantic Annotation operator is designed for embodied data collection scenarios such as real robots, UMI, and human data. It identifies interactive objects during operations, annotates action descriptions, start and end times, and skill labels, and supports temporal annotation with candidate task descriptions.
Operator introduction
Operator ID: las_embodied_caption
Applicable scenarios
- Interactive object recognition in real-robot, UMI, and human operation videos.
- Generation of task descriptions, action segments, and skill labels for first-person operation videos.
- Locating actual operations and time intervals in videos based on candidate task descriptions.
Key features
- Interactive object recognition: Identifies main interactive objects and distinguishable instances, and annotates interaction time intervals.
- Action temporal annotation: Splits continuous operations and generates task descriptions, action start and end times, action descriptions, and skill labels.
- Candidate task-assisted annotation: Locates actual operations and time intervals in videos based on candidate task descriptions.
- Frame extraction and resolution configuration: Supports configuring the frame extraction frame rate and video processing resolution level.
- Segment interval analysis: Supports specifying the video analysis interval through start and end.
- Video content caching: After the Responses API is enabled, cached content from the same video can be reused.
Supported tasks
LAS provides preset task templates for typical Embodied Semantic Annotation scenarios. You can directly use templates to quickly call the operator for corresponding scenarios. The supported preset task templates are as follows.
Note
When you do not use a preset task template, you can also provide prompts for custom semantic annotation tasks through the request parameter query. If both a custom task prompt and a preset task template are provided, the custom task prompt overrides the default template configuration.
Task template | Description |
embodied_active_object_detection@v2 | Interactive object recognition, applicable to horizontally stitched left-right dual-wrist videos |
embodied_active_object_detection@v3 | Interactive object recognition, supporting both horizontally stitched left-right dual-wrist videos and single-channel head-mounted videos |
embodied_action_captioning@v2 | First-person main-view action annotation, outputting the task description and action segments with skill labels |
embodied_action_captioning_prior@v2 | Performs temporal annotation with candidate task descriptions (task_context.prompt_context) and outputs actions and time intervals. Note When using this template, you must provide the JSON string serialized from the candidate task description array through task_context.prompt_context. |
Effect demonstration
- Input task template: embodied_action_captioning@v2
- Input main-view video
-
Replay Play00:00 / 00:00 Live 00:00FullscreenCssfullscreenClick and hold to drag
- Output result excerpt
-
Operator performance
Item | Performance impact description |
Maximum Submit RPM | 600 |
Maximum Poll RPM | 600 |
Maximum concurrency | 10 |
Time consumption | Increasing the frame extraction frame rate increases processing latency and Token consumption. |
Input and output requirements
Input requirements
Output requirements
Item | Detailed requirements |
Supported data modalities |
Returns structured results such as interactive objects, task descriptions, action segments, start and end times, and skill labels based on the task template |
Output path: API response | After the operator finishes processing, the result is returned directly through the response parameters of the operator Poll API. |
Billing description
When calling the LAS Embodied Semantic Annotation operator, the ModelArk model under your account is used directly. LAS does not charge additional operator processing fees. Model calls use the pay-as-you-go postpaid Token billing method of ModelArk online inference (standard). The currently supported models and list prices are as follows.
Note
- Input tokens generated by the current operator's text and video visual content are billed at the "Input (non-audio)" unit price. The audio prices in the table are the complete pricing information for the corresponding ModelArk model.
- The Seed 2.0 series is billed by segment based on the input length of a single request. Both input and output tokens use the unit price of the corresponding interval.
- When video content caching is enabled, cache storage and cache hit tokens are billed at the corresponding unit prices in the table.
- The preceding table shows the list prices for ModelArk online inference (standard). The actual purchasable product specifications and fees are subject to the actual order and bill. For the latest prices, see ModelArk model pricing.
Notes and prerequisites
Rest API calls
Submit
API description
Call the las_embodied_caption operator to submit an Embodied Semantic Annotation task. After a successful submission, a task ID is returned. You can query the task status and result through Poll.
Request parameters
operator_id string required Example value: las_embodied_caption
Operator ID, fixed to las_embodied_caption.
operator_version string required Example value: v1
Operator version, fixed to v1. It is independent of the task template version.
data las_embodied_caption_info required
Embodied Semantic Annotation parameters.
video_url string required Example value: https://example.com/main.mp4
Accessible address of the video to be analyzed. HTTP/HTTPS and Volcengine TOS are supported. Up to 3 hours and 10 GB are supported. Supported formats are mp4, mov, avi, mkv, flv, and webm.
ark_api_key string required Example value: ark-xxxx
ModelArk API Key.
model_name string required Example value: doubao-seed-2-1-pro-260628
Model name. Supports doubao-seed-2-0-mini-260215, doubao-seed-2-0-mini-260428, doubao-seed-2-0-lite-260215, doubao-seed-2-0-lite-260428, doubao-seed-2-0-pro-260215, doubao-seed-2-1-pro-260628, doubao-seed-2-1-pro-260915, and doubao-seed-2-1-turbo-260628.
ark_endpoint_id string Optional Example value: ep-xxxx
ModelArk inference endpoint. If it is not empty, it must correspond to the same model minor version as model_name. If omitted, the model default endpoint is used.
query string Optional Example value: Annotate the actions of picking up and placing items, and their start and end times
Custom prompt. Required when task_template is not passed. If explicitly passed, it overrides the template default prompt.
task_template string Optional Example value: embodied_action_captioning@v2
Task template in the format "template name@version number". If the version number is omitted, v1 is used by default. Supports templates for interactive object recognition, action temporal annotation, and candidate task-assisted annotation.
task_context task_context Optional
Task template context.
prompt_context string Optional Example value: Main interactive objects: white cylindrical trash can, plastic bottle
You can provide object information or a skill vocabulary during action annotation. When using embodied_action_captioning_prior@v2, you must pass the JSON string serialized from the candidate description string array, for example ["调整垃圾桶位置","拿起塑料瓶"].
start float Optional Example value: 0.0
Analysis start time, in seconds. The default value is 0. The value range is [0, 视频时长], and the value must be less than end.
end float Optional Example value: 106.9
Analysis end time, in seconds. If omitted, the video is analyzed until the end. The value must be greater than start.
fps float Optional Example value: 1.0
Frame extraction frame rate. The range is 1/60 to 10. The default is 1.0 when no template is used. Templates can provide different defaults. Explicit parameters take precedence.
media_resolution string Optional Example value: medium
Video processing resolution level. Supports low, medium, high, and xhigh. The default is medium when no template is used.
reasoning_effort string Optional Example value: minimal
Model reasoning intensity. Common values include minimal, low, medium, and high. The default is minimal when no template is used. Available levels depend on the model.
clip_context string Optional Example value: medium
Single-segment context range. Supports medium and long. The default is medium.
use_responses_api boolean Optional Example value: false
Whether to enable the Responses API. The default is false. After it is enabled, video content can be cached. Subsequent requests can reuse the cache through previous_response_ids. The default cache validity period is 1 hour.
previous_response_ids list of string Optional Example value: ["cache_resp_test"]
Pass the previous response ID when reusing cached video content. This takes effect only when use_responses_api=true.
Response parameters
metadata metadata
Request metadata.
task_id string Example value: task-xxx
Asynchronous task ID, used for Poll queries.
task_status string Example value: PENDING
Task status: PENDING, RUNNING, COMPLETED, FAILED, TIMEOUT.
business_code string Example value: 0
Business code. It is usually 0 when submission succeeds.
error_msg string
Exception details.
Request example
Response example
Error codes
HttpStatusCode | Error code | Error message | Description |
400 | Model.InvalidName | The model name is invalid. | Invalid model name |
401 | Authorization.Missing | Missing Authorization. | Authentication information is missing |
401 | ApiKey.InValid | The api key is invalid. | Invalid API Key |
Poll
API description
Query the status and result of an Embodied Semantic Annotation task based on the task ID. A task is successful only when both task_status=COMPLETED and business_code=0 are met.
Request parameters
Response parameters
metadata metadata
Request metadata.
task_id string Example value: task-xxx
Asynchronous task ID.
task_status string Example value: COMPLETED
Task status: PENDING, RUNNING, COMPLETED, FAILED, TIMEOUT.
submit_time string Example value: 2026-10-09T16:00:00+08:00
Task submission time, in ISO 8601 format.
end_time string Example value: 2026-10-09T16:01:00+08:00
Task end time, in ISO 8601 format.
business_code string Example value: 0
Business code. It is usually 0 when processing succeeds.
error_msg string
Exception details.
data las_embodied_caption_result
Task result.
token_usages list of token_usage_info
Token usage of each model.
model_name string Example value: doubao-seed-2-1-pro-260628
Model name.
token_usage token_usage
Token usage.
prompt_tokens integer Example value: 10000
Number of input tokens, including text, audio, and all other types.
completion_tokens integer Example value: 1000
Number of tokens generated by the model.
audio_tokens integer Example value: 0
Number of audio tokens in the input.
cached_tokens integer Example value: 0
Number of hit tokens.
cached_store_tokens integer Example value: 0
Number of tokens written to cache.
cached_audio_tokens integer Example value: 0
Number of hit audio tokens.
total_tokens integer Example value: 11000
Total number of tokens, which is prompt_tokens + completion_tokens.
video_duration float Example value: 106.9
Video duration, in seconds.
resolution string Example value: 854x480
Original video resolution, not the media_resolution level.
final_summary string
JSON string of the annotation result. Interactive object recognition returns objects. Action temporal annotation returns task_description and segments. Candidate task-assisted annotation returns segments. The time unit is seconds.
response_ids list of string Example value: ["cache_resp_test"]
Response IDs returned after the Responses API is enabled. They can be used to reuse the cache through previous_response_ids in subsequent Submit requests.
Request example
Response example
The following annotation content and Token values are for demonstration only.
Error codes
HttpStatusCode | Error code | Error message | Description |
400 | Task.NotFound | The task is not found. | Task does not exist |
401 | Authorization.Missing | Missing Authorization. | Authentication information is missing |
401 | ApiKey.InValid | The api key is invalid. | Invalid API Key |
Last updated: 2026.10.13 15:00:17
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
