Docs

Lake AI Service

Operator Overview

Supported operator list: online operators

Video

Video generation

Operator name	Usage	Operator introduction
Enhanced/basic video generation video_generation (Doubao-Seedance-1.x series)	Online	LAS Enhanced Video Generation/Basical Video Generation service is powered by Seedance series models. It focuses on the pre- and post-processing pipelines for video generation. It integrates capabilities such as video format conversion, video segmentation, Video Inpaint, subtitle generation, Audio and Video Merge, and Video Super Resolution. It extends the capability boundary of video generation models, improves generated video quality and duration, reduces token consumption costs, and helps customers improve production efficiency.
Video Edit Enhanced	Online	The Video Edit Enhanced online service replaces video content based on an input video and reference images. It supports scene replacement and object replacement. It preserves the subject actions, shot rhythm, and temporal continuity in the original video as much as possible, and outputs the replaced video result. Key features: Supports scene_replace scene replacement. It keeps the characters, products, actions, and shot structure in the original video, and replaces the background or environment style. Supports object_replace object replacement. It replaces the target object in the video with a specified style or product based on a reference image. Supports adding natural-language editing constraints through user_prompt, such as preserving actions, removing watermarks, avoiding subtitles, or controlling style details. Supports a limited number of automatic evaluations and retries, which helps obtain more stable replacement results on complex materials. Supports returning the final video result and the result.json diagnostic file, so callers can retain the generated result and process information.
Video Edit Enhanced 2.0	Online	The Video Edit Enhanced 2.0 online service provides video editing templates for business scenarios. It completes pass-through generation, general keyframe editing, face swapping, full character replacement, general subject replacement, or global style transfer while preserving the original video actions, shots, composition, and timeline. It provides higher-quality video editing capabilities for targeted business scenarios. Key features: Template-based routing: The business only declares the editing goal. The server-side template selects the VLM, keyframe repainting, structure video, and Seedance prompt assembly process. General keyframe editing: The VLM understands the video and user intent, selects keyframes, and uses Seedream to edit based on the source-frame canvas. Full character replacement: Uses the original video as the basis for actions and timeline, and locks the final character appearance with the target character image and repainted keyframes. Style transfer: Uses pure depth-structure video by default to preserve space, actions, and camera motion. This reduces interference from the original video's colors, materials, subtitles, and photography style. Multi-subject mapping: Multiple source subjects can be bound to different target images in image_urls through user_prompt. Controllable fallback: When allow_fallback=true, preprocessing failures can fall back to basic-equivalent pass-through generation. For strong semantic editing, set it to false to avoid returning unfinished editing results.
Intelligent Short Drama & Film Narration	Online	The "Intelligent Short Drama & Film Narration" operator can automatically build a plot event index based on multiple short drama episodes or a single movie asset. It plans jump-cut narration videos by storyline in three approximate duration ranges: short, mid, and long. The operator automatically completes visual editing, narration script generation, TTS dubbing, subtitle generation, mixing and rendering, and result upload. It supports Chinese and English narration, optional reference-audio voice cloning, and subtitle background box and position control. One task outputs one highlight narration material suitable for delivery on video platforms. Key features: Cross-episode plot understanding: Automatically analyzes multiple short drama episodes or movie materials, and extracts key events, character relationships, conflict progression, and visual highlights. Clue discovery and filtering: Discovers and filters candidate clues suitable for generating highlight narration, including main storylines, character lines, conflict lines, and relationship lines. Multilingual narration: Supports generating Chinese or English narration through narration_language. TTS voice control: Supports providing reference audio through tts_clone_prompt_audio_url. If not passed, the default voice of the execution service is used. Subtitle style and position control: Supports selecting translucent blurred, white rounded-corner, or black rounded-corner subtitle background boxes, and controls subtitle position through subtitle_position. Dense narration for ad delivery: Generates a fast-paced narration script by default, with a hook first and suspense at the end. It automatically completes TTS dubbing and subtitle rendering.
Audio Rebuild	Online	The Audio Rebuild operator can merge video footage based on the order of multiple input videos to be processed. It replans an audio timeline covering the whole film, generates unified audio, and replaces the original video tracks. It finally outputs the composed video, standalone rebuilt audio, and SRT subtitles. It is suitable for unified final-film scenarios for multiple AI-generated sub-videos. Key features: Sequential multi-segment merging: Splices videos in the segment order in the request to form a unified final film. Authoritative voice-over script pass-through: When voiceover_script is provided, it strictly uses that script as the only read-aloud content. It does not consume original segment audio, and does not perform semantic rewriting or expansion. Original audio transcription and correction: If no full-film voice-over script is provided, the original audio is transcribed segment by segment. When speech is recognized, prompt is only used as a reference for typos, proper nouns, pronunciations, and voice style. It does not add sentences that do not exist in the original audio. Explicit line-safe extraction: If speech is not recognized in a segment's original audio, only lines, voice-over, or narration explicitly marked in prompt are extracted. Actions, subtitles, lip shapes, and visual descriptions are filtered out. If there is no explicit read-aloud content, no lines are generated from scratch. Cross-segment audio unification: Merges speakers and voice styles across different segments, automatically plans a continuous timeline, and routes segments as speech, sound, or silence based on the content. It keeps the final film's voice timbre and listening experience as consistent as possible. Automatic audio generation and assembly: The system automatically selects full-film, layered, or segmented generation based on video duration and the audio timeline. Voice timbre, tone, BGM, sound effects, and mixing cues in the prompt can assist new-track generation, but BGM or sound effects from the original audio track are not retained. Voice-over density control: When using a full-film voice-over script, the operator can check whether the copy matches the video duration, avoiding excessive text density per unit time. Multiple result outputs: Returns the final video, standalone audio, SRT subtitles, and total rebuilt audio duration.
HyperFrames Video Generation	Online	The HyperFrames Video Generation operator supports describing video creation requirements in natural language. It combines image, audio, and video reference assets to complete video structure planning, content generation, rendering, mixing, and encoding, and outputs an accessible video URL and TOS object path. Key features: Supports describing creation requirements such as video themes, styles, subtitles, and on-screen copy using natural-language prompts. Supports passing in image, audio, and video reference assets, and precisely referencing them with @图像N, @音频N, and @视频N. Supports configuring the target duration, frame rate, aspect ratio, and resolution. Supports driving visual changes based on the rhythm of TTS voice-over, BGM, or reference audio. Supports completing video rendering, mixing, encoding, and result upload.

Content creation

Operator name	Usage	Operator introduction
Short Drama Script Generation	Online	The short drama/movie script generation operator is an automated script reverse-extraction tool for serialized or ultra-long video content, such as short dramas, long videos, and movies. Powered by a visual multimodal large model (VLM), the operator can automatically extract characters across the entire drama or film, analyze character relationships, and reverse-generate high-quality text scripts and character tables based on visuals and dialogue. The scripts include details such as scenes, actions, expressions, and dialogue. This helps with secondary creation, overseas translation, and copyright protection for video content. Key features: Consistent character recognition: Breaks through the limitation of isolated understanding for single episodes and single segments. It can stably track core characters in long serialized content or ultra-long movies. When facing costume changes, side faces, or complex scene switches, it ensures high consistency in cross-episode character identities and settings, and finally builds a complete global character table. High-fidelity script reverse restoration: Combines video visuals with dialogue lines to output professional storyboard scripts in reverse. In movie mode, the output includes precise timestamps. It accurately restores scene layouts, character emotions, body movements, and key dialogue. It provides high-quality text drafts that can be directly used for secondary development or translation proofreading. Dual-mode adaptive architecture: Short drama mode: Supports batch input of multiple short drama episodes. It processes them strictly in input order, and maintains the continuity of the serialized plot and character consistency. Movie mode: For single-episode movies or long recordings lasting several hours, it automatically starts a long-video adaptive processing strategy. This effectively mitigates detail forgetting caused by long contexts in large models. Flexible output format customization: Provides an open custom instruction (Prompt) interface. You can freely adjust the generation style of single-episode scripts according to specific business requirements, such as focusing on psychological descriptions, specific storyboard layout formats, or specific text markers. This meets the direct integration needs of different downstream businesses. Convenient result delivery: Supports securely writing the generated character table and complete scripts directly to your specified cloud storage (TOS). It can also generate packaged presigned download links.
Short Drama Script Fission	Online	Used to complete targeted rewriting based on existing episode scripts. The operator preserves the original script's core character structure, conflict chain, emotional curve, and satisfying-beat rhythm. It systematically rewrites character identities, world settings, regional context, industry background, and key terms, generating a new script version adapted to the target market and target genre. The output includes the fissioned episode script directory, complete script, outline, and before-and-after mapping table. It is suitable for short drama globalization, viral-hit replication, and content direction testing. Key features: Targeted rewriting with skeleton preservation: Preserves the original script's character functions, relationship topology, key event order, and mainline progression logic. Only the surface settings and expression context are rewritten. Localized rewriting: Supports generating target scripts for different regional contexts, including synchronized adaptation of character naming style, institutional background, dialogue expression, and social relationship logic. Genre-based rewriting: Supports overall style transfer by target genre, rewriting the same plot skeleton into new reusable scripts for different content tracks. Batch episode processing: Supports submitting multiple episode scripts at once through a directory or list. It processes them in input order and maintains complete-story narrative continuity. Structured result delivery: Outputs fissioned episode scripts, a complete script, an outline, and a mapping table. This facilitates subsequent video production, proofreading, manual refinement, and effect review. Episode script directory: The script for each fissioned episode, usually located at {output_tos_path}/{task_id}/episodes/. Complete script: The fully concatenated fissioned script, which facilitates overall review. Script outline: The refined structured outline, which facilitates quick viewing of the mainline direction. Mapping table: Records the mapping relationships for roles, events, conflicts, and terms before and after fission, which facilitates manual proofreading and review.
Short Drama Script Parsing	Online	The Short Drama Script Parsing operator automatically converts short drama script text into structured asset tables. The operator reads one or more script files, automatically identifies the script format and output language, extracts three types of assets: characters, props, and scenes, and supplements character appearances, character styling, prop visual descriptions, and scene spatial descriptions through visual expansion. It finally saves the results as JSON files. Key features: Multi-script input parsing: Supports passing one or more script files. It concatenates and processes them in input order, making it suitable for parsing single-episode, multi-episode, or complete scripts. Character asset table extraction: Identifies character names, genders, basic appearances, and appearance scenes. It supplements multiple styling descriptions that match the script background for main characters. Prop asset table extraction: Identifies key props, appearance locations, plot usage, and visual features, assisting prop design and asset management. Scene asset table extraction: Identifies main scenes, spatial types, time states, atmosphere, and visual layout, assisting scene construction and storyboard generation. Chinese and English script adaptation: Supports Chinese short drama numbering formats and English Hollywood/Fountain slug-line formats. In auto mode, it automatically identifies the script format and determines the asset table output language. Stable processing for ultra-long scripts: Long scripts exceeding the threshold are automatically chunked by scene boundaries, reducing timeout and truncation risks caused by ultra-long context requests. TOS artifact delivery: Character tables, prop tables, scene tables, and script background information are all saved as JSON files, and the corresponding TOS paths are returned.
Video Script Planner	Online	Enter a video topic and creation requirements to generate a video production plan for subsequent production. The plan breaks down ideas into a clear content structure, shot arrangement, material suggestions, and duration rhythm. It is suitable for early-stage planning of educational and marketing videos. Key features: From idea to production plan: Organizes the topic, audience, and expression goals into an executable video production plan. Shot and rhythm planning: Breaks down video content and plans the shot direction, duration, and transitions for each part. Material usage suggestions: Arranges the usage of video, image, and other materials based on content needs, facilitating subsequent production. Adaptation to different content types: Supports planning for course explanations and marketing e-commerce videos.
Video Prompt Generation	Online	The Video Prompt Generation operator organizes natural-language creative requirements into an executable video generation prompt. You can optionally pass up to 9 reference images to help the model understand people, products, scenes, or style assets. Key features: Organizes creative goals, content requirements, and language requirements into a single video generation prompt. Supports describing people, products, scenes, and visual styles with reference images.
Video Scene Segmentation	Online	The Video Scene Segmentation operator uses a multimodal large model to perform shot/scene segmentation, global character recognition, scene-level character association, and character image extraction on the input video. The operator outputs scene summary results, a character registry, sliced videos for each scene, and image files archived by character. This facilitates subsequent retrieval, editing, and content understanding. Key features: Supports VLM-based scene segmentation, and also supports equal-duration segmentation when min_segment_duration == max_segment_duration. Supports global character extraction and deduplication aggregation to generate a character registry. Supports character association within scenes, and outputs character appearance time ranges, keyframe timestamps, and bbox information in each scene. Supports automatically cutting independent video files for each scene. Supports extracting and filtering representative images for each character, and outputs them archived by character. Supports outputting token usage and LLM request count, facilitating cost evaluation.
LAS Storyboard Rewrite	Online	The LAS Storyboard Rewrite operator targets script production and e-commerce voice-over video workflows. It converts input text into a standard storyboard_bundle. It currently provides two capabilities: generating storyboards from a script combined with three asset JSON files for characters, props, and scenes, and directly generating storyboards from voice-over copy. Key features: Script and authoritative asset table to storyboard: Preserves the identifiers and settings of the input character, prop, and scene assets. It breaks down the script into shots, and binds each shot to the corresponding asset. Voice-over copy to storyboard: Preserves the original meaning and core selling points of the voice-over. It organizes shot duration, visual actions, voice-over, and structured descriptions that can be directly used for subsequent video generation by semantics. Faithful visual expansion: When the input already contains visuals and actions, it prioritizes reuse and supplements necessary details. When only voice-over exists, it designs visuals based on verifiable content in the copy, avoiding unfounded fabrication. Standard result delivery: Uniformly outputs storyboard_bundle_manifest.json, associating storyboards, characters, scenes, props, changes, and validation reports.
Video Storyboard Gen	Online	The reference-video product voice-over and storyboard generation operator targets e-commerce short video creation scenarios. The operator jointly understands the reference video, target product images, and optional natural-language requirements, and generates a standard storyboard package that can be directly handed to downstream text-to-video workflows. Key features: Jointly understands the reference video visuals, voice-over, and target product images, and identifies the product's core uses, target users, and visible selling-point evidence. Supports reference storyboards, copy, selling points, hooks, or free fusion, and specifies creative focus through user_prompt. Generates up to N+1 candidate sets by default, where N is the number of input videos. Partial candidates are allowed to pass, but at least one structurally usable result is returned. Each candidate set includes complete voice-over, a complete readable storyboard, structured descriptions shot by shot, and the generation basis and creative explanation. The output aligns with the standard storyboard bundle protocol, and can continue to be handed to text-to-video or other storyboard downstream processes.

Video understanding

Operator name	Usage	Operator introduction
Fine-grained Video Understanding	Online	The LAS Fine-grained Video Understanding API provides multidimensional and fine-grained structured understanding for various video content. Whether it is a short video, a movie clip, or a long meeting recording, users can upload a video to obtain searchable and question-answerable content data and fine-grained summaries. Key features Global fine-grained understanding: Supports hour-level videos, up to 3 hours and 10 GB, and generates coherent timelines and chapter summaries. Event and behavior recognition: Accurately detects key events, character actions, scene changes, and logical relationships. Video Q&A: Natural-language Q&A based on video content, quickly locating answers and timestamps. Efficient summaries and tags: Automatically generates chapter summaries, topic tags, and character relationships, facilitating knowledge management. Structured output: Provides JSON-format timeline and event lists, facilitating secondary processing or knowledge base construction.
Video prompt reverse generation	Online	The video prompt reverse generation operator is designed for video replication and storyboard re-creation scenarios. The operator first understands the source video facts. It then combines each storyboard's time range, visual description, and reference images to output executable video generation prompts for each segment. Key features: Generates prompts segment by segment based on source video facts and storyboard time windows, preventing content from being mixed across segments. Preserves verified subject identities, spatial relationships, key action sequences, original dialogue, and voice performance. Supports specifying the target visual style and target voice timbre without changing source video facts. Supports binding reference images in storyboards and returning the corresponding references in the generated results.
Enhanced Video Content Understanding (Doubao Series)	Online	The video content understanding operator supports understanding video files with Doubao models, including video content parsing and natural-language description generation. Compress the video to within 50 MB, and then use the Doubao model for video understanding. Supported video formats: mp4, wmv, webm, mkv, m4v, flv, avi, and mov. Because video file formats have many variants, not all files are guaranteed to be recognized. Verify through testing that the file can be recognized normally.

Video editing

Operator name	Usage	Operator introduction
Intelligent Video Editing	Online	The Intelligent Video Editing operator implements intelligent video editing capabilities based on a multimodal large model. It helps users quickly extract valuable content segments from long videos. It supports understanding editing requirements described in natural language, reference-image assisted recognition for characters, objects, scenes, and more, multidimensional video content analysis for visuals, subtitles, and plot, and outputs standardized editing decision information, such as timestamps, descriptions, and tags. Key features: Supports multiple editing scenarios: character segment extraction, highlight segment detection, product segment detection, custom editing, and more. Flexible understanding of editing requirements based on natural-language descriptions, supporting custom user requirements. Supports reference-image assisted recognition for characters, objects, scenes, and more. Multidimensional video content analysis for visuals, subtitles, and plot. Supports ASR-enhanced semantic understanding. It is suitable for videos with rich dialogue content and no subtitles, improving segment boundary smoothness. Supports rendering the three elements of short dramas: title, prompt text, and corner mark. It is suitable for vertical short drama scenarios. Supports the highlight-first feature, automatically extracting a 10-15 second attractive segment as the opening. Standardized editing decision output, such as timestamps, descriptions, and tags. Automatically generates video segment files and uploads them to TOS.
Viral Clip Generation	Online	The Viral Clip Generation operator can automatically complete storyboard detection, semantic analysis, editing decisions, and video composition based on multiple short drama episodes. It batch-generates multiple traffic-driving material videos suitable for delivery on short video platforms. Key features: Batch processing of multiple short drama episodes: Inputs multiple short drama episodes at once, automatically completes all analysis and editing, and batch-outputs multiple viral materials. Intelligent editing scheme generation: Automatically analyzes plot content and generates multiple editing schemes, including sequential cuts and jump cuts, to meet different delivery needs. Automatic cleaning of non-main-content footage: Automatically detects and removes irrelevant footage, such as freeze frames and repeated content before and after episodes. Storyboard-level semantic understanding: Performs plot understanding and importance rating for each shot, and outputs structured storyboard analysis results. Script restoration: Automatically restores the complete script of multiple short drama episodes, including character relationships, scene structure, and plot context. Rendering the three elements of short dramas: Supports adding the drama title, prompt text, and corner mark to materials, adapting to vertical short drama scenarios. Highlight-first: Supports placing highlight segments at the beginning of materials to quickly attract audience attention.
Intelligent Video Remix	Online	The Intelligent Video Remix operator is based on understanding input materials, speech recognition, semantic retrieval, and large-model orchestration capabilities. It automatically matches, sorts, edits, and composes voice-over, b-roll, or long real-shot materials into a final video according to copy, structured storyboards, or natural-language creative requirements. Key features: The operator first completes material understanding and storyboard matching, then generates an editing scheme according to the configured editing scenario mode (mode). Supported editing scenario modes include: Voice-over original-audio matching: mode=speech_match. Input the script copy to edit and extract. The operator locates corresponding segments in the video material ASR, and edits and outputs the original audio and visuals of matched segments. In such scenarios, both the video visuals and audio of the input video materials can be directly used for editing output. Visual semantic matching: mode=visual_match. Use natural-language descriptions to describe the video materials that need to be extracted and edited, such as descriptions of video visuals or video content copy. In this scenario, the visuals of the input video materials can usually be used for editing output, but the audio is usually unavailable. You can choose to input TTS narration to replace the original material audio. Voice-over marketing remix: mode=marketing_headtalk performs intelligent rough cuts on recorded voice-over materials and selectively inserts b-roll based on semantics. A common editing template for real-person automotive voice-over scenarios is currently preset for you, and you can use it directly. Long real-shot process editing: mode=long_real_shot targets long real-shot materials such as price negotiation, vehicle delivery, repair, and store visits. It organizes a complete storyline with original audio and visuals for intelligent editing, and supports setting the output video duration through the target_duration_sec parameter. In different editing scenario modes, you can flexibly set the application scenarios of the input video materials. Use the material role (role) parameter to mark input video materials as speech (voice-over/subject material), voiceover (audio only), or broll (visuals only). Structured storyboard: Supports using speech, visual_description, duration, and video_id to control each script unit. Intelligent packaging: Supports subtitle effects, sound effects, and BGM. Scheme and artifact delivery: After processing is complete, the operator returns the final editing scheme, SRT subtitles, a final video without subtitles, a final video with subtitles, and other artifacts. You can also view the matching relationship between input script units and video material segments in the generated editing scheme and final video through data.match_scheme.

Video translation

Operator name	Usage	Operator introduction
Video Translation	Online	The Video Translation operator can efficiently and accurately convert video content from the source language to one or more target languages. Its service scope includes not only subtitle translation, but also speech translation. It finally outputs dubbed videos and subtitle files in the corresponding languages. Key features: Multilingual support: Supports translation for multiple languages. Input videos are supported in 25 languages, and output audio dubbing is supported in 31 languages, covering common languages such as Chinese, English, Japanese, Indonesian, Spanish, Portuguese, Korean, French, and German. With the powerful translation capability of large models, it can achieve extremely high translation accuracy and term localization capabilities, meeting the needs of global content distribution. Voice cloning: Accurately extracts the speaker's voice from the video to achieve a 1:1 restoration of the speaker's voice characteristics. At the same time, the translated speech can be precisely aligned with the duration of the original video, ensuring video fluency and consistency. Convenient result delivery: Supports securely writing translated subtitles, dubbed vocal audio, and videos directly to your specified cloud storage (TOS), while generating presigned download links.
Video Subtitle Translation	Online	The Video Subtitle Translation operator supports extracting subtitles from videos and translating them into multiple languages. Users can choose to recognize embedded subtitles in the picture through OCR, or extract audio subtitles through ASR. The recognized subtitles are then refined and translated, and subtitle files in multiple formats are output. Key features: Dual subtitle sources: Supports two methods: OCR recognition of embedded subtitles in the picture, and ASR extraction of audio subtitles. Multiple accuracy levels: Each subtitle source supports two configuration levels: low accuracy and high accuracy, meeting needs in different scenarios. Multilingual translation: Supports subtitle translation in 25 languages, including Chinese, English, Japanese, Korean, French, German, and Spanish. Multi-format output: Supports simultaneously outputting subtitle files in multiple formats. Bilingual subtitle recognition: OCR mode supports recognizing translation-type bilingual subtitles and dialogue-type bilingual subtitles.

Video restoration

Operator name	Usage	Operator introduction
Video Inpaint	Online	Inputs a single video and outputs a restored video with specified content erased. It supports subtitles, scrolling subtitles, text watermarks, corner marks, logos, and episode information. It can separately control the erasure ranges for subtitles and watermarks. Key features: Supports multiple erasure targets: watermarks, subtitles, scrolling subtitles, and more. Intelligent detection based on a multimodal large model to accurately locate regions that need restoration. Supports precise mask generation, preserving edge details. Supports video segment processing, making long-video processing more stable. Automatically preserves audio without extra operations. Supports outputting TOS addresses, with results automatically uploaded.

Content safety

Operator name	Usage	Operator introduction
Sensitive Content Masking	Online	The Sensitive Content Masking operator uses multimodal large models, OCR, and ASR to detect sensitive images, text, and speech in images or videos. It automatically locates and masks them, mutes sensitive speech, and outputs the processed media file and detection result JSON. Key capabilities: Supports identifying sensitive images, text, and speech in images or videos. Supports multiple masking methods, including Gaussian blur, mosaic, solid-black occlusion, and radial blur. Supports muting sensitive speech in videos. Supports outputting processed media files, structured detection results, and per-frame video mask information.

Video processing

Operator name	Usage	Operator introduction
Face Blur	Online	The Face Blur operator is an automated face blurring tool for video content. The operator can automatically recognize faces in videos and blur them according to the blur level specified by the user, protecting user privacy. Key capabilities Automatically detects faces in video frames and blurs them Supports multiple blur types, including mosaic and Gaussian Supports fine-grained blurring of multiple different regions, such as face oval blur, face-fitting blur, and eye-region blur Uniformly outputs the blurred video path, even if no face is detected, the video is re-encoded and output
Video Resolution Adjustment (online)	Online	Video Resolution Adjustment operator. Key features: Intelligently adjusts resolution to within the specified range Supports multiple aspect ratio preservation strategies Controls video quality and encoding parameters Keeps the audio stream unaffected
Audio and Video Merge	Online	The Audio and Video Merge operator uses FFmpeg to sequentially splice, adjust durations, and finally compose input video and audio materials. The operator supports multiple input combinations, such as 1-to-1, 1-to-many, many-to-1, and many-to-many. When the total durations of video and audio are inconsistent, it automatically selects speed adjustment alignment or trimming to the shorter duration based on the configuration, and uploads the result video and processing mapping file to TOS. Key features: Supports sequential splicing of multiple video segments. Supports sequential splicing of multiple audio segments. Supports preprocessing video and audio separately to the target duration before composition. Supports automatically selecting an alignment strategy: speed adjustment first, and automatic trimming when the threshold is exceeded. Supports outputting the final video file and mapping file, facilitating tracking of the inputs, durations, and alignment strategies of each merge. The output directory is automatically isolated by account, request link, and input hash to avoid overwriting results from different tasks.
Video Frame Interpolation	Online	The Video Frame Interpolation operator is used to increase the frame rate of input videos. It improves picture smoothness by generating intermediate frames, and outputs a new high-frame-rate video file. You can specify the target frame rate, select the frame interpolation mode, and decide whether to preserve the original audio stream as needed. Limitations The input video must be accessible by the service, with support for http/https and tos://. output_tos_path must be a TOS directory writable by the current account. target_fps must be greater than 0, and cannot be lower than the source video frame rate. The current version limits video duration to 3 hours and video size to 10GB. The operator depends on CUDA GPU and a video frame interpolation model. Higher resolution, higher target frame rate, and longer video result in longer overall processing time.
Video frame sampling	Online	The Video Frame Extraction operator supports extracting frames from an input video at a specified frame rate, and uploading the extracted image frames to a specified TOS storage path. After the task is complete, you can obtain video metadata and the access address of each frame. The operator supports configurable frame extraction frequency, maximum frame count limit, output image format, and scaling strategy. It is suitable for video understanding, detection, moderation, summaries, cover generation, and other scenarios. Key features: Frame extraction by frame rate: Uniformly extracts frames from video at a specified FPS (0.1 ~ 5.0). Maximum frame count limit: You can set the max_frames parameter to control the upper limit of output frames and prevent long videos from generating too many frames. Multi-format output: Supports outputting two image formats: jpg and png. Flexible scaling: Supports scaling by short side (resize_short_side) or specified target resolution (resize_hw), meeting needs in different scenarios. Custom output path: Customizes the storage path of frame-extracted images on TOS through an output path template. Long video support: Suitable for processing long videos. You can control output scale through max_frames.

Audio

Audio generation

Operator name	Usage	Operator introduction
Voice Cloning Synthesis	Online	This operator synthesizes input text into speech audio. It supports designing a voice timbre through a voice style description, and also supports controllable cloning or high-fidelity continuation based on reference audio. Key features: Multiple speech synthesis capabilities: Voice Design text-controlled timbre: Generates speech only through style_prompt describing the target voice style, with no sample audio required. Controllable voice cloning: Provides sample audio through reference_audio_uri, and synthesizes target text based on the sample audio timbre. High-fidelity voice continuation: Provides both prompt_audio_uri and prompt_text, improving consistency in timbre, speech rate, and prosody. Long-text splitting and splicing: Supports splitting long text by natural boundaries, synthesizing it segment by segment, and automatically splicing the segments. Quality self-check and retry: Supports automatic regeneration when suspected low-quality results are detected. Multi-format audio output: Supports wav, mp3, ogg_opus, and pcm, with configurable sample rate and bit rate. Object storage persistence: Supports multiple audio formats. It can write results to TOS, and return information such as audio address, format, sample rate, duration, and character count.
Speech synthesis	Online	The speech synthesis operator synthesizes text or supported SSML into speech. By default, it uses Doubao Speech Synthesis 2.0. Through the synchronous Process API, it returns audio, subtitles, and synthesis completion events frame by frame over SSE. It supports configuring voice timbre, sample rate, speech speed, and volume. Key capabilities: Supports text and SSML speech synthesis. You can use preset voice timbres or activated Voice Cloning 2.0 voice timbres. Continuously returns audio segments through SSE, suitable for online playback and real-time processing. Supports adjusting speech speed, volume, pitch, and end-of-sentence silence, and supports configuring audio format, sample rate, and bit rate. Supports returning subtitles and character-level timestamps for audio-video alignment and subsequent editing.

Audio recognition

Operator name	Usage	Operator introduction
Speech-to-text (Doubao speech ASR)	Online	The Speech-to-Text (Doubao Series) operator is a speech recognition module and a recording transcription solution based on the LAS ASR service. Key features Integrates with the Volcengine LAS ASR API Supports automatic sentence segmentation, inverse text normalization for numbers, and speaker or channel separation (optional) Concurrently processes multiple audio files, and provides two outputs: structured JSON and readable text Suitable for transcribing recording files up to 2 hours long. It supports advanced features such as punctuation completion, intelligent sentence segmentation, and speaker separation.
Speech-to-Text (Doubao - Audio File Recognition) Enhanced Version	Online	The LAS Speech-to-Text (Doubao - Audio File Recognition) Enhanced Version operator is based on the Doubao audio file recognition large model. It can transcribe speech in input audio/video files into text output. It supports multiple audio/video formats, multiple languages, audio noise reduction, and large-file processing. It is suitable for content quality inspection and review, audio/video subtitle generation, speech search, classroom content analysis, and other scenarios. Key features Multi-format audio/video input recognition: In addition to audio, video file input is newly supported. The LAS operator can automatically extract the video audio track for recognition. In addition to raw/wav/mp3/ogg, container formats such as mp4/mov/mkv/flac are supported. The LAS operator has no file size or duration limit for input audio/video files. In addition to public Internet https url access, tos intranet path access is also supported (tos://bucket-name/path name/file name). Enhanced audio preprocessing to improve model performance: Built-in audio noise reduction module can effectively reduce the impact of background noise on recognition, improving transcription accuracy for recording files. Multilingual support: Can automatically identify the language or recognize by a user-specified language. Expands recognition languages to 99, meeting audio data processing needs for multiple languages and regions.
Speech-to-Text (Doubao-Seed-2.0-lite) Enhanced Version	Online	The LAS Speech-to-Text (Doubao-Seed-2.0-lite) Enhanced Version operator enables the Doubao-Seed-2.0-lite multimodal refinement capability by default on top of underlying SeedASR recognition. It first completes input download, audio extraction, duration detection, and basic ASR recognition. It then sends audio slices, recognition drafts, and hotword context to the multimodal model for refinement, correcting proper nouns, numbers, punctuation, and semantic coherence, and completing missed recognition content. It supports multiple audio/video formats, multiple languages, speaker separation, hotword enhancement, and sensitive word protection. It is suitable for meetings, interviews, podcasts, courses, content quality inspection, and other scenarios with high text quality requirements. Key features: Multimodal refinement, enabled by default: On top of the basic recognition results, calls the Doubao-Seed-2.0-lite multimodal model to correct and complete proper nouns, numbers, punctuation, and semantic coherence. This significantly improves transcription text quality. Falls back to basic ASR results when refinement fails, ensuring availability. Multi-format audio/video input recognition: Supports audio and video file input, and automatically extracts video audio tracks for recognition. Rich recognition capabilities: Supports text normalization (ITN), automatic punctuation, disfluency detection and correction (DDC), and speaker separation. Hotwords and sensitive words: Supports direct hotword passing and hotword list enhancement. Hotwords enter both the underlying ASR and the refinement Prompt. Supports sensitive word protection. System-desensitized content can be excluded from the refinement model, and custom blanking or asterisk replacement is supported.

Audio processing

Operator name	Usage	Operator introduction
Vocal Separation	Online	Separates the first audio track in audio or video into two tracks: a vocal track and an accompaniment/background track. It supports WAV, MP3, and AAC output. Video input can also automatically generate an MP4 that retains the original visuals and contains only the vocal track. It is suitable for scenarios such as removing background music for derivative creation, extracting cleaner vocals for ASR/subtitle preprocessing, and accompaniment extraction. Key features: Dual-track output: Returns a vocal track and a non-vocal residual track based on the input audio or video. Vocal video output: Video input can additionally generate an MP4 that retains the original visuals and contains only the vocal track. Common peak gain: Applies the same safe gain to both audio tracks while preserving their relative ratio. Multi-format output: Supports WAV, MP3, and AAC audio formats to meet needs such as post-production editing, algorithm processing, auditioning, and distribution.
Audio Format Conversion (online)	Online	The "Audio Format Conversion" operator. Audio Format Conversion converts audio or video files into a specified audio format uniformly, and outputs them to a specified storage path. This operator is mainly used for scenarios such as audio format standardization in data processing pipelines, audio extraction from videos, and training data preparation. It supports batch concurrent processing and configurable audio encoding parameters Key features Uniformly converts audio/video to audio Supports custom output audio formats Supports custom output paths (TOS) Supports audio encoding parameter extension Batch concurrent processing capability
Audio Segmentation	Online	The "Audio Segmentation" operator is used to extract audio from audio or video files, split the audio into multiple segments according to specified rules, and output them to the storage path specified by the user. This operator is mainly used for structured processing scenarios for long audio or video, such as audio preprocessing, data segmentation, and training data construction. It supports batch concurrent processing and flexible output path organization. Key features Audio/video extraction and segmentation Supports custom segmentation rules Supports custom output audio formats Supports output path templates Supports audio encoding parameter extension

Images

Image generation

Operator name	Usage	Operator introduction
Image Generation (Seedream Series)	Online	The Image Generation (Seedream Series) operator can generate high-quality images based on user-input text or reference images. It supports image groups and streaming output. Key features Text-to-image/image-to-image/image group generation Supports streaming output (SSE) and non-streaming output Output formats support url and b64_json

Image processing

Operator name	Usage	Operator introduction
Image Resampling	Online	The image resampling operator is used to resize input images by resampling, with downsampling only supported, and save the results to the user-specified TOS directory. It supports 4 interpolation algorithms (nearest/bilinear/bicubic/lanczos) and .jpg / .png output formats. It is suitable for image preprocessing, data standardization, offline dataset construction, and other scenarios. Key features Multiple interpolation algorithms nearest: Fastest speed, suitable for pixel-style images bilinear: Balanced speed and quality bicubic: Smoother high-quality scaling lanczos: Better anti-aliasing effect, suitable for photos URL / TOS input support image_src_type=image_url: Inputs a public Internet URL image_src_type=image_tos: Inputs a tos:// address Output to TOS tos_dir is required, specifying the output directory at the folder level The output file name is generated by the server, can be named with help from image_name, and appends the _resample suffix identifier Output format and DPI control Output formats support .jpg / .png Supports setting output DPI (target_dpi)

Documents

Document parsing

Operator name	Usage	Operator introduction
PDF Document Parsing (Doubao)	Online	The PDF content parsing operator supports visual model parsing and Markdown structured output for PDF files. Key features Supports PDF page rendering and visual model parsing, outputs high-fidelity Markdown, and fully restores the original structure, including heading hierarchy, tables, formulas, and image regions. Automatically recognizes image regions and returns boundingbox information and image presigned URLs. Supports per-page and whole-book Markdown aggregation, facilitating subsequent content processing and display.

Multimodal

Multimodal vectorization

Operator name	Usage	Operator introduction
Image-Text Embedding (Doubao Models)	Online Offline	Multimodal vector generation processor. It supports joint vector generation for images/videos and text, implementing cross-modal retrieval capabilities. Key features Multimodal vectorization support: Supports joint vector generation for images/videos and text, implementing cross-modal retrieval capabilities. Reference document. Input format adaptation: Natively supports input formats such as base64 encoding, binary data, and URLs for images/videos Automatically processes media format conversion (JPEG/PNG/MP4/AVI and more)

Multimodal deep thinking

Operator name	Usage	Operator introduction
Multimodal Deep Thinking (Doubao-seed-2.0)	Online	Provides deep thinking capabilities of large models in multimodal scenarios. It uses models with deep thinking capabilities to analyze and understand images, videos, or text, and returns structured text output. The operator automatically constructs a message structure that conforms to multimodal model specifications. Users only need to provide image/video/text data as agreed to complete inference. Key features Deep thinking mechanism: The model automatically performs problem decomposition and logical reasoning before answering questions, generating a chain of thought (reasoning_content) Multimodal scenario support: Simultaneously supports image/video/text input, and automatically completes multimodal message assembly Simplified input mechanism: Supports multiple data sources, such as local files, HTTP/HTTPS URLs, and TOS/S3 object storage. Visual understanding can be implemented through simple configuration Flexible thinking mode: Supports controlling the deep thinking mode through the thinking_type parameter (enabled / disabled / auto), flexibly balancing answer quality and performance
Multimodal Deep Thinking (Doubao-seed-1.8)	* Online

Object detection and segmentation

Operator name	Usage	Operator introduction
Spatial Perception	Online	The Spatial Perception operator provides object detection, instance segmentation, and cross-frame tracking capabilities for images and videos. Callers can specify targets using natural language, pixel points, or pixel boxes, and obtain object boxes, instance masks, and video trajectory identifiers mapped to source media coordinates. It is suitable for embodied operations, industrial inspection, and content analysis scenarios that require structured visual results. Key features Text target detection: Returns the target bbox in images or video sampling frames based on object name, category, or appearance description. Video results include cross-frame trajectory identifiers. Interactive instance segmentation: Uses positive and negative points or pixel boxes to constrain the target, and returns an instance mask on the source-resolution grid. Segmentation after text localization: Uses grounded_segmentation@v1 to first find the target based on the text description, then accurately outlines it from the background along the target contour, and continuously tracks it in subsequent video frames. Video target tracking: Maintains object instances and trajectory identities between sampled frames, and returns bbox, mask, and time information frame by frame.

Embodied intelligence

Semantic annotation

Operator name	Usage	Operator introduction
Embodied semantic annotation	Online	The embodied semantic annotation operator is designed for embodied data collection scenarios such as real robot, UMI, and human data. It identifies interactive objects in operations, annotates action descriptions, start and end times, and skill tags, and supports temporal annotation combined with candidate task descriptions. Key capabilities: Supports identifying main interactive objects and distinguishable instances, and annotating interaction time periods. Supports splitting continuous operations and generating task descriptions, action start and end times, action descriptions, and skill tags. Supports combining candidate task descriptions to locate operations that actually occur in the video and their time ranges. Supports configuring the frame extraction frame rate, video processing resolution, and analysis time range.

Vector retrieval

Generic search

Operator name	Usage	Operator introduction
Lance Generic Search	Online	The Lance Generic Search operator provides unified and efficient retrieval capabilities for data in Lance and LanceDB formats stored on Volcengine Object Storage (TOS). It supports vector retrieval, full-text retrieval, hybrid retrieval, and scalar queries, and is suitable for massive structured and unstructured data retrieval scenarios. Key features: Multi-mode retrieval support: Supports four retrieval modes: vector retrieval (vector), full-text retrieval (fts), hybrid retrieval (hybrid), and scalar query (scalar). Multimodal input: Vector retrieval supports three input types: vector arrays, text strings, and Base64-encoded images. Data filtering: Supports SQL WHERE clauses for pre-filter or post-filter data filtering. Result reranking: Hybrid retrieval mode supports the RRF (Reciprocal Rank Fusion) reranking algorithm. Column selection: Supports specifying returned columns to reduce data transfer.

Last updated: 2026.10.13 15:09:42