Visual content understanding (Doubao series models)

• Offline

• Use ModelArk's large model to analyze and understand images, videos, or text, and return text output.
• Key features:
  • Multimodal scenario support: Supports any combination of image, video, and text inputs. The operator automatically constructs a message structure that complies with multimodal model specifications.
  • Rich input sources: Supports http/https/tos/s3 protocol URLs, Base64 encoding, and raw binary data. TOS/S3 addresses are automatically pre-signed.
  • Flexible data formats: Image, video, and text columns all support string (single) or list (multiple) input forms.

Visual content understanding (Doubao-1.5-vision-pro)

• Offline

• Use ModelArk's large models to analyze and understand images, videos, or text, and return text output.
• Key features:
  • Multimodal scenario support: Supports any combination of image, video, and text inputs. The operator automatically constructs a message structure that complies with multimodal model specifications.
  • Diverse input sources: Supports http/https/tos/s3 protocol URLs, Base64 encoding, and raw binary data. TOS/S3 addresses are automatically pre-signed.
  • Flexible data formats: Image, video, and text columns all support string (single) or list (multiple) input forms.

Visual content understanding (Doubao-1.5-vision-pro-32k)

• Offline

• Use ModelArk's large models to analyze and understand images, videos, or text, and return text output.
• Key features:
  • Multimodal scenario support: Supports any combination of image, video, and text inputs. The operator automatically constructs a message structure that complies with multimodal model specifications.
  • Diverse input sources: Supports http/https/tos/s3 protocol URLs, Base64 encoding, and raw binary data. TOS/S3 addresses are automatically pre-signed.
  • Flexible data formats: Image, video, and text columns all support string (single) or list (multiple) input forms.

Text generation (Doubao/DeepSeek series models)

• Offline

• Large model text generation dedicated processor (Doubao/DeepSeek)
• Key features:
  • Pure text scenario optimization: Automatically constructs a message structure that complies with model specifications based on user input text data
  • Input simplification mechanism: Natively supports input of type "str" and automatically encapsulates it as {role: user, content: text} format
  • Supports NLP scenarios such as translation, summarization, and question answering out of the box
  • Dual prompt system:
    • system_content: System-level behavior guidance (such as translation style control)
    • prompt: User-level instruction template (supports {query} placeholder replacement)

Text generation (Doubao-lite-32K)

• Offline

• Large model text generation dedicated processor (Doubao/DeepSeek)
• Key features:
  • Pure text scenario optimization: Automatically constructs a message structure that complies with model specifications based on user input text data
  • Input simplification mechanism: Natively supports input of type "str" and automatically encapsulates it as {role: user, content: text} format
  • Supports NLP scenarios such as translation, summarization, and question answering out of the box
  • Dual prompt system:
    • system_content: System-level behavioral guidance (such as translation style control)
    • prompt: User-level instruction template (supports {query} placeholder replacement)

Text generation (Doubao-1.5-pro-32K)

• Offline

• Dedicated processor for large model text generation (Doubao/DeepSeek)
• Key features:
  • Plain text scenario optimization: Automatically constructs message structures compliant with model specifications based on user input text data
  • Input simplification mechanism: Natively supports str-type input and automatically encapsulates it into the {role: user, content: text} format
  • Multi-task support: Out-of-the-box for NLP scenarios such as translation, summarization, and question answering
  • Dual prompt system:
    • system_content: System-level behavioral guidance (such as translation style control)
    • prompt: User-level instruction template (supports {query} placeholder replacement)

Text generation (Deepseek-V3)

• Offline

• Dedicated processor for large model text generation (Doubao/DeepSeek)
• Key features:
  • Plain text scenario optimization: Automatically constructs message structures compliant with model specifications based on user input text data
  • Input simplification mechanism: Natively supports str-type input and automatically encapsulates it into the {role: user, content: text} format
  • Multi-task support: Out-of-the-box for NLP scenarios such as translation, summarization, and question answering
  • Dual prompt system:
  • system_content: System-level behavioral guidance (such as translation style control)
  • prompt: User-level instruction template (supports {query} placeholder replacement)

Text generation (Doubao-1.5-lite-32K)

• Offline

• Dedicated processor for large model text generation (Doubao/DeepSeek)
• Key features:
  • Plain text scenario optimization: Automatically constructs message structures compliant with model specifications based on user input text data
  • Input simplification mechanism: Natively supports str-type input and automatically encapsulates it into the {role: user, content: text} format
  • Multi-task support: Out-of-the-box for NLP scenarios such as translation, summarization, and question answering
  • Dual prompt system:
    • system_content: System-level behavioral guidance (such as translation style control)
    • prompt: User-level instruction template (supports {query} placeholder replacement)

Deep reasoning (DeepSeek-R1)

• Offline

• Leveraging the deep reasoning capability of the DeepSeek-R1 model, the model analyzes and understands text and returns structured output. Before answering questions, the model performs question analysis and decomposition, and generates answers based on this process.
• Key features:
  • Deep thinking mechanism: Before answering questions, the model automatically decomposes the question and performs logical reasoning, generating a chain of thought (reasoning_content) to enhance the comprehensiveness and depth of the answer.
  • Pure text input: This operator variant is designed specifically for text tasks and only supports input from the texts column.
  • Flexible thinking mode: Supports controlling the deep thinking mode via the thinking_type parameter (enabled / disabled / auto), allowing flexible trade-offs between answer quality and performance.

Deep thinking (Doubao-1.5-thinking-pro)

• Offline

• Provides deep thinking capabilities of large models in modal scenarios, uses the Doubao-1.5-thinking-pro model with deep thinking capabilities to analyze and understand text, and returns structured output. Before answering questions, the model analyzes and decomposes the question, and generates answers based on this analysis.
• Key features:
  • Deep thinking mechanism: Before answering questions, the model automatically decomposes the question and performs logical reasoning, generating a chain of thought (reasoning_content)
  • Input simplification mechanism: Supports multiple data sources including local files, HTTP/HTTPS URLs, and TOS/S3 object storage. Visual understanding capabilities can be achieved through simple configuration.
  • Flexible thinking mode: Supports controlling the deep thinking mode via the thinking_type parameter (enabled / disabled / auto), allowing flexible trade-offs between answer quality and performance

Text embedding (Doubao-embedding-large)

• Offline

• Text vectorization generation processor
• Text vectorization support: Supports generating text vectors to enable text retrieval capabilities. Reference documentation: https://www.volcengine.com/docs/82379/1521766

Text vectorization (Doubao-embedding)

• Offline

• Text vectorization generation processor
• Text vectorization support: Supports generating text vectors to enable text retrieval capabilities. Reference documentation: https://www.volcengine.com/docs/82379/1521766

Multimodal deep reasoning (Doubao series models)

• Offline

• Provides deep thinking capabilities of large models in multimodal scenarios, uses models with deep thinking capabilities to analyze and understand images, videos, or text, and returns structured text output. The operator automatically constructs a message structure that complies with multimodal model specifications. Users only need to provide image, video, or text data as specified to complete inference.
• Key features:
  • Deep reasoning mechanism: The model automatically decomposes questions and performs logical reasoning before answering, generating a reasoning chain (reasoning_content).
  • Multimodal scenario support: Supports image, video, and text input simultaneously, and automatically assembles multimodal messages.
  • Input simplification mechanism: Supports multiple data sources such as local files, HTTP/HTTPS URLs, and TOS/S3 object storage. Visual understanding capabilities can be enabled through simple configuration.
  • Flexible reasoning mode: Supports controlling the deep reasoning mode (enabled / disabled / auto) via the thinking_type parameter, allowing flexible trade-offs between answer quality and performance.

Multimodal deep reasoning (Doubao-Seed-1.6)

• Offline

• Provides deep reasoning capabilities of large models in multimodal scenarios, uses models with deep reasoning capabilities to analyze and understand images, videos, or text, and returns structured text output. The operator automatically constructs a message structure that complies with the multimodal model specification. Users only need to provide image, video, or text data as specified to complete reasoning.
• Key features:
  • Deep reasoning mechanism: The model automatically decomposes questions and performs logical reasoning before answering, generating a reasoning chain (reasoning_content).
  • Multimodal scenario support: Supports image, video, and text input simultaneously, and automatically assembles multimodal messages.
  • Input simplification mechanism: Supports multiple data sources such as local files, HTTP/HTTPS URLs, and TOS/S3 object storage. Visual understanding capabilities can be enabled through simple configuration.
  • Flexible reasoning mode: Supports controlling the deep reasoning mode (enabled / disabled / auto) via the thinking_type parameter, allowing flexible trade-offs between answer quality and performance.

Multimodal deep reasoning (Doubao-Seed-1.6-flash)

• Offline

• Provides deep reasoning capabilities of large models in multimodal scenarios, uses models with deep reasoning capabilities to analyze and understand images, videos, or text, and returns structured text output. The operator automatically constructs a message structure that complies with the multimodal model specification. Users only need to provide image, video, or text data as specified to complete reasoning.
• Key features:
  • Deep reasoning mechanism: The model automatically decomposes questions and performs logical reasoning before answering, generating a reasoning chain (reasoning_content).
  • Multimodal scenario support: Supports simultaneous input of images, videos, and text, and automatically assembles multimodal messages.
  • Input simplification mechanism: Supports multiple data sources including local files, HTTP/HTTPS URLs, and TOS/S3 object storage. Visual understanding capabilities can be achieved through simple configuration.
  • Flexible reasoning mode: Allows control of the deep reasoning mode via the thinking_type parameter (enabled / disabled / auto), enabling flexible trade-offs between answer quality and performance.

Multimodal deep reasoning (Doubao-Seed-1.6-thinking)

• Offline

• Provides deep reasoning capabilities of large models in multimodal scenarios. Models with deep reasoning capabilities analyze and understand images, videos, or text, and return structured text output. The operator automatically constructs a message structure compliant with multimodal model specifications. Users only need to provide image, video, or text data as required to complete inference.
• Key features:
  • Deep reasoning mechanism: The model automatically decomposes questions and performs logical reasoning before answering, generating a reasoning chain (reasoning_content).
  • Multimodal scenario support: Supports simultaneous input of images, videos, and text, and automatically assembles multimodal messages.
  • Input simplification mechanism: Supports multiple data sources including local files, HTTP/HTTPS URLs, and TOS/S3 object storage. Visual understanding capabilities can be achieved through simple configuration.
  • Flexible reasoning mode: Allows control of the deep reasoning mode via the thinking_type parameter (enabled / disabled / auto), enabling flexible trade-offs between answer quality and performance.

Multimodal deep reasoning (Doubao-1.5-thinking-vision-pro)

• Offline

• Provides deep reasoning capabilities of large models in multimodal scenarios. Models with deep reasoning capabilities analyze and understand images, videos, or text, and return structured text output. The operator automatically constructs a message structure compliant with multimodal model specifications. Users only need to provide image, video, or text data as required to complete inference.
• Key features:
  • Deep reasoning mechanism: The model automatically decomposes questions and performs logical reasoning before answering, generating a reasoning chain (reasoning_content)
  • Multimodal scenario support: Supports simultaneous input of images, videos, and text, and automatically assembles multimodal messages
  • Input simplification mechanism: Supports multiple data sources including local files, HTTP/HTTPS URLs, and TOS/S3 object storage. Visual understanding capabilities can be enabled with simple configuration
  • Flexible reasoning mode: Allows control of the deep reasoning mode via the thinking_type parameter (enabled / disabled / auto), enabling flexible trade-offs between answer quality and performance

Image-text embedding (Doubao series models)

• Online
• Offline

• Multimodal vector generation processor. Supports joint vector generation for images/videos and text, enabling cross-modal retrieval capabilities.
• Key features
  • Multimodal vectorization support: Supports joint vector generation for images/videos and text, enabling cross-modal retrieval capabilities. Reference documentation: https://www.volcengine.com/docs/82379/1523520
  • Input format adaptation:
    • Native support for input formats such as base64 encoding, binary data, and URLs for images/videos
    • Automatic handling of media format conversions (JPEG, PNG, MP4, AVI, and more)

Audio scoring (Audiobox Aesthetics)

• Offline

• Audio scoring operator – Uses audiobox_aesthetics to score audio quality
• Key features
  • Uses the audiobox_aesthetics model to score the quality of audio segments
  • Provides four scoring dimensions: CE (coherence/listening engagement), CU (clarity/comprehensibility), PC (production quality/compositional quality), PQ (perceived quality/subjective audio quality)
  • Supports local and remote (tos://) audio files
  • Applicable to audio quality assessment, audio filtering, quality control, and similar scenarios

Audio silence detection

• Offline

• Audio silence detection processor, intelligently identifies whether audio is completely silent
• Key features:
  • Silence detection: Analyzes the entire audio file to determine whether it is silent
  • Configurable threshold: Supports custom sensitivity threshold (dB) for silence detection
  • Efficient processing: Optimized volume analysis algorithm for fast processing
  • Accurate analysis: Determines silence status using professional volume detection technology
  • Supports local files, HTTP/HTTPS URLs, and TOS/S3 storage

Audio file size calculation

• Offline

• Audio file metadata analysis processor, precisely calculates file size
• Key features
  • Precisely calculates the byte size of audio files
  • Supports local files and TOS storage
  • Lightweight and efficient, suitable for batch processing

Multilingual CTC alignment

• Offline

• "Multilingual CTC alignment operator" is a multilingual audio CTC alignment operator based on the MMS Forced Aligner model (CTC stands for Connectionist Temporal Classification), designed to precisely align audio content with the corresponding text script along the time dimension.
• Key features
  • Supports multilingual input: Native support for Chinese and English text alignment, meeting mainstream language processing needs.
  • Supports multiple input channels: Seamlessly processes audio data from local file paths, HTTP URLs, TOS/S3 object storage, or raw byte streams.
  • Automated preprocessing: Built-in text preprocessing workflows for Chinese and English, including case conversion, number expansion, pinyin conversion, and romanization, to meet model input requirements. In addition, built-in audio decoding and preprocessing capabilities automatically resample input audio to 16kHz mono format, simplifying the invocation process.

Audio filter processing

• Offline

• Audio filter processor, flexible audio effects application based on FFmpeg.
• Key features:
  • Applies common audio filters (volume, highpass, lowpass, bass, treble, aecho, and more) using FFmpeg
  • Supports automatic download and processing for local paths and TOS/S3 remote paths
  • Supports uploading processed results to TOS or returning binary results

Audio format conversion (offline)

• Offline

• Audio format conversion processor
• Key features:
  • Supports audio format conversion (WAV, MP3, FLAC)
  • Automatically selects the appropriate encoder
  • Supports custom ffmpeg parameters via extra_params
  • Supports local files, HTTP/HTTPS URLs, and TOS/S3 storage

Audio duration calculation

• Offline

• Audio duration analysis processor, precisely calculates audio content duration
• Key features
  • Precisely calculates audio duration (second-level accuracy)
  • Supports local files, TOS storage, and binary
  • Based on the professional audio processing library librosa

Audio segment splitting (timestamp)

• Offline

• Audio timestamp segmentation processor, supports precise segment extraction.
• Key features
  • Precisely segments audio based on timestamps
  • Supports multiple input formats:
  • Local file path
  • TOS storage path
  • Binary data stream
  • Supports segmented binary output or TOS storage

Audio to MP3

• Offline

• Audio format conversion processor for converting various audio formats to MP3
• Key features:
  • Supports conversion of multiple audio formats to MP3
  • Customizable audio quality and encoding parameters
  • Supports fine control of audio sample rate and bitrate
  • Automatically selects the first audio track
  • Supports local files, HTTP/HTTPS URLs, and TOS/S3 storage

Audio signal-to-noise ratio calculation

• Offline

• Audio signal-to-noise ratio (SNR) calculator, performs signal-noise separation based on non-negative matrix factorization (NMF)
• Key features:
  • Performs STFT spectral analysis on audio
  • Uses NMF to decompose the spectrum into signal and noise components
  • Calculates SNR (in dB) based on the energy ratio estimated from the reconstructed time-domain signal and noise

Audio concatenation

• Offline

• Audio concatenation processor, supports concatenating multiple audio files into a single audio segment
• Key features
  • Supports concatenating multiple audio files into one audio file
  • Supports multiple input formats:
  • Local file path
  • TOS/S3 storage path
  • HTTP/HTTPS path
  • Supports output to a specified path
  • Automatically handles audio files with different sample rates and channels
  • Automatically selects the appropriate encoder based on the output format

Fast audio concatenation (same source)

• Offline

• Fast audio concatenation processor (homogeneous audio)
• Key features
  • Uses concat demuxer for fast concatenation of homogeneous audio (no re-encoding required)
  • Applicable to audio files with identical format, encoding, and sample rate
  • High speed, no quality loss
  • Supports multiple input formats:
  • Local file path
  • TOS/S3 storage path
  • HTTP/HTTPS path
  • Supports output to a specified path

Audio metadata extraction

• Offline

• Audio metadata extraction operator
• Features:
  • Supports audio file formats
  • Extracts complete audio metadata information, including:
  • Basic information: duration, format, bitrate
  • Audio information: sample rate, channel count, encoder
  • Uses ffprobe tool for metadata extraction
  • Supports local files, remote files (TOS/S3), HTTP/HTTPS links

Audio denoising (MossFormer2_SE_48K)

• Offline

• Performs audio denoising using MossFormer2_SE_48K
• Key features
  • Performs audio denoising using MossFormer2_SE_48K
  • Supports local files and TOS storage (tos, s3, http, https, and other schemas)

Audio segment splitting (duration)

• Offline

• Audio segmentation by duration, supports splitting audio clips by fixed duration
• Key features
  • Split audio into multiple segments by specified duration
  • Supports multiple input formats:
  • Local file path
  • TOS/S3 storage path
  • Binary data stream
  • Supports segment binary output or TOS storage
  • Supports minimum segment duration filtering

Audio quality scoring (DNSMOS)

• Offline

• Audio quality scoring module - uses the DNSMOS model to evaluate audio quality
• Key features
  • Uses the DNSMOS (Deep Noise Suppression Mean Opinion Score) model for audio quality evaluation
  • Provides scores in three dimensions: overall quality (OVRL), signal quality (SIG), and background noise (BAK)
  • The score ranges from 1 to 5, with higher scores indicating better audio quality
  • Supports automatic sample rate conversion and audio preprocessing
  • Applicable to scenarios such as speech quality evaluation, audio filtering, and quality control

Audio extraction and splitting (LAS)

• Offline

• Audio extraction and segmentation (LAS) operator, based on the LAS audio extraction and segmentation service, extracts audio from audio/video and segments it according to rules
• Key features:
  • Extract audio from audio or video
  • Supports segmenting into multiple clips by duration
  • Supports output format and path templates
  • Supports batch concurrent processing

Speech quality scoring (SpeechScore)

• Offline

• Audio speech quality evaluation operator - calculates multiple audio quality metrics based on ClearerVoice-Studio/speechscore
• Key features
  • Unified encapsulation of SpeechScore, supports both non-reference and reference metrics
  • The set of metrics to compute can be configured via the metrics parameter; by default, all supported metrics are computed
  • Supports local/TOS/HTTP/S3 path input; the reference audio column reference_audio_paths is optional
  • Output is a structured result, including common metrics such as BSSEval, DNSMOS, NISQA, PESQ, STOI, SNR, and SRMR

Audio format conversion (LAS)

• Offline

• Audio format conversion (LAS) operator, based on the LAS audio format conversion service, converts input audio to the specified format
• Key features
  • Supports conversion between multiple audio formats
  • Supports specifying output format and FFmpeg parameters
  • Supports batch concurrent processing
  • Applicable to audio format standardization and transcoding scenarios

Audio multilingual recognition (whisper)

• Offline

• AudioLidWhisper audio language identification processor
• Key features
  • Audio decoding: Supports multiple audio formats → 16kHz waveform data
  • Language detection: Uses the Whisper-large model to identify dozens of language codes
  • Full name mapping: Automatically converts language codes to full language names

Speech-to-Text (LAS ASR Service)

• Offline

• Speech recognition module – Recording transcription solution based on LAS ASR service
• Key features
  • Integrates the Volcengine LAS ASR interface
  • Supports automatic sentence segmentation, numeral normalization, and optional speaker or channel separation
  • Processes multiple audio files concurrently, providing both structured JSON and readable text outputs
  • Suitable for transcribing recordings up to 2 hours long, supporting advanced features such as punctuation restoration, intelligent sentence segmentation, and speaker separation.

Speech-to-text (FireRed)

• Offline

• Speech recognition module – Multilingual speech-to-text solution based on the FireRed ASR model
• Key features
  • Multilingual recognition: Supports Chinese, English, and Chinese dialects
  • Multiple model selection: Supports various models, including AED models and LLM models
  • Audio type: Supports mono, 16kHz sampling rate WAV audio files

Speech-to-text (Whisper model series)

• Offline

• Speech recognition module – Multilingual speech-to-text solution based on the Whisper model
• Key features
  • Multilingual recognition: Supports major languages such as Chinese and English
  • Speech translation: Can translate recognition results into English

Language identification and ASR (Whisper series models)

• Offline

• Language identification + speech recognition module – Multilingual LID + ASR solution based on the Whisper model
• Key features
  • Multilingual recognition: Supports hundreds of languages including Chinese and English
  • Language identification (LID): Outputs language tags (such as en, zh) while recognizing text
  • Punctuation restoration: Optional Chinese and English punctuation restoration to improve text readability

Speech endpoint detection (Silero model)

• Offline

• Voice endpoint detection module – Efficient audio segmentation solution based on Silero VAD
• Key features
  • Voice endpoint detection: Automatically identifies the start and end times of speech segments in audio, enabling precise segmentation of speech and silence
  • Batch processing: Supports efficient endpoint detection for large-scale audio data
  • Multi-format input: Compatible with various audio input methods, including raw binary, Base64 encoding, TOS/HTTP links, and more
  • GPU acceleration: Supports high-performance inference in GPU environments
  • High versatility: Delivers excellent performance when processing audio from various domains, with different background noise and quality levels

Speech endpoint detection (FSMN model)

• Offline

• Speech endpoint detection module – Efficient audio segmentation solution based on FSMN VAD
• Key features
  • Speech endpoint detection: Automatically identifies the start and end times of speech segments in audio, enabling precise separation of speech and silence
  • Batch processing: Supports efficient endpoint detection for large volumes of audio data
  • Multi-format input: Compatible with multiple audio input methods, including raw binary, Base64 encoding, TOS/HTTP links, and more
  • GPU acceleration: Supports high-performance inference in GPU environments

Speaker verification (ERes2Net)

• Offline

• AudioSpeakerVerificationEres2net audio speaker verification processor
• Key features
  • Speaker verification: Determines whether two audio segments are from the same speaker and outputs a similarity score
  • Multi-source audio support: Supports various audio input types, including local files, URLs, object storage, base64, binary, and more
  • Batch processing: Supports efficient verification and exception handling for batches of audio pairs

Text-to-speech (Doubao voice large model)

• Offline

• Speech synthesis module – Text-to-audio solution based on Doubao speech large model
• Key features
  • Integrates the speech synthesis interface of Volcano Engine large model (volc.tts)
  • Supports various parameter configurations, such as timbre, emotion, encoding format, speech rate, sampling rate, and more
  • Processes multiple text inputs concurrently and outputs Base64 encoded audio and the original response
  • Suitable for scenarios such as voice broadcasting, virtual human voice generation, hearing content production, and more

Audio format standardization

• Offline

• Audio normalization module – Standardizes audio to specified formats (sampling rate, channels, loudness, and more)
• Key features
  • Supports sampling rate resampling
  • Supports channel unification (for example, converting to mono)
  • Supports loudness normalization (target dBFS, with limited gain range)
  • Default input and output audio are in bytes format

Multilingual audio classification

• Offline

• Speech classification module – A multilingual speech classification solution based on the BEATs model,​AudioBeatsClassifier is an audio classification operator based on the BEATs model, designed to identify the main sound events in audio and return the Top K classification labels with the highest probability.
• Key features
  • Supports multiple audio types: Automatically processes various types of audio and identifies 527 categories of sounds defined by Google AudioSet, such as "music", "speech", "alarm sounds", or "animal sounds" and many others.
  • Supports multiple input channels: Seamlessly handles audio data from local file paths, HTTP URLs, TOS/S3 object storage, or raw byte streams.
  • Automated preprocessing: Built-in audio decoding and preprocessing capabilities automatically resample input audio to a 16 kHz mono format, simplifying the invocation process.

Audio risk identification

• Offline

• Audio risk identification
• Operator prerequisites: Activate the Business Risk Identification product – Audio Risk Identification – Audio on-demand service.

Generate ordinary pre-signed URLs for TOS

• Offline

• TOS pre-signed URL generation processor
• Key features:
  • Signature generation mechanism: Generates time-limited pre-signed URLs based on the Volcano Engine TOS SDK.
  • URL schema processing:
    • Natively supports signature conversion for tos/s3 protocol paths.
    • Automatically skips paths containing http/https protocols.
  • Security control:
    • Configurable signature validity period (default: 3600 seconds).

URL ratio calculator

• Offline

• URL ratio calculator – Text feature extraction based on URL character ratio
• Key features
  • URL ratio calculation: Accurately calculates the proportion of URL characters in text.
  • Multi-protocol support: Supports multiple URL protocols such as HTTP and HTTPS.
  • Intelligent recognition: Uses regular expressions to accurately identify URL formats.

Chinese simplified-traditional conversion

• Offline

• Chinese simplified-traditional conversion operator based on OpenCC
• Key features
  • Multi-directional conversion: Supports multiple conversion directions including simplified-traditional, Taiwan traditional, and Hong Kong traditional.
  • Mixed text processing: Correctly handles mixed Chinese and English content, converting only the Chinese part.
  • Efficient batch processing: Supports fast conversion of large volumes of text.

Text safety scorer

• Offline

• Text safety scorer - Safety evaluation based on ShieldLM-6B-chatglm3
• Key features
  • Multilingual support: Supports safety evaluation for Chinese and English text
  • Three-class evaluation: Outputs probabilities for safe, unsafe, and controversial categories
  • Batch processing: Supports batch text safety evaluation to improve processing efficiency

Maximum English word length calculator

• Offline

• Maximum English word length calculator - Calculates the maximum length of English words in text
• Key features
  • English word recognition: Uses regular expressions to identify English words in text
  • Maximum length calculation: Calculates the maximum length among all English words
  • Batch processing: Supports batch calculation of the maximum English word length in text

Word repetition ratio calculator

• Offline

• Word repetition ratio calculator - Text feature extraction based on N-gram phrase repetition ratio
• Key features
  • Word repetition ratio calculation: Accurately calculates the ratio of repeated phrases in text
  • Bilingual support: Supports Chinese and English word segmentation
  • Flexible configuration: Supports calculation of N-gram phrases of different lengths

Duplicate row calculator

• Offline

• Duplicate line calculator - Calculates the ratio of duplicate lines in text
• Key features
  • Duplicate line detection: Automatically identifies duplicate lines in text
  • Ratio calculation: Calculates the ratio of duplicate lines to original lines
  • Quality assessment: Evaluates the degree of repetition and quality of the text

Text chunking (semantic-based)

• Offline

• Semantic text chunk processor - Intelligent text segmentation solution based on sentence similarity
• Key features
  • Semantic chunking strategy
    • Intelligent segmentation based on sentence similarity
    • Combines semantic and syntactic rules
    • Overlap optimization to maintain contextual coherence
  • Supports mixed Chinese and English text
    • Supports Chinese and English characters and punctuation
    • Intelligent sentence boundary detection

Copyright statement removal

• Offline

• Copyright statement remover - Removes copyright-related text from content
• Key features
  • Copyright statement detection: Automatically identifies copyright statement comments in code
  • Intelligent cleanup: Precisely removes content based on copyright statement features
  • Multi-format support: Supports block comment and line comment formats

CommonCrawl WARC file content extraction

• Offline

• CommonCrawl web content extractor, supports multiple parsing strategies
• Key features:
  • Multi-parser support: trafilatura/justext/goose3
  • Supports local and remote WARC files
  • Supports binary data and base64 encoding
  • Batch processing of WARC files to extract web page main content
  • Intelligent content extraction, filters advertisements and navigation elements

Text length calculator

• Offline

• Text length calculator - calculates the character length of text
• Key features:
  • Text length calculation: Calculates the number of characters in input text
  • Batch processing: Supports batch text length calculation
  • Numeric output: Returns the length as an integer

Character ratio calculator

• Offline

• Character proportion calculator - text feature extraction based on the proportion of alphabetic and numeric characters
• Key features:
  • Character proportion calculation: Accurately calculates the proportion of alphabetic and numeric characters in text
  • Tokenization mode support: Allows calculation based on tokenization or character level
  • Multi-language support: Supports character recognition for English, Chinese, Japanese, Korean, and many other languages

Special character ratio calculator

• Offline

• Special character proportion calculator - text feature extraction based on the proportion of special characters
• Key features:
  • Special character proportion calculation: Accurately calculates the proportion of special characters in text
  • Multi-granularity support: Allows calculation based on different types of special characters
  • Flexible configuration: Supports calculating the proportion of all special characters or specific types

Text chunk segmentation (based on sentence structure)

• Offline

• Multi-format text chunk processor - structured parsing and intelligent segmentation solution
• Key features:
  • Supported formats
    • Plain text: Semantic chunking based on paragraphs and punctuation
    • Markdown: Preserves document structure, filters image links
    • HTML: Extracts the main content, preserves section structure
  • Intelligent chunking strategy
    • Combines semantic and syntactic rules
    • Overlap optimization to maintain contextual coherence

Remove text links

• Offline

• Hyperlink removal operator - regex replacement for text links
• Key features:
  • Identifies common hyperlink formats such as protocol links, www links, domain and path
  • Replace matched hyperlinks with the specified string (repl), default replacement is ""
  • Process strings in batches; for exceptions, return None

Perplexity calculator

• Offline

• Perplexity calculation operator – a text quality evaluation solution based on language models
• Key features
  • Language model evaluation
    • Calculate text perplexity using the KenLM language model
    • Supports quality evaluation for both Chinese and English texts
    • Provides text readability metrics
  • Quality evaluation
    • The lower the perplexity, the higher the text quality
    • Suitable for text quality filtering and evaluation

Whitespace character normalizer

• Offline

• Whitespace character normalizer – replaces various types of whitespace in text with standard spaces
• Key features
  • Whitespace character recognition: Automatically identifies various Unicode whitespace characters
  • Normalization: Replaces all whitespace characters with standard spaces

Bullet-point line ratio calculator

• Offline

• Bullet line ratio calculator – calculates the proportion of bullet lines in text
• Key features
  • Bullet detection: Automatically identifies lines in text that start with bullet points
  • Ratio calculation: Calculates the ratio of bullet lines to total lines

MD5 hash calculation

• Offline

• MD5 hash value calculator – calculates the MD5 fingerprint of text
• Key features
  • Generates the corresponding MD5 hash value for each text entry
  • Outputs a fixed-length (32-character lowercase hexadecimal) fingerprint
  • Supports batch processing

Multilingual audio classification

• Offline

• Speech classification module – a multilingual speech classification solution based on the BEATs model, AudioBeatsClassifier is an audio classification operator based on the BEATs model, used to identify major sound events in audio and return the Top K classification labels with the highest probability.
• Key features
  • Supports multiple audio types: Automatically processes various types of audio and identifies 527 sound categories defined by Google AudioSet, such as "music", "speech", "alarm", or "animal sounds" and more.
  • Supports multiple input channels: Seamlessly processes audio data from local file paths, HTTP URLs, TOS/S3 object storage, or raw byte streams.
  • Automated preprocessing: Built-in audio decoding and preprocessing capabilities automatically resample input audio to a 16kHz mono format, simplifying the invocation process.

Multilingual text quality scoring

• Offline

• Multilingual text quality scoring operator – Multilingual text quality assessment based on the E5 model
• Key features
  • Multilingual support: Uses the multilingual-e5 model to support text quality scoring in multiple languages
  • Deep learning evaluation: Quality assessment based on the E5 model with Transformer architecture
  • GPU acceleration: Supports GPU inference acceleration to improve processing efficiency
  • Batch processing: Supports batch processing of text to optimize inference performance

English text quality scoring

• Offline

• English text quality scoring operator – Text quality assessment based on FastText
• Key features
  • Quality scoring: Uses the FastText model to score English text quality, with a preference for scientific knowledge; only supports CPU environments.
  • Batch processing: Supports batch processing of text to improve processing efficiency

HTML tag removal

• Offline

• HTML document purification processor – Multi-structure parsing and intelligent cleaning solution
• Key features
  • Multi-structure parsing
    • Title extraction: Automatically identifies <h1>-<h6> tags
    • Main content extraction: Intelligently identifies the main content of the article
    • Redundancy filtering: Removes non-text tags such as <script>/<style>
  • Intelligent processing
    • Fault tolerance mechanism: Supports parsing of incomplete HTML fragments
    • Format retention: Maintains paragraph structure and line break logic in the text

Email address cleaning

• Offline

• Regex-based email address cleaning operator.
• Key features
  • Multi-scenario support: Built-in general matching patterns, while allowing injection of custom regular expressions.
  • Controllable replacement: Configurable replacement strings for desensitization or placeholder filling.
  • Batch processing compatibility: Supports batch processing of arrays

Specific character replacement

• Offline

• Regular expression batch replacement processor
• Key features
  • Dual-mode replacement mechanism:
    • Exact string match replacement
    • Regular expression pattern match replacement
  • Batch processing capability:
    • Supports parallel processing of multiple pattern-replacement pairs
  • Fault tolerance mechanism:
    • Skips abnormal patterns and records detailed logs

Text embedding (BGE model)

• Offline

• Based on the BGE series of text embedding models, supports dense vector generation
• Key features
  • Multi-model support:
    • Supports BGE-M3, BGE-Large-zh-v1.5, BGE-Large-en-v1.5, BGE-Multilingual-Gemma2, and other models
    • Each model has different input prompts; users can select the appropriate model as needed
  • Hardware acceleration: Supports FP16 quantization and GPU parallel computing

Text sparse & dense embedding (BGE model)

• Offline

• Based on the BGE-M3 text embedding model, supports generation of sparse, dense, and token-level vectors
• Key features
  • Multi-granularity embedding output:
    • Sparse vector: term weight representation, suitable for keyword retrieval
    • Dense vector: 1024-dimensional semantic representation, suitable for semantic similarity calculation
    • Token vector: fine-grained contextual representation
  • Hardware acceleration: Supports FP16 quantization and GPU parallel computing

Text content risk identification

• Offline

• Text content risk identification
• Operator prerequisite: Enable the Business Risk Identification product - Text Risk Identification

Multilingual text translation

• Offline

• Seed-X multilingual text translation model – cross-language text translation key features
• Key features
  • Multilingual intelligent translation
  • Supports text conversion between multiple languages; source and target languages can be customized via the source_language and target_language parameters
  • Based on the Seed-X-Instruct-7B and Seed-X-PPO-7B models, provides high-quality translation results
  • For supported languages, see: https://huggingface.co/ByteDance-Seed/Seed-X-Instruct-7B
  • For general scenarios, it is recommended to use the Seed-X-PPO-7B model, which offers better translation quality
  • Flexible configuration and optimization
  • Supports multiple computation precision options (such as bfloat16), adaptable to different performance requirements
  • Integrates tensor parallel processing and prefix caching technologies to significantly improve inference efficiency
  • Supports automatic or manual device allocation, fully compatible with single-GPU and multi-GPU environments
  • Resource usage
  • It is recommended to use a GPU with at least 24GB of memory

Xlsx document parsing

• Offline

• Excel spreadsheet parsing processor, supports multi-format output and structured data extraction
• Key features:
  • Supports xlsx/xls format parsing
  • Outputs in Markdown or HTML format
  • Preserves table structure and data relationships
  • Supports multi-worksheet processing
  • Provides TOS storage options

Intelligent PDF document parsing

• Offline

• Intelligent PDF document parsing
• Operator prerequisites: Enable the Visual Intelligence product - Text Recognition - Intelligent Document Parsing service

PPT file conversion

• Offline

• First release of the "PPT file conversion" operator, serving as a PPT conversion processor, supporting PPT/PPTX to PDF and per-page PNG images
• Main features:
  • Supports converting PPT/PPTX documents to PDF files
  • Supports converting PPT/PPTX to per-page PNG images
  • Automatically handles downloads and uploads for local and remote paths (TOS/HTTP and more)
  • Provides conversion timeout control and basic failure handling

Doc format conversion

• Offline

• Document format conversion processor, supports conversion between multiple office document formats
• Key features:
  • Supports conversion from doc/docx to multiple formats
  • Uses LibreOffice for high-quality conversion
  • Supports local and cloud storage paths
  • Provides conversion timeout control
  • Automatically handles file upload and download

Timestamp segment merging

• Offline

• Timestamp merge operator - normalization, merging, and splitting of second-level intervals
• Core capabilities
  • Normalization and sorting: Standardizes input format to (start, end) floating-point seconds and verifies validity.
  • Pre-merge small gaps: Merges overlapping or adjacent segments with gaps less than or equal to the threshold (pre_merge_gap_seconds).
  • Maximum silence priority splitting: Splits at the longest silence first, ensuring each segment span does not exceed max_span_seconds.
  • Forced chunking (optional): Splits overly long segments by fixed windows to ensure each segment length does not exceed the upper limit (enforce_chunking).

Image resampling

• Offline

• Image resampling processor, supports multiple interpolation algorithms and output formats.
• Key features
  • Provides four professional-grade interpolation algorithms:
    • Nearest neighbor interpolation (nearest) - fastest speed, suitable for pixel art
    • Bilinear interpolation (bilinear) - balances speed and quality
    • Bicubic interpolation (bicubic) - high-precision smoothing
    • Lanczos interpolation (lanczos) - best anti-aliasing, suitable for photos
  • Multi-format input support:
    • URL address (image_url)
    • Base64 encoding (image_base64)
    • Binary stream (image_binary)
  • Dual output modes:
    • Direct Base64 encoding output
    • TOS persistent storage

Image aesthetic scoring

• Offline

• Image aesthetics scoring processor, intelligently evaluates the aesthetic quality and composition of images
• Key features:
  • Aesthetic scoring: Performs professional aesthetic quality assessment on input images
  • Composition analysis: Analyzes image composition based on visual perception theory
  • Standardized output: Normalizes score range to 0-1 for easier subsequent processing
  • Batch processing: Supports efficient batch image scoring
  • Multi-format support: Compatible with multiple image input formats

Image hash value

• Offline

• Image hash calculation operator: Supports three input formats—URL, Base64, and binary—and outputs unified hexadecimal and binary hashes.
• Key features
  • Supports five hash methods: ahash, dhash, phash, whash, md5
  • Selectable input types: image_url, image_base64, image_binary
  • Supports batch processing

Image safety detection

• Offline

• Image safety (NSFW) detector—supports multi-source input and batch inference
• Key features
  • Uses pretrained image classification models for NSFW probability detection, outputs NSFW confidence scores for each image
  • Supports multiple input sources: URL address (image_url), Base64 encoding (image_base64), and binary stream (image_binary)
  • Batch processing: Configure batch_size for batch inference to improve throughput performance

Image face blur

• Offline

• First release of the "image face blur" operator.
• Main features:
  • Automatically detects faces in images and applies blur processing
  • Supports multiple blur types (mean, box, Gaussian)
  • Supports multiple input methods including URL, local path, Base64, and binary
  • Optional output of blurred image in Base6

Image face detection

• Offline

• First release of image face detection operator
• Main features:
  • Automatically detects faces in images and returns a list of bounding rectangles (x, y, w, h)
  • Supports multiple input methods including URL, local path, Base64, and binary
  • Outputs a list of face bounding rectangles
  • Runs on both CPU and GPU environments

Image embedding (ViT series models)

• Offline

• ViT image semantic embedding processor, suitable for image similarity search, content retrieval, and related scenarios.
• Key features
  • Multi-model support:
    • Google official ViT model
    • Meta DINOv2 vision model
  • Feature extraction modes:
    • CLS Token embedding vector
    • Global average pooling
  • Input format compatibility:
    • URL
    • Base64 encoding
    • Binary stream
  • Performance optimization:
    • FP16 inference acceleration
    • Multi-GPU parallel computation

Image OCR (EasyOCR)

• Offline

• Multilingual OCR recognition module based on EasyOCR, supports text detection and recognition in mixed Chinese and English scenarios.
• Key features
  • Supports recognition of more than 100 languages (requires configuration of corresponding language models)
  • Input format compatibility:
    • TOS URL
    • Base64 encoding
    • Binary stream
    • Numpy array
  • Performance optimization:
    • GPU-accelerated inference
    • Model quantization (enabled by default)
    • Batch processing optimization

Video safety detection

• Offline

• Video safety detector – multi-source input, unified frame sampling, and batch inference
• Key features
  • Based on a pre-trained image classification model, performs NSFW probability detection on frames sampled from video and outputs the final score according to aggregation strategy.
  • Supports multiple video input sources:
  • URL path (video_url)
  • Base64 encoding (video_base64)
  • Binary stream (video_binary)
  • Supports multiple sampling methods (via VideoFrameSampler):
  • by_count_uniform / by_interval_time / by_interval_frames / by_fps / by_timestamps
  • Batch inference:
  • Batch size can be controlled via batch_size to improve throughput performance.

General video format conversion

• Offline

• General video format conversion processor
• Key features:
  • Supports conversion between multiple video formats
  • Automatically selects the appropriate encoder
  • Supports custom ffmpeg parameters via extra_params

Video frame sampling

• Offline

• Image frame sampling processor for videos, supports multiple sampling modes and time range control.
• Key features:
  • Multiple sampling methods:
  • by_count_uniform: Uniformly sample K frames within the time range
  • by_interval_time: Sample at intervals of Δt seconds
  • by_interval_frames: Sample at intervals of N decoded frames
  • by_fps: Sample at target fps
  • by_timestamps: Sample according to a given list of timestamps
  • Supports start and end time range restrictions and whether to include the last frame
  • Outputs original frames (array), base64 encoding, timestamps, frame indices, and optional TOS storage path
  • Two input sources: path input or binary input, compatible with remote URI (via run_on_local_path)

Video metadata extraction

• Offline

• Video metadata extraction operator
• Key features:
  • Supports multiple video file formats
  • Extracts complete video metadata information
  • Uses ffprobe tool for metadata extraction
  • Supports local files, remote files (TOS/S3), HTTP/HTTPS links

Video resolution adjustment (offline)

• Offline

• Video resolution adjustment
• Key features:
  • Intelligently adjusts resolution within the specified range
  • Supports multiple aspect ratio preservation strategies
  • Supports control of video quality and encoding parameters
  • Keeps the audio stream unaffected
  • Supports path input, binary input, and TOS output

Remove audio tracks from video

• Offline

• Video audio track removal processor
• Features:
  • Removes video audio tracks by invoking ffmpeg as a subprocess
  • Preserves original video encoding and quality (no re-encoding required)
  • Automatically downloads remote files / uploads the results
  • Supports timeout control
  • Supports multiple video formats

Video and audio detection

• Offline

• Video audio detection processor
• Features:
  • Uses ffprobe to detect whether an audio stream exists in the video
  • Automatically downloads remote files
  • Supports timeout control
  • Supports multiple video formats
  • Returns a boolean value indicating whether audio exists

Video segment splitting (keyframe)

• Offline

• Video keyframe segmentation processor that supports intelligent segment splitting
• Key features:
  • Multi-algorithm keyframe detection:
  • I-frame: Detection based on I-frames (recommended)
  • Difference: Pixel difference detection
  • Histogram: Histogram difference detection
  • Supports segment binary output or TOS storage
  • Provides timestamp positioning functionality

Video segment splitting (timestamp)

• Offline

• Video timestamp segmentation processor that supports splitting by specified time range
• Key features:
  • Splits video by given timestamp intervals
  • Supports segment binary output or TOS storage
  • Provides automatic format inference and customization

Video conversion to MP4

• Offline

• Video format conversion processor that converts various video formats to MP4
• Key features:
  • Supports converting multiple video formats to MP4
  • Automatically selects the first audio track
  • Supports customization of video quality and encoding parameters
  • Supports limiting video height and scaling
  • Provides fine-grained control of audio encoding parameters

Video segment splitting (duration)

• Offline

• Video segment splitting processor that intelligently splits by fixed duration
• Key features:
  • Splits video by fixed duration
  • Supports filtering by minimum remaining segment duration
  • Supports segment binary output or TOS storage
  • Provides automatic format inference and customization

Adaptive video compression

• Offline

• Video adaptive compression
• Key features:
  • Compresses video adaptively according to the target file size
  • Multi-level compression strategy: frame rate adjustment -> resolution adjustment -> bitrate control
  • Compress file size as much as possible while maintaining video quality
  • Supports CPU and GPU encoding
  • Supports path input, binary input, and TOS output

Video and audio extraction

• Offline

• Video audio extraction processor, supports multi-stream separation
• Key features:
  • Supports extracting audio streams from videos via multiple paths or binary input, including local, TOS, HTTP, and more
  • Supports selection of multiple audio streams, extracting only the first stream or all streams
  • Supports outputting audio to TOS, returning audio as binary, and specifying the sampling rate
  • Supports extracting audio from a specified time interval (start_second, end_second)
  • All output audio is in mp3 format

Video keyframe extraction

• Offline

• Video keyframe extraction processor that supports dynamic detection with multiple algorithms.
• Key features:
  • Multi-algorithm support:
    • Pixel difference method (difference)
    • Optical flow method (optical_flow)
    • Histogram method (histogram)
    • I-frame identification (I_frame)
  • Supports custom thresholds and control over the number of keyframes
  • Provides timestamp positioning functionality
  • Supports multiple output formats and storage options

Video face blurring

• Offline

• First release of video face blurring operator
• Main features:
  • Automatically detects faces in video frames and applies blurring
  • Supports multiple blur types (mean, box, Gaussian)
  • Supports local path, remote URL/TOS path, and binary/BASE64 video input
  • Outputs a unified path for the blurred video (re-encodes and outputs even if no face is detected)
  • Runs in both CPU and GPU environments

Video motion score calculation

• Offline

• Video motion score calculation
• Key features:
  • Multi-algorithm support: Supports various optical flow calculation algorithms to suit different scenarios and performance requirements:
  • Farneback: Classic dense optical flow algorithm, balancing accuracy and speed.
  • TV-L1: Optical flow algorithm based on total variation, robust to illumination changes.
  • DIS (Dense Inverse Search): Fast optical flow algorithm, supports multiple presets (ULTRAFAST, FAST, MEDIUM, accurate).
  • MEMFOF: Deep learning-based optical flow estimation algorithm, highest accuracy.
  • Multi-dimensional metrics: Provides a rich variety of motion scoring metrics:
  • Basic metrics: mean, median, 95th percentile.
  • Adaptive metric: A dynamic score normalized based on frame size to eliminate the impact of resolution.
  • Density metric: A density score reflecting the proportion of motion regions.
  • Intelligent sampling: Supports proportional sampling and intelligent downsampling to balance computational overhead.
  • High-performance computing: Supports CUDA GPU acceleration and multi-process parallel processing.

Video quality scoring

• Offline

• First release of the video quality scoring operator and video quality scoring processor, which scores sampled frames based on CLIP-IQA and aggregates them into a video quality score.
• Key features:
  • Video quality scoring: Scores sampled frames and aggregates the results.
  • Multi-source support: Supports URL/TOS/local path/binary input.
  • Configurable sampling strategies: Sampling by frame count, time interval, FPS, and more.
  • Configurable aggregation methods: avg / max / min.

Video risk identification

• Offline

• Video risk identification is performed using the business risk identification product—video risk identification—video on demand service.

Image content understanding (Qwen VL series models, supports large-scale models)

• Offline

• Qwen2.5-VL multimodal image understanding model, supports visual semantic interpretation and natural language description generation, suitable for various image analysis scenarios.
• Main features
  • Multimodal temporal modeling: Supports three image input formats (URL, Base64 encoding, binary stream) for flexible adaptation to different data sources.
  • Conversational prompt support: Customize generation direction via the prompt parameter to meet diverse business needs.
  • Efficient inference: Integrates the VLLM inference engine and supports bfloat16, float16, and float32 precision, fully utilizing GPU computing power.
  • It is recommended to use a GPU with 48 GB or more of VRAM.

Image content understanding (Qwen VL series models)

• Offline

• Qwen2.5-VL multimodal image understanding model – visual semantic interpretation and natural language description generation
• Key features
  • Multimodal input processing
  • Supports three image formats: URL, Base64 encoding, and binary stream
  • Joint modeling of vision and language
  • Accurately maps image content to semantic space
  • Conversational prompt support
  • Guides generation direction via the prompt parameter
  • Resource usage
  • It is recommended to use a GPU with 48 GB or more of VRAM

Image content understanding (LLaVA series models)

• Offline

• LLaVA multimodal image understanding model – visual-semantic parsing and natural language description generation
• Key features
  • Multimodal input processing
  • Supports three image formats: URL, Base64 encoding, and binary stream
  • Vision-language joint modeling
  • Accurately maps image content to semantic space
  • Conversational prompt support
  • Guides generation direction via the prompt parameter
  • Resource usage
  • It is recommended to use a GPU with 48 GB or more of VRAM

Video content understanding (Qwen VL series models)

• Offline

• Qwen2.5-VL multimodal video understanding model – temporal-semantic parsing and natural language description generation
• Key features
  • Multimodal temporal modeling
  • Supports three video formats: URL, Base64 encoding, and binary stream
  • Spatiotemporal joint modeling
  • Captures spatiotemporal features and semantic associations in video
  • Conversational prompt support
  • Guides generation direction via the prompt parameter
  • Resource usage
  • It is recommended to use a GPU with 48 GB or more of VRAM

Video content understanding (Qwen VL series models, supports large-scale models)

• Offline

• The Qwen2.5-VL multimodal video understanding model supports temporal-semantic parsing and natural language description generation, and is suitable for various video analysis scenarios.
• Main features
  • Multimodal temporal modeling: Supports three video input formats (URL, Base64 encoding, binary stream) for flexible adaptation to different data sources.
  • Spatiotemporal joint modeling: Automatically captures spatiotemporal features and semantic associations in video for more accurate video content understanding.
  • Conversational prompt support: Customize the generation direction via the prompt parameter to meet diverse business needs.
  • Efficient inference: Integrates the VLLM inference engine and supports three precisions—bfloat16, float16, and float32—to fully utilize GPU computing power.
  • It is recommended to use a GPU with 48 GB or more of VRAM

Image-text embedding (CLIP model)

• Offline

• CLIP (Contrastive Language-Image Pretraining) cross-modal embedding generator, achieves joint image-text embedding space mapping based on the CLIP model
• Key features
  • Unified multimodal encoding
  • Text encoding: Chinese text → 512/768/1024-dimensional semantic vector
  • Image encoding: Image → 512/768/1024-dimensional visual feature vector
  • Cross-modal similarity calculation
  • Supports calculating the correlation between image-text embedding vectors using cosine similarity or inner product

Audio content understanding (Qwen Omni model)

• Offline

• Qwen2.5-Omni multimodal audio understanding model - audio content analysis and natural language description generation: Key features
• Key features
  • Intelligent content understanding and description generation
  • Automatically generates detailed and accurate natural language descriptions based on audio signals, with support for customizing prompts via the prompt parameter
  • Supports audio formats such as mp3, aac, and m4a
  • Efficient model loading and inference optimization
  • Supports multiple computation precision options (bfloat16, float16, float32) to meet different performance requirements
  • Integrates FlashAttention2 acceleration technology to significantly improve inference efficiency
  • Supports automatic or manual device allocation, fully compatible with single-GPU and multi-GPU environments
  • Resource usage
  • It is recommended to use a GPU with 48GB or more of video memory

Audio understanding (Kimi-Audio series models)

• Offline

• Kimi-Audio multimodal audio understanding model - audio semantic analysis and natural language description generation
• Key features
  • Multimodal audio processing
  • Supports three audio data representations: URL, Base64 encoding, and binary stream
  • Audio-language joint modeling
  • Achieves precise mapping from audio content to semantic space
  • Conversational prompt support
  • Guides generation direction via the prompt parameter
  • Resource usage
  • It is recommended to use a GPU with 48GB or more of video memory