Lake AI Service
Lake AI Service
- Documentation
Lake AI ServiceAudioAudio understanding Audio understanding (Kimi-Audio series models)
Copy page
Download PDFCopy pageDownload PDFDescription
Kimi-Audio multimodal audio understanding model – audio semantic parsing and natural language description generation
Key features
- Multimodal audio processing
- Supports three audio formats:
URL,Base64 encoding, andbinary stream - Audio-language joint modeling
- Accurate mapping of audio content to semantic space
- Dialog-style prompt support
- Guides generation direction via the
promptparameter - Resource usage
- It is recommended to use a GPU with 48GB or more VRAM
Scenario optimization
- Optimization for mixed Chinese and English scenarios, with a focus on enhancing Chinese semantic understanding
- Supports multiple audio understanding tasks: ASR, audio description, emotion recognition, and content analysis
Operator parameters
Input
Input column name | Description |
|---|---|
audios | This column contains audio data. Each element is either a string or binary. |
Output
The processed column, with each element being the result of audio understanding for each audio.
Parameters
If a parameter does not have a default value, it is required
Parameter name | Type | Default value | Description |
|---|---|---|---|
audio_src_type | str | audio_url | Audio data source type. Supports three formats. Optional values: ["audio_url", "audio_base64", "audio_binary"]. Default value: "audio_url" |
model_path | str | /opt/las/models | The absolute path for storing local model files. The default is the preset path within the container. This path must be modified when using a custom model. Default value: "/opt/las/models" |
model_name | str | moonshotai/Kimi-Audio-7B-Instruct | Supported audio language model versions. Currently, only Kimi-Audio series models are supported. Optional values: ["moonshotai/Kimi-Audio-7B-Instruct"]. Default value: "moonshotai/Kimi-Audio-7B-Instruct" |
prompt | str | Please analyze the content of this audio. | A prompt for understanding audio content. The model will generate the analysis result based on this prompt. Default value: "Please analyze the content of this audio." |
batch_size | int | 4 | The number of audios processed per batch. Larger values may increase VRAM consumption. Default value: 4 |
text_temperature | float | 0.0 | The temperature parameter for text generation, which controls the randomness of generated text. 0.0 indicates deterministic generation. Default value: 0.0 |
text_top_k | int | 5 | The number of top-k candidate words considered during text generation. Default value: 5 |
text_repetition_penalty | float | 1.0 | Text repetition penalty coefficient, used to avoid generating duplicate content. Default value: 1.0 |
text_repetition_window_size | int | 16 | Window size for text repetition detection. Default value: 16 |
rank | int or None | None | Specifies the GPU device number to use (effective in multi-GPU environments). For example: 0 indicates the first GPU, 1 indicates the second GPU. Default value: None |
Examples
The following code demonstrates how to use Daft to run the operator for audio understanding and content analysis.
