Lake AI Service
Lake AI Service
- Documentation
Lake AI ServiceTextText processingText chunking (semantic-based)
Copy page
Download PDFCopy pageDownload PDFDescription
Semantic text chunk processor – intelligent text chunking solution based on sentence similarity
Key features
- Semantic chunking strategy
- Intelligent chunking based on sentence similarity
- Combines semantic and syntactic rules
- Overlap optimization to maintain contextual coherence
- Supports mixed Chinese and English text
- Supports Chinese and English characters and punctuation
- Intelligent sentence boundary detection
Technical implementation
- Chunking engine
LlamaIndex: Uses semantic split node parserHuggingFace: Uses pretrained embedding model to calculate sentence similarity
- Chunking algorithm
- Breakpoint detection based on sentence similarity
- Supports custom breakpoint percentile threshold
Operator parameters
Input
Input column name | Note |
|---|---|
text | The text column to be processed. Each element is a string (Utf8). |
Output
The operator returns a list (List[str]), where each element is a text chunk split based on semantics.
If the input text is a null value (None), the operator returns an empty list ([]).
Parameters
If a parameter does not have a default value, it is required.
Parameter name | Type | Default value | Description |
|---|---|---|---|
chunk_size | int | 500 | The target length of each text chunk, in characters. |
chunk_overlap | int | 50 | When splitting text, the maximum allowed overlap length between adjacent chunks, in characters. |
model_path | str | "/opt/las/models" | Root directory path for storing embedding models. |
embedding_model_name | str | "BAAI/bge-m3" | Name of the embedding model used for semantic analysis. |
breakpoint_percentile_threshold | int | 80 | Defines the semantic breakpoint percentile threshold. The higher the value, the more conservative the chunking (that is, stronger semantic differences are required for a split to occur). |
Examples
The following code demonstrates how to use the ChunkTextSentenceSimilarity operator in Daft to perform semantic chunking on text.
