Lake AI Service
Lake AI Service
- Documentation
Lake AI ServiceTextText processingText chunk segmentation (based on sentence structure)
Copy page
Download PDFCopy pageDownload PDFDescription
Multi-format text chunk processor – structured parsing and intelligent segmentation solution
Key features
- Supported formats
Plain text: Semantic chunking based on paragraphs and punctuationMarkdown: Preserve document structure and filter image linksHTML: Extract main content and retain section structure- Intelligent chunking strategy
- Combines semantic and syntactic rules
- Overlap optimization to maintain contextual coherence
Technical implementation
- Parsing engine
- HTML：
BeautifulSoup - Markdown：
llama-index - Chunking algorithm
- Recursive semantic segmentation (sentence splitter)
Operator parameters
Input
Input column name | Description |
|---|---|
texts | Array of texts to be processed; each element must be a string. |
Output
Segmented text chunks, with elements of type List[str].
Parameters
If a parameter does not have a default value, it is required.
Parameter name | Type | Default value | Description |
|---|---|---|---|
content_type | str | text | Text type Description: The type of text to be segmented. Optional values: ["text", "md", "html"] Default value: "text" |
chunk_size | int | 500 | Chunk length Description: The length of each chunk, in characters. Default value: 500 |
chunk_overlap | int | 50 | Chunk overlap length Description: The maximum length of overlap between chunks when segmenting text. Default value: 50 |
Examples
The following code demonstrates how to use daft to run the operator and segment text based on sentence structure.
