Natural language processing means that visual modeling supports processing data using various natural language processing methods, making subsequent visual queries and analysis more intuitive and convenient. This document will introduce the functions of natural language processing operators.

2.1 Generate sentence vectors #

Generate sentence vectors for text based on word vectors. You can select pre-trained word vectors or generate sentence vectors using custom-trained word vectors.
Field settings
Input column: Input column (input should be an array of strings; for example, the output of the tokenization operator can be used as input. The right port is optional and can accept user-trained word vectors).
Parameter settings
Output column: Output column

2.2 Tokenization #

The tokenization operator supports tokenizing Chinese documents. Unlike English, Chinese uses characters as units, and the meaning of a sentence is conveyed by the sequence of all its characters. Segmenting a sequence of Chinese characters into meaningful words is called Chinese word segmentation. For example: "我是一个学生", the tokenization result is: 我, 是, 一个, 学生. (Input is text of type string)
Field settings
Input column: Input column
Parameter settings
Output column name: Output column name

2.3 Remove stop words #

Remove specified stop words from a text segment. To save storage space and improve search efficiency, certain characters or words are automatically filtered out before or after processing natural language data (or text). These characters or words are called stop words. (Input must be an array of type string, usually following the tokenization operator)
Field settings
Input word column: Input word column
Parameter settings
Default stop word list language: The system provides stop word lists in three languages: Chinese, English, and French.