It supports importing raw files of output into the general knowledge base of the enterprise knowledge engine, and supports document deduplicate, knowledge segmentation and storage configuration when importing. This article shows you how to use the imported general knowledge base operator.

You need to create a knowledge base of general content types in the enterprise knowledge engine to implement subsequent knowledge imports, see for details Create a New Knowledge Base.

Create import general knowledge base operator #

After the knowledge segmentation operator, click the plus sign + to add the import knowledge base operator and configure the operator.

Configure parameters #

Import General Knowledge Base #

1. Configure basic information.

   Parameter	Description
   Knowledge base type	Fixed as a general knowledge base.
   Knowledge Base	Select the knowledge base for the generic content type to import. Support click to create , quickly jump to the knowledge base page to create a common content type of knowledge base.
   Knowledge base storage path	The storage path after knowledge is imported into the knowledge base.
   Knowledge data name	Select the specified output parameter as the knowledge data name from the upstream node.
   Knowledge address	Select the specified output parameter as the knowledge address from the upstream node.
   Document deduplicate	The MD5 method of verifying the file content identifies whether the file content is consistent, and only identical files will be determined to be duplicates. After turning on the switch, the imported document content is exactly the same as the existing knowledge in the knowledge base, and it will continue to be imported.

2. Click Next to configure segmentation and storage.

   Parameter	Description
   Document type	Select the document type for knowledge segmentation.
   Segmented mode	Automatic segmentation: Adopting a general slicing strategy, the model automatically recognizes and slices documents according to their document structure. Custom segmentation: Support custom length, rules, and scene slicing that is closer to business custom rules. tip The configuration items for different document types may be different, please refer to the actual page information for configuration.
   Parsing strategy	Configure image VLM on demand to understand, remove headers, footers, document OCR and other parsing strategies
   Maximum segmentation length	The maximum character length of a single slice, including English and Chinese characters, punctuation marks, spaces, and so on.
   Knowledge enhancement configuration	Invoking the large model to extract richer knowledge points for each segment to build an index can improve the effect of segment recall, but it will increase the processing time and resource consumption. Document summary: A paragraph summarizing the document's main theme and core conclusions for document-level coarse screening. Document Guide: A detailed summary of the core points of the document, such as background, findings, and conclusions, to facilitate quick grasp of the document content. It is mainly used by users to browse the key information of the document. Segment summary: A one-sentence condensed slice core is used for fast and accurate retrieval of slices, which can improve the accuracy of retrieval recall. Key questions for segmentation: high-frequency questions that can be answered by slicing, enhancing the semantic distance between Query and chunk, and improving the accuracy of retrieval recall. Segmented keywords: easy-to-retrieve labels for keyword filtering, which can improve the accuracy of retrieval recalls.
   Valid period	Set the valid period of knowledge. Support settings: Permanent validity Fixed deadline: You need to set a specific deadline and set whether knowledge will be automatically deleted after expiration. Dynamic field cutoff: You need to specify the upstream field as the cutoff field, and set whether the knowledge will be automatically deleted after expiration.
   Import concurrency	Sets the upper limit on the number of concurrent imports of knowledge.

Dependency configuration #

Configure the data dependencies of tasks, and support configuring upstream dependencies and self-dependencies.

• Upstream dependencies support system auto-configuration and custom configuration.
  • The automatic configuration of the system will automatically generate upstream data dependencies, and you can decide whether to use them according to the actual situation.
  • When selecting a custom configuration, you can manually add data dependencies.
    
    After entering the new dependency interface, you can search for the corresponding task or dataset to create a dependency, and the corresponding dependency will be generated after selecting the corresponding task. If you need to create a system custom dependency manually, you can contact technical support for assistance.
    
• Self-dependency:
  • When the switch is turned on, the current task is dependent on the operation of the task in the past, and supports setting the earliest data backtracking time configuration. If the task of the previous cycle fails, the task of the next cycle cannot be run.
  • When the switch is turned off, the current task does not depend on the operation of the task in the past, and even if the task of the previous cycle fails, the task of the next cycle can be run normally.
    

Advanced Configuration #

Further configuration of advanced parameters for the operator to perform tasks is supported.

Monitoring configuration #

Support setting multiple alarm rules, each alarm rule is set up to 5 alarm conditions. Each rule follows the logic of "or", that is, meeting anyone will trigger a notification. Supporting users choose results from the 3 types of alarm--abnormal alarm, failure alarm, timeout alarm--according to the drop-down box selection.
When the task execution meets the alarm conditions, it supports notifying the designated user through Lark, DingTalk, etc.