This topic introduces how to use field extraction plugins in LogCollector.

delimiter plugin

You can use the delimiter plugin to specify that LogCollector extracts log fields based on a separator mode.

Parameters description

Name	Type	Required	Description
field	String	Yes	Field name to be processed by the LogCollector plugin.
delimiter	String	Yes	Separator Enter the correct separator based on the log format to ensure each segment can be accurately extracted. Notice Invisible characters are not supported as separator. Only one separator can be specified, but a separator can include multiple characters.
quote	String	No	Quotation marks. Content wrapped by quotation marks will not be separated by the separator, but will be parsed as a complete field. Notice Setting the quotation marks as invisible characters is not supported. Only one pair of quotation marks can be specified, but multiple characters can be wrapped within it. Only LogCollector V1.0.34 and later versions support setting quotation marks.
keys	Array of String	Yes	The field name (key) for the subfields parsed from the field specified. The order of each key should correspond one-to-one with the parsed fields, and the quantity must match. Otherwise, the LogCollector plugin configuration fails to execute. Notice Keys must not be duplicated or left blank.
when	Object	No	The plugin execution condition. This plugin is executed only when the condition is evaluated as true. For more details, see Plugin execution conditions.
trim_values.mode	String	No	The trim_values specifies whether to remove strings from both ends of the value string, with the mode parameter defining the removal method. none: No string is removed. all: (default) Remove strings from both ends of the value. left: Remove strings from the beginning of the value. right: Remove strings from the end of the value string.
trim_values.chars	String	No	The trim_values is used to specify whether to remove characters from both ends of the value. The chars parameter defines the range of characters to be removed. By default, it is double quotation mark (").
allow_empty_values	Boolean	No	If a specified field value contains a subfield with an empty value after parsing, this parameter determines whether to keep this key-value pair. true: Keep key-value pairs whose value is empty. false: (Default) Do not keep key-value pairs with empty values.
allow_overwrite_keys	Boolean	No	If the key specified in keys duplicates an existing key in the collection configuration, this parameter determines whether to overwrite the field corresponding to the existing key. true: overwrite false: (Default) If a key is duplicated, the LogCollector plugin configuration fails to execute.

Configuration examples

Raw log:

"delimiter_key": "\"Click&&Here\"&&36&&bold&&text1&&250&&100&&#center#&&sun1.opacity = (sun1.opacity / 100) * 90;"

LogCollector plugin processing configuration

{
"processors":[
{
"delimiter": {
"field": "delimiter_key",
"delimiter": "&&",
"quote": "\"",
"keys": [
"data",
"size",
"style",
"name",
"hOffset",
"vOffset",
"alignment",
"onMouseUp"
],
"trim_values": {
"mode": "all",
"chars": "#"
},
"allow_overwrite_keys": true,
"allow_empty_values": true
}
}
]
}

Processing result

"data": "\"Click&&Here\""
"size": "36"
"style": "bold"
"name": "text1"
"hOffset": "250"
"vOffset": "100"
"alignment": "center"
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"

fullregex plugin

You can use the fullregex plugin to specify that LogCollector extracts log fields using complete regular expression mode.

Parameters description

Name	Type	Required	Dscription
field	String	Yes	Field name to be processed by the LogCollector plugin.
pattern	String	Yes	Each capture group in a regular expression corresponds to one parsed value.
keys	Array of String	Yes	The field name (key) for the subfields parsed from the field specified. The order of each key should correspond one-to-one with the parsed fields. Notice Keys must not be duplicated or left blank.
when	Object	No	The plugin execution condition. This plugin is executed only when the condition is evaluated as true. For more details, see Plugin execution conditions.
trim_values.mode	String	No	The trim_values specifies whether to remove strings from both ends of the value string, with the mode parameter defining the removal method. none: No string is removed. all: (default) Remove strings from both ends of the value. left: Remove strings from the beginning of the value. right: Remove strings from the end of the value string.
trim_values.chars	String	No	The trim_values is used to specify whether to remove characters from both ends of the value. The chars parameter defines the range of characters to be removed. By default, it is double quotation mark (").
allow_empty_values	boolean	No	If a specified field value contains a subfield with an empty value after parsing, this parameter determines whether to keep this key-value pair. true: Keep key-value pairs whose value is empty. false: (Default) Do not keep key-value pairs with empty values.
allow_overwrite_keys	boolean	No	If the key specified in keys duplicates an existing key in the collection configuration, this parameter determines whether to overwrite the field corresponding to the existing key. true: overwrite false: (Default) If a key is duplicated, the LogCollector plugin configuration fails to execute.

Configuration examples

Raw log

"fullregex_key": "Click Here, 36, bold, text1, 250, 100, #center#, sun1.opacity = (sun1.opacity / 100) * 90;",

LogCollector plugin processing configuration

{
"processors":[
{
"fullregex": {
"field": "fullregex_key",
"pattern": "([\\s\\w]+),\\s(\\d+),\\s(\\w+),\\s(\\w+),\\s(\\d+),\\s(\\d+),\\s(\\S+\\w+\\S+),\\s(.*).*",
"keys": [
"data",
"size",
"style",
"name",
"hOffset",
"vOffset",
"alignment",
"onMouseUp"
],
"trim_values": {
"mode": "all",
"chars": "#"
},
"allow_overwrite_keys": true,
"allow_empty_values": true
}
}
]
}

Processing result

"data": "Click Here"
"size": "36"
"style": "bold"
"name": "text1"
"hOffset": "250"
"vOffset": "100"
"alignment": "center"
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"

kv plugin

You can use the kv plugin to extract log fields. This topic explains the parameters and provides configuration examples for the kv plugin.

Parameters description

Name	Type	Required	Dscription
field	String	Yes	Field name to be processed by the LogCollector plugin.
when	Object	No	The plugin execution condition. This plugin is executed only when the condition is evaluated as true. For more details, see Plugin execution conditions.
split_pattern	string	No	The separator between key-value pairs. If this parameter is not specified, the separator is a space.
quote	String	No	Quotation marks. Content wrapped by quotation marks will not be separated by the separator, but will be parsed as a complete field. Notice Setting the quotation marks as invisible characters is not supported. Only one pair of quotation marks can be specified, but multiple characters can be wrapped within it. Only LogCollector V1.0.34 and later versions support setting quotation marks.
kv_split_pattern	string	No	The separator between the key and value in each key-value pair. If this parameter is not specified, the separator defaults to equals sign (=).
trim_keys.mode	string	No	The trim_keys specifies whether to remove characters from both ends of the key. The mode parameter defines the removal method. none: No string is removed. all (default): Remove strings from both ends of the key. left: Remove strings from the beginning of the key. right: Remove strings from the end of the key.
trim_keys.chars	string	No	The trim_keys mode specifies whether to remove characters from both ends of the key; the chars parameter defines the range of characters to be removed. By default, it is double quotation mark (").
trim_values.mode	String	No	The trim_values is used to specify whether to remove characters from both ends of the value; the mode parameter determines the removal method. none: No string is removed. all: (default) Remove strings from both ends of the value. left: Remove strings from the beginning of the value. right: Remove strings from the end of the value string.
trim_values.chars	String	No	The trim_values specifies whether to remove strings from both ends of the value string; the chars parameter defines the range of characters to remove. By default, it is double quotation mark (").
allow_empty_values	boolean	No	If a specified field value contains a subfield with an empty value after parsing, this parameter determines whether to keep this key-value pair. true: Keep key-value pairs whose value is empty. false: (Default) Do not keep key-value pairs with empty values.
allow_overwrite_keys	boolean	No	If the key specified in keys duplicates an existing key in the collection configuration, this parameter determines whether to overwrite the field corresponding to the existing key. true: overwrite false: (Default) If a key is duplicated, the LogCollector plugin configuration fails to execute.

Configuration examples

raw logs

"kv_key": "data: \"Click,Here\", size: 36, style: bold, name: text1, hOffset: 250, vOffset: 100, #alignment#: #center#, onMouseUp: sun1.opacity = (sun1.opacity / 100) * 90;"

Plugin configuration for LogCollector

{
"processors":[
{
"kv": {
"field": "kv_key",
"split_pattern": ",",
"kv_split_pattern": ":\\s+",
"quote": "\"",
"trim_keys": {
"mode": "all",
"chars": "#"
},
"trim_values": {
"mode": "all",
"chars": "#"
},
"allow_overwrite_keys": true,
"allow_empty_values": true
}
}
]
}

Processing result

"data": "\"Click,Here\""
"size": "36"
"style": "bold"
"name": "text1"
"hOffset": "250"
"vOffset": "100"
"alignment": "center"
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"

Torch Log Service