This topic describes the syntax, format, and related usage examples of the key-value extraction function.

ext_sep function

Extracts field values based on a single character separator.

Function syntax/format

The parameters are described below:

Parameters	Parameter type	Required	Default value	Value range	Description
field	String	Yes	/	/	Name of the log field to be separated.
output	String	Yes	/	/	The field names that are extracted and output; separated by commas.
sep	String	No	,	Single character	Separator
quote	String	No	"	Single character	The String within this character pair is not split.
restrict	Boolean	No	False	/	When the number of extracted values does not match the number of target fields entered by the user: True: Ignore and do not perform any extraction. False: Match the leading fields as many as possible.
mode	String	No	overwrite	overwrite,fill	Processing logic when the specified field does not exist. overwrite: Cover. fill: Set the target field when the target field does not exist.

Function examples

Scenario: Splits the value of the content field in logs by commas (,) into multiple fields.

Processing rules:

ext_sep("content", "city-1,city-2,city-3")

Log example

{
"content":"shanghai,hangzhou,chengdu"
}

Processing result:

{
"content":"shanghai,hangzhou,chengdu",
"city-1":"shanghai",
"city-2":"hangzhou",
"city-3":"chengdu"
}

ext_sepstr function

Extracts field values based on a specified string.

Function syntax/format

The parameters are described below:

Parameters	Parameter type	Required	Default value	Value range	Description
field	String	Yes	/	/	Name of the log field to be separated.
output	String	Yes	/	/	Extracted output field names, separated by commas (,) in English half-width format.
sep	String	No	,	/	Separator string.
restrict	Boolean	No	False	/	When the number of extracted values does not match the number of target fields entered by the user: True: Ignore and do not perform any extraction. False: Match the leading fields as many as possible.
mode	String	No	overwrite	overwrite,fill	Processing logic when the specified field does not exist. overwrite: Cover. fill: Set the target field when the target field does not exist.

Function examples

Scenario: Extracts the value of the content field into multiple fields from the log based on the string ||.

Processing rules:

ext_sepstr("content", "city-1,city-2,city-3", sep="||")

Log example

{
"content":"shanghai||hangzhou||chengdu"
}

Processing result:

{
"content":"shanghai||hangzhou||chengdu",
"city-1":"shanghai",
"city-2":"hangzhou",
"city-3":"chengdu"
}

ext_json function

Expand JSON strings in the specified field.

Function syntax/format

The parameters are described below:

Parameters	Parameter type	Required	Default value	Value range	Description
field	String	Yes	/	/	Original field name to be expanded.
depth	Number	No	100	1-500	The expanded JSON hierarchy.
prefix	String	No	No prefix added	/	The prefix for the deepest-level nodes in the expanded JSON string. Only the deepest-level JSON hierarchy needs prefix.
suffix	String	No	No suffix added	/	The suffix for the deepest node in the expanded JSON string. Suffixes are only added to the deepest JSON level.
fmt	String	No	simple	full simple	Formatting method. Supports configuring as the followings: full: Field names use the full path by merging the parent and current node, formatted as {parent}{sep}{prefix}{key}{suffix}. The separator is the character specified by the sep parameter, with a default value of the period (.). For example, extracting a json key from the data field, the field name is data.key. simple: The field name uses the node name. Format: {prefix}{key}{suffix}.
sep	String	No	.	/	The separator used when formatting parent and child nodes. This setting is available only when the fmt parameter is set to full.
expand_array	Boolean	No	True	True False	Expand the array. True: The array is expanded; and each element in the array is used as a key. False: The array is not expanded; and the entire array is used as a value.
include_node	String	No	No processing	/	The node names allowed for screening and filtering. The names are matched using regular expressions. By default, only nodes containing Chinese characters, numbers, letters, and _.- will be automatically expand. After setting the allowed node list, only nodes included in the list will be displayed in the results. For example, ext_json("json_data_filed", ...., include_node="k\d+").
exclude_node	String	No	No processing	/	The nodes to be excluded during filtering. The nodes are matched using regular expressions. After setting the exclusion list, all content not in the list will be excluded. For example, ext_json("json_data_filed", ...., exclude_node="k\\d+").
include_path	String	No	No processing	/	Node paths included during filtering. The paths are matched by regular expressions. The regular expressions for include_path and exclue_path match from the beginning of the path, and the path is separated by a period (.). For example, ext_json("json_data_filed", ...., include_path="content.k\\d+").
exclude_path	String	No	No processing	/	The node paths to be excluded during filtering. The paths are matched using regular expressions. The regular expressions for include_path and exclue_path match from the beginning of the path, and the path is separated by a period (.). For example, ext_json("json_data_filed", ...., exclude_path="content.k\\d+").

Function examples

Function example 1:

Scenario: Expand JSON data from the content field in the logs. Use pre_ as the prefix, suf_ as the suffix, and the default fmt format is simple.

Processing rules:

ext_json("content",prefix="pre_",suffix="suf_")

Log example

{
"content":"{\"k1\": \"v1\",\"k2\": {\"k3\": \"v3\"}}"
}

Processing result:

{
"content":"{\"k1\": \"v1\",\"k2\": {\"k3\": \"v3\"}}",
"pre_k1suf_":"v1",
"pre_k3suf_":"v3"
}

Function example 2:

Scenario: Expand JSON data from the content field in logs, with fmt set to full path and depth set to 1.

Processing rules:

ext_json("content",depth=1,fmt="full")

Log example

{
"content":"{\"k1\": \"v1\",\"k2\": {\"k3\": \"v3\"}}"
}

Processing result:

{
"content":"{\"k1\": \"v1\",\"k2\": {\"k3\": \"v3\"}}",
"content.k1":"v1",
"content.k2":"{\"k3\": \"v3\"}}"
}

Function example 3:

Scenario: Expand JSON data from the content field in logs, retaining only data from the k1 node.

Processing rules:

ext_json("content",include_node="k1")

Log example

{
"content":"{\"k1\": \"v1\",\"k2\": {\"k3\": \"v3\"}}"
}

Processing result:

{
"content":"{\"k1\": \"v1\",\"k2\": {\"k3\": \"v3\"}}",
"k1":"v1"
}

ext_json_jmes function

Extracts field values from JSON using jmes expressions.

Function syntax/format

The parameters are described below:

Parameters	Parameter type	Required	Default value	Value range	Description
field	String	Yes	/	/	Name of the original field to be extracted.
jmes	String	Yes	/	/	JMES expression.
output	String	Yes	/	/	Extracted output field name.
ignore_null	Boolean	No	True	/	Determines whether to ignore nodes with null values: True: Ignore fields with null values. False: When the extract result is null, output an empty string.
mode	String	No	overwrite	overwrite,fill	Processing logic when the specified field does not exist. overwrite: Cover. fill: Set the target field when the target field does not exist.

Function examples

Scenario: Extracts the field value from the content field in the log using the "k3.k4" jmes path and output it to the k3-k4 field.

Processing rules:

ext_json_jmes("content", jmes="k3.k4", output="k3-k4")

Log example

{
"content":"{\"k1\": 123, \"k2\": \"456\", \"k3\": {\"k4\": -34.5, \"k5\": true}}"
}

Processing result:

{
"content":"{\"k1\": 123, \"k2\": \"456\", \"k3\": {\"k4\": \"-34.5\", \"k5\": True}}",
"k3-k4":"-34.5"
}

ext_kv function

Extracts field values based on two levels of separators.

Function syntax/format

The parameters are described below:

Parameters	Parameter type	Required	Default value	Value range	Description
field	String	Yes	/	/	Source field name.
pair_sep	String	No	\s (space)	/	Primary separator. Can be used to separate multiple key-value pairs.
kv_sep	String	No	=	/	Secondary separator. Can be used to divide keys and values.
prefix	String	No	/	/	New field prefix
suffix	String	No	/	/	New field suffix.
mode	String	No	overwrite	overwrite fill	Processing logic when the specified field does not exist. overwrite: Cover. fill: Set the target field when the target field does not exist.

Function examples

Scenario: After extracting the value of the content field in logs, use the symbol (&) to split the value into key-value pairs, and then use the symbol (=) to separate keys and values.

Processing rules:

ext_kv("content", pair_sep="&", kv_sep="=")

Log example

{
"content":"f1=a&f2=b&f3=c"
}

Processing result:

{
"content":"f1=a&f2=b&f3=c",
"f1":"a",
"f2":"b",
"f3":"c"
}

ext_regex function

Extract the value of the specified field using the designated regular expression and assign the value to other fields.

Function syntax/format

The parameters are described below:

Parameters	Parameter type	Required	Default value	Value range	Description
field	String	Yes	/	/	Field name of the value to extract. If the specified field does not exist, no action is taken.
regex	String	Yes	/	/	Regular expression used to extract the field. Field values that match the regular expression rules will be extracted.
output	String	No	/	/	The matched field will be output. If the specified field does not exist, no action is taken. If you specify multiple fields, use commas to separate these fields.
mode	String	No	fill	overwrite fill	Specify the cover mode for output fields. overwrite: Always overwrite regardless of the condition. fill: Set the target field when the target field does not exist.

Function examples

Function example 1:

Scenario: Extracts the caller from the call_record field in the call log.

Processing rules:

ext_regex("call_record","\d{11}","caller")

Log example

{
"call_record": "caller:13800000000 called:13800000001"
}

Processing result:

{
"call_record": "caller:13800000000 called:13800000001",
"caller": "13800000000"
}

Function example 2:

Scenario: Extracts the caller and callee from the call_record log field.

Processing rules:

ext_regex("call_record","\d{11}","caller,called")

Log example

{
"call_record": "caller:13800000000 called:13800000001"
}

Processing result:

{
"call_record": "caller:13800000000 called:13800000001",
"caller": "13800000000",
"called": "13800000001"
}

Function example 3:

Scenario: Extract the caller from the call_record log field, and cover the origin_caller field.

Processing rules:

ext_regex("call_record","\d{11}","origin_caller","overwrite")

Log example

{
"call_record": "caller:13800000000 called:13800000001",
"origin_caller": "13800000002"
}

Processing result:

{
"call_record": "caller:13800000000 called:13800000001",
"origin_caller": "13800000000"
}

ext_grok function

The GROK function extracts specific values using regular expressions. GROK expressions provide ready-to-use definitions for common regular expressions, such as IP addresses and file paths. See the official documentation for common GROK expressions.

Function syntax/format

The parameters are described below:

Parameters	Parameter type	Required	Description
field	String	Yes	Field name to be extracted.
grok	String	Yes	GROK expression.
extend	Dict	No	Customize GROK expressions, where both key and value must be in string format.

Function examples

Function example 1:

Scenario: Extract year, month, day, and quoted strings from the content log field.

Processing rules:

ext_grok("content","%{YEAR:year} %{MONTH:month} %{MONTHDAY:day} %{QUOTEDSTRING:motto}")

Log example

{
"content":"2023 August 7 \"It is sunny day\""
}

Processing result:

{
"content":"2023 August 7 \"It is sunny day\"",
"year": "2023",
"month": "August",
"day": "7",
"motto": "\"It is sunny day\""
}

Function example 2:

Scenario: Extract IDs from logs using a custom GROK expression.

Processing rules:

ext_grok("content", "%{ID1:user_id1},%{ID2:user_id2}",extend=dict_make("ID1","%{WORD}-%{INT}","ID2","%{WORD}-%{WORD}"))

Log example

{
"content": "Chengdu-1023,Chongqing-Chongqing"
}

Processing result:

{
"content": "Chengdu-1023,Chongqing-Chongqing",
"user_id1": "Chengdu-1023",
"user_id2": "Chongqing-Chongqing"
}

Torch Log Service