In TLS data transformation functions, string syntax defines query rules and data filtering methods. This topic explains the general rules for query string syntax in data transformation functions.

Range of application

When using the following data transformation functions, you can filter data with query string syntax.

Type	Function	Scenarios
Event check function	f_search	Query string can be used to determine whether the field value of a specified event meets the given condition.

Features overview

The query string syntax supports keyword search, fuzzy matching, and other features. The table below shows the search capabilities for key-value and full-text query modes.

Feature	Key-value query	Full-text query
Keyword search	Supported	Supported
Fuzzy matching	Supported	Supported
Exact match	Supported	Not supported
Regex matching	Supported	Not supported
Numerical comparison and range evaluation	Supported	Not supported
Logical relationship evaluation	Supported	Supported
Field evaluation	Supported	Not supported

Special character escaping

Within the string query syntax, certain special characters must be escaped. Special characters that require escaping include the asterisk (*), colon (:), parentheses (()), single quotes (''), double quotes (""), question mark (?), and tilde (~). Escaping methods include:

Escape method	Description
Field name escaping	When a field name contains special characters, use a backslash (\) for escaping. For example, new\*:content and $path$\?: api. If the field name is in Chinese, wrap it in double quotation marks, such as e_search('"Chinese field":"Chinese value"').
Search values escaping	If the search value contains special characters, use the backslash (\) to escape them. For example, e_search('content:a\*b') uses a backslash to escape the asterisk.

Keyword search

Query methods	Description
Full-text query	Search for the specified string in all field values. Add quotation marks ("") when searching for Chinese keywords. The syntax format is as follows: f_search('keyword1') Example: f_search('"Chinese characters"'): Search for the specified string Chinese characters. f_search('"byte dance"'): The search keyword contains a space. f_search('byte dance'): queries multiple keywords. Among different keywords, the relationship is defaulted to OR.
Key-value query	Search for the specified string among all values in a field. The syntax format is as follows: f_search('key:value') Example: f_search('method: get'): determines whether the value of the method field contains the keyword get. f_search('company: "volc engine"'): determines whether the value of the company field contains the keyword volc engine, which contains a space. f_search('company: volc engine'): Equivalent to f_search('company: volc OR engine'); determines whether the value of the company field contains the keyword volc or if engine appears anywhere in the full log.

Fuzzy matching

The query string syntax supports fuzzy matching, allowing wildcard-based matches. Currently supported wildcards include:

Asterisk (*): matches zero or more characters.

Halfwidth question mark (?): matches one character, or a single-width Chinese character.

The syntax for fuzzy matching is as follows:

Query methods	Description
Full-text query	Perform fuzzy matching across all fields. The syntax format is as follows: f_search('keyword') Example: f_search('dance'): queries all strings that start with "dance", such as dancers. f_search('dance?'): queries six-character strings that start with "dance", such as dancer.
Key-value query	Perform fuzzy matching on field values within the specified fields. The syntax format is as follows: f_search('key:value') Example: f_search('company: bytdance'): Match zero or more characters. f_search('company: tik?ok'): Match one character.

Exact match

The query string syntax supports exact matching, which means the field value must match the specified string exactly from start to finish.

Syntax format:

f_search('key==value')

Example: f_search('company== "byte dance"'): The value of the company field must exactly match byte dance.

Regex matching

The query string syntax supports regex matching, which allows you to match the value of a specified field using a regular expression.

Syntax format:

f_search('key~=Regular expression')

Syntax description:

If a regex pattern contains a backslash (\), enclose the expression in double quotation marks.

Regex matching is partial by default. To match the entire field value, add ^ at the start and $ at the end of the expression. For example, ^\d+$.

Example:

f_search('path~= "\d+"'): The path field value contains digits.

f_search('id~= "^\\d+$"'): The value of the id field is a number.

Numerical comparison and range evaluation

The query string syntax supports numerical comparisons and range evaluation. The supported comparison operators are >, >=, =, <, and <=.

Feature	Description
Value comparison	Compare the value of the specified field with a given number. For example: f_search('count >= 9') # >=9 f_search('count > 9') # > 9 f_search('count = 9') # = 9 f_search('count <= 9') # <=9 f_search('count < 9') # < 9
Data range check	Compare the value of the specified field with a given numeric range. The numeric range must be a closed interval. For example: f_search('age: [1, 18]') # >=1 and <=18

Logical relationships

Evaluate logical relationships for multiple search results. Parentheses () can be used to adjust the evaluation order.

Logical relationships	Keyword
And	AND, &&, and similar options; only uppercase is supported.
OR	OR, \|\|, and similar options; only uppercase is supported.
NOT	NOT, ! and similar options; only uppercase letters are supported.

Example:

Field evaluation

Use search strings to evaluate fields.

Example:

f_search('field:*'): determines the field exists.

f_search('NOT field:*'): determines the field does not exist.

f_search('field~=".+"'): determines the field exists.

f_search('NOT field~=".+"'): determines the field does not exist.

Torch Log Service