Docs

Torch Log Service

Configure log indexes

Index data types

When setting a key-value index for log fields, you need to set a data type (text, long, double, or json) for each field with key-value indexing enabled. The data type of a field's index determines how you can query the field and affects the precision and accuracy of query. This topic introduces configuration details and notes for each data type.

Data type

Log field values are typically text or numeric. You can set the index data type for a field as text (string), long (integer), or double (floating-point) to meet different log query needs.

For more complex structured data such as json, you can set the index data type to json. TLS supports expanding standard and valid json fields. You can set the index data type for leaf nodes as text, long, or double to support queries of json leaf field values in nested structures.

The data types for index fields are described below:

Type	Description
text	Suitable for fields of string type.
long	Suitable for fields of numeric type, such as integer (Int 64). After setting the type to long, you can query log fields by value range using operators such as >.
double	Suitable for fields of numeric type, such as double (64 bit). After setting the type to double, you can query log fields by value range using operators such as >.
json	Applies to fields of the JSON type. Supports creating indexes for standard JSON fields, and allows setting the index data type for each expanded subfield.

Text type

Text type is suitable for fields with string values.

The text type index supports fuzzy queries for this field using wildcards like * or ?.

After enabling the full-text index, TLS sets the entire log (all fields except __time__) to the text type by default.

Long type

Long type is suitable for numerical fields, such as integer (Int 64).

Long type index supports querying log fields by numeric ranges using range operators like >.

The long type index does not support case sensitive, Chinese word segmentation, delimiter settings, and does not support fuzzy queries.

If you set the index type for floating-point or string fields to long, you cannot query those fields.

Double type

The double type is suitable for numeric fields, such as 64-bit floating point.

Double type indexes support querying log fields by numeric range, for example, using range operators such as >.

Double type indexes do not support setting Case sensitive, Chinese word segmentation, or Delimiter, and do not support fuzzy queries.

If the index type of a string field is set to double, that field can not be queried.

json type

TLS supports creating indexes for standard json fields. Once a field is set to the json index data type, each expanded subfield supports individual index data type settings. After setting subfield indexes, query and analysis can be conducted at the subfield level.

You can set index data types to text, long, or double based on the actual data type of the json subfield.

If a subfield value is of Boolean type, set its index data type to text.

json Subfield description

Naming rules

When setting the key-value index for json subfields, use . to represent the hierarchical relationship between fields in the json data.

For example, key1.key2.key3 means that key3 is a subfield of key2, and key2 is a subfield of key1.

The subfield name can not include the name of the top-level parent field.

Child field names can not be prefixed by a parent field with a conflicting data type.

You can not configure subfields key1.key2 and key1.key2.key3 at the same time.

You can configure subfields key1.key2.key3 and key1.key2.key4 at the same time.

Automatic indexing

When enabled, TLS will automatically create indexes for all fields with text values in json fields. Automatic indexing and automatic index statistics are mutually exclusive; you can select only one. For details, see What is the difference between automatic JSON indexing and manual index configuration?.

Subfield configuration	Description
Automatic indexing	Takes effect within seconds. Supports subfield queries without updating the subfield list in the index configuration. If subfields are not configured, the parent field can be analyzed separately but can not be queried separately.
Automatic index statistics	Takes effect within minutes. Supports querying and analyzing subfields; the subfield list in index configuration will be updated.

Notice

Query and analysis differences exist between automatic JSON indexing and manual index configuration. For example, automatically created subfield indexes can not be used in SQL statements or not key: * statements.

Automatic index creation is not supported for subfields of numeric or array type values.

json subfield restriction settings

TLS does not support: indexing fields with JSON array values; indexing fields within a JSON array.

Only key-value index configurations like case sensitive, Chinese word segmentation, delimiter, and enable statistics are supported at the JSON field dimension. The key-value index configuration for subfields of type text inherits the JSON field settings by default and can not be configured separately.

Supports up to 20 levels of JSON nesting. That is, TLS can parse at most 19 levels of data within a JSON subfield.

json subfield exception scenario handling

For invalid JSON objects, TLS only parses the standard valid part.

If TLS detects that the data types of the indexed JSON subfields do not match those of the actual log field values, the entire JSON log data can not be queried and analyzed. For example, if the actual field value is "zhangsan" and the field type is set to double, then none of the fields in the JSON object can be queried or analyzed.

TLS supports indexing leaf nodes in JSON objects, but it is not recommended to index sub-nodes that contain leaf nodes. If you index a nested JSON object within a JSON object, TLS will serialize the JSON value of the subfield as a string.

For common issues encountered during query and analysis of JSON logs, see FAQ about query and analysis of JSON logs.

Configuration example

For example, in application service log collection scenarios, each log represents a user request. The info field is formatted as a JSON object and provides details of a service request. Within the info field, the response and developers fields are also nested JSON objects.

Log sample:

Index configuration:

Set the index data type of the info field to JSON and specify the corresponding index data types for its subfields based on their actual data types. For instance, configure basic key-value indexes for subfields like response.error_msg and response.status.

The field value of regions, info.ip, and info.developers is array format. TLS does not support effective JSON key-value indexing for these fields. You can use JSON functions to parse the parent field of these fields and extract their contents for query and analysis.

Sample query and analysis:

Analysis scenarios	Query statements
Query logs containing NewApp	NewApp
Query logs where total users exceed 10,000 and the average daily visits exceed 10,000.0.	total_users: >10000 AND avg_pv_per_day: >10000.0
View all logs	* \| SELECT *
Filter logs by product name: NewApp	* \| SELECT * WHERE product='NewApp'
Filter logs where total users exceed 10,000 and the average daily visits exceed 10,000.0.	* \| SELECT * WHERE total_users>10000 AND avg_pv_per_day>10000.0
Filter logs where the requested API is CreateIndex	* \| SELECT * WHERE info.api='CreateIndex'
Count logs with response status codes other than 200	* \| SELECT COUNT(*) WHERE info.response.status<>200
Count logs aggregated by requests to the NewApp API	* \| SELECT info.api, COUNT(*) AS c WHERE product='NewApp' GROUP BY info.api

Last updated: 2026.04.27 14:36:40