Docs

Torch Log Service

Configure log indexes

Configure indexes

TLS supports log query and analysis. However, before performing log query and analysis, you must configure indexes. When configuring the index, you need to specify the index type and the data type for each field. The index configuration determines the query methods and the accuracy of query results.

This topic introduces the steps to configuring an index and provides detailed explanations of the index configuration.

Prerequisites

The log project and log topic are created. For more details, see Log project and Log topic.

We recommend that you collect log data via LogCollector or other methods before you configure index. For more details, see Collect log data.

Notice

Category	Description
Enable index	At least enable one index between full-text index and key-value index. If no index is enabled, logs that are collected can not be searched.
Delimiter	TLS words have a character limit of 32,766 characters. If you do not set token delimiters, logs whose full-text or word fields are too long might be discarded.
Full-text index	Enable full-text index before performing full-text index. When only the full-text index is enabled, you can use Query syntax to query logs.
Key-value index	Before executing key-value query, key-value index must be enabled for the target field. After configuring key-value index and enabling statistics function for target field, you can use the field in analysis statements. When configuring key-value index, you can add up to 500 log fields.
Edit index	You can modify the index anytime after configuration. After you modify index, note that: The new index only applies to new data that is collected after the modification. The index for existing data will not update. When querying logs, old data still follows the previous index rules. However, if the key-value index is deleted, key-value queries are no longer supported. When analyzing logs, old data is excluded; only logs that are collected after modifying the index are used for analysis.

Index type

Index type	Description
Full-text index	After enabling the full-text index, TLS splits logs into multiple text-type tokens based on the delimiter. Then you can query the full-text by keywords. For example, use the error or warning statement to query logs containing the keywords error or warning.
Key-value index	After enabling key-value index for a target field, TLS first splits logs by field name (Key), and then splits field values (Value) within each field based on the delimiter. Key-value index supports four data types: text, long, double, and json. For details, see Index data types. After creating a key-value index, you can specify field names and field values (Key:Value) to query logs.
Reserved field index	When writing log data to TLS, TLS automatically adds some metadata fields to each log. The metadata fields are used to record log source, timestamp, and other details. These fields are called reserved fields in TLS. After enabling the index feature, TLS automatically applies key-value indexing and statistics to some reserved fields, such as __time__. For default reserved field index configurations, see Reserved field. For fields without preset indexes, such as __content__, you can manually set index for it via Reserved Field Index.

Create index

Select the region where log service is located from the top navigation bar.

In the left navigation pane, select Common functions > Query and analysis.

On Query and analysis page and in the list of left pane, select the specified project and topic.

On the query and analysis page for the specified log topic, click Index configuration.

Turn on Enable index, and select the index type as needed.

TLS supports the following types of index configurations. For detailed configuration methods, see:

Full-text index configuration

Key-value index configuration

Reserved field index configuration

Click OK.

The index configuration is expected to take effect within 1 minute. Refresh the page later to view real-time logs or query and analysis logs.

Full-text index configuration

On the index configuration page, enable full-text index and fill in the following settings.

Configuration	Description
Case sensitive	During query, English letters can be case sensitive. You can choose to enable or disable this feature. Enable: Queries are case sensitive. For example, when searching for logs containing error, you can only query using the error keyword. Using ERROR will not find such logs. Disable: Queries are not case sensitive. For example, you can query logs that contain error using either error or ERROR.
Delimiter	When querying, TLS can split the Chinese content of logs according to Chinese grammar. You can choose to enable or disable this function. Either Delimiter or Chinese grammar must be enabled. Enable: Chinese characters in logs: Logs are split based on common Chinese grammar rules. Delimiter that contains custom Chinese content is not supported. Non-Chinese characters in logs: Split logs based on the delimiter specified in the delimiter parameter. Disabled: Split logs based on the delimiter specified in the delimiter parameter. Notice Even if the Chinese word segmentation configuration is not enabled, you can still query Chinese characters in logs. In this case, the keyword must exactly match the Chinese field. For example, if the log contains Chinese field Access request failed: If not enabled, querying with the keyword failed will not find the log. You must use the full keyword Access request failed to retrieve the log. Once enabled, searching with keywords like Access, request, or failed can all find the log.
Delimiter	Split log content based on the specified delimiter. Either Chinese word segmentation or delimiter must be enabled. Enabled: TLS splits log content based on specified delimiter. If you query, query logs based on the split log content. If the default delimiter does not meet your needs, you can customize the delimiter. For details on delimiters supported by TLS, refer to the console interface. Disabled: No splitting. Each log is queried as a whole. For example, the log content is /var/log/access.log. If you set the delimiter as a forward slash (/), the log will be split into var, log, and access.log. You can query this log using any of these words, such as var, va, or access.log. If you use the default delimiter, the log will be split into var, log, access, and log. You can query this log using any of these words, such as var, va, or access. If the delimiter is disabled, the entire log is treated as a single word: /var/log/access.log. You can query the log only by using the complete string /var/log/access.log or by performing a fuzzy query with the string /var/log/*.

Key-value index configuration

On the index configuration page, enable key-value index and fill in the related settings.

After enabling the key-value index, you can extract indexes from raw logs or add indexes manually. Auto-update index is also supported.

Extract indexes from raw logs

Click Extracting fields from original log. In the pop-up window, the first log in the preview data is obtained by default to automatically configure key-value indexes.

If you click Add, the existing index attributes are retained but configurations for fields with the same key will update.

If you click Cover, existing index attributes will be replaced directly.

Automatically update index

Select Automatic update, and the system will automatically add newly discovered fields to the Key-value index. The automatically generated index fields will display an Auto generation label in the field name column. You can modify the index attributes as needed. It may take 2 to 3 minutes for the interface to display relevant indexes the first time it is enabled.

Notice

The indexes generated by automatic updates follow these specifications:

Field name: Only fields that meet the TLS field name specification are automatically added.

Field type: Only text, long, and double types are supported. Fields of the json type are automatically added as text type.

You can edit or delete automatically added indexes. Modified indexes will not be covered by automatically updated indexes.

After Rebuild Index is performed, automatic index updates will not be triggered.

A maximum of 500 indexes can be generated and manually added.

Manually configure index

Click Add and manually add index fields by referring the table below.

Configuration			Description
Field name			The name of the log field, i.e., the key in the key-value pair. Support only letters, numbers, spaces, underscores (_), hyphens (-), and slashes (/). Do not support starting or ending with space. Within the same index, the key name must be unique. Case insensitive. Different case combinations of the same word are treated as unique. Length must be between 1 and 128 characters. Notice You can add up to 500 log fields.
Field type			The data type of the log field. Select data type based on your actual needs. The data type setting determines the accuracy and precision of the query and analysis results. For more details, see Index data types. Supported data types include text, long, double, and json.
Enable analytics			This configuration determines whether to enable or disable statistics. Enabled: This field will be used for statistical analysis, which means this field can be used in SQL analysis statement. Disabled: this field will not be used for statistical analysis.
Field configuration	Case sensitive	This configuration determines whether case sensitivity for English letters is applied when you query and analyze based on this field. Enable: Queries are case sensitive. For example, when searching for logs containing error, you can only query using the error keyword. Using ERROR will not find such logs. Disable: Queries are not case sensitive. For example, you can query logs that contain error using either error or ERROR.
		Chinese word segmentation	When querying, TLS can split the Chinese content of logs according to Chinese grammar. You can choose to enable or disable this function. Enable: Chinese characters in logs: Logs are split based on common Chinese grammar rules. Delimiter that contains custom Chinese content is not supported. Non-Chinese characters in logs: Split logs based on the delimiter specified in the delimiter parameter. Disabled: Split logs based on the delimiter specified in the delimiter parameter.
		Delimiter	Query by field determines whether log content is split based on the specified delimiter. Enabled: Split log content by the specified delimiter, and query based on the split log content. If the default delimiter does not meet your needs, you can customize the delimiter. For details on delimiters supported by TLS, refer to the console interface. Disabled: No splitting. Each log is queried as a whole. For example, the log content is /var/log/access.log. If you set the delimiter as a forward slash (/), the log will be split into var, log, and access.log. You can query this log using any of these words, such as var, va, or access.log. If the default delimiter is used, the log will be split into var, log, access, and log. You can query this log using any of these words, such as var, va, or access. If the delimiter is disabled, the entire log is treated as a single word: /var/log/access.log. You can query the log only by using the complete string /var/log/access.log or by performing a fuzzy query with the string /var/log/*.
Action	Modify description	Descriptive information of index fields enables the intelligent assistant to extract and analyze corresponding fields more effectively. The intelligent assistant feature is only available in the China (Beijing), China (Guangzhou) and China (Shanghai) regions.

Reserved field index configuration

For reserved fields without preset indexes, such as the __content__ field, you can manually configure indexes for them by using Reserved field index. On the index configuration page, enable Reserved field index and fill in the related settings.

Notice

Currently, only the __content__ field supports reserved field index settings. If the __content__ field is set as json type, you can enable automatic indexing for all text in JSON.

Supports enabling both full-text indexing and __content__ reserved field indexing simultaneously. If the two indexes are enabled, index traffic fees and index storage space fees will both double.

Index traffic description

Index type	Description
Full-text index	Both field names and field values are stored as text types and both generate index traffic.
Key-value index	Text type: Both field name and field value generate index traffic. Numeric types: Field names of long type and double type do not generate index traffic. The index traffic that each field value occupies is 8 bytes. JSON type: Both field names and field values (including subfields) generate index traffic. If no index is created for a JSON subfield, the subfield is categorized as the value of a JSON type field. TLS calculates index traffic based on text type. When JSON subfield is indexed, TLS calculates index traffic based on the data type of subfield, such as text, long, and double type.

Related operations

Set maximum field length

When analyzing logs, TLS supports a maximum field value length of 2048 bytes (2 KB) by default. If you need to adjust the maximum field value length, you can set the maximum length of statistical field, with a range of 64-1048576.

Notice

If a single field value exceeds the maximum length specified, the excess portion is truncated and excluded from analysis.

After updating the maximum field length, only incremental data is affected.

Next steps

Single-topic log query

Log analysis

Last updated: 2026.07.07 16:42:36