Docs

Torch Log Service

Data import

Import data from Kafka

TLS supports Kafka data import. This topic describes the steps to import data from Kafka to TLS.

Background information

The data import feature of TLS allows you to import messages from Kafka clusters into a specified log topic. The Kafka data import feature is typically used for scenarios such as cloud migration. For example, you can import various system logs and application data aggregated by a self-built ELK system into TLS for centralized storage, query, analysis, and processing. The data import feature of TLS supports importing data from both Volcanoengine Message Queue Kafka clusters and self-managed Kafka clusters. After you create an import task, you can view details of the task in the TLS console or service logs.

TLS also adds the following metadata fields to imported log data.

Field	Description
__content__	Kafka messages.
__path__	The field value is empty.
__source__	The service address of the Kafka cluster.

Notice

The following limitations apply to importing data from Kafka:

Limits	Description
Kafka version	The Kafka version must be 0.11.x or later.
Number of concurrent subtasks	TLS creates multiple concurrent subtasks for log import based on the number of Kafka topics. When Kafka topics exceed 2,000, TLS creates 16 subtasks. When Kafka topics exceed 1,000, TLS creates 8 subtasks. When Kafka topics exceed 500, TLS creates 4 subtasks. When there are 500 or fewer Kafka topics, TLS creates 2 subtasks.
Number of data import configurations	You can create up to 100 different types of data import configurations in a single project.

Billing description

Importing data from Kafka involves charges for write traffic in TLS and log storage. For detailed pricing information, see TLS Billing items based on pay-by-feature mode.

Billing item	Description
Write traffic	When importing Kafka data to TLS, write traffic fees for TLS will occur.
Log storage	After saving Kafka data to TLS, the backend automatically compresses the data. Storage fees are based on the actual compressed size.
Others	We recommend that you enable indexing for imported data to facilitate future queries, analysis, and data statistics. Enabling indexing incurs index storage fees. Storage fees are calculated based on the space occupied by indexes that are created from the log data before compression. If imported data is later read, transformed, or shipped, charges will apply according to the billing standards.

Prerequisites

A Kafka cluster that meets the version requirements has been created. For version requirements, see Notice.

A log project and log topic are created. For detailed steps, see Create resources.

Create import task

Step 1 Initial configuration

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the Data import section, click Kafka Import.

Select the project and log topic, and then click Next step: configure import rules.

Configuration	Description
Log project	The log project used to store Kafka data.
Log topic	Name of the log topic used to store Kafka data. Tip Importing data to dedicated log topics such as cloud product logs or service logs is not supported.

Step 2 Import configuration

Enter the import configuration.

Configuration	Description
Task name	Name of the log import task.
Service address	Different types of Kafka clusters have different service addresses. Details are provided below: Volcengine Message Queue for Kafka: the access point of the Kafka instance. For more information, see View access point. If the Kafka instance and the TLS Project are in the same region, you can use the private network to access. Otherwise, you need to use public network to access. Self-hosted Kafka cluster: IP address and port or domain name and port of the Kafka broker. Only supports public network. Multiple service addresses need to be separated with commas (,).
Kafka instance ID	If you are using Volcengine Message Queue for Kafka, set this value to the Kafka instance ID. For details on how to obtain this value, see View instance details.
Is authentication required	If you are using a public service address, we recommend enabling authentication. Complete the following configuration based on the Kafka settings. For authentication details, see Supported security mechanisms. Protocol type: Secure transmission protocol. Options include plaintext, sasl_ssl, ssl, and sasl_plaintext. Authentication mechanism: Password authentication mechanism. Options include PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512. Username: Kafka SASL username used for authentication. Password: User password.
Topic list	Kafka Topic name. Multiple Kafka Topic need to be separated with commas (,).
Consumer group	Kafka consumer group. If left blank, the system will automatically create a Kafka consumer group.
Starting position	Starting position for data import. Optional values: Earliest data: Import starts from the first data in the specified Kafka topic. Latest data: Import starts from the most recently generated data in the specified Kafka topic.
Encoding format	Encoding format of the data. Optional values include UTF-8 and GBK.
Log extraction mode	The format of the data to be imported. Optional values: Single-line full text: If data structuring is not required, select the single-line full text mode. JSON: If the data to be imported is in JSON format, select JSON mode. Import tasks parse the data into key-value pairs, only parsing the top-level of JSON data. Single-line complete regular expression: Use a regular expression to parse the data into key-value pairs. In this case, additional parameters are required. For details, see the table below.
Upload failed parsing logs	Choose whether to upload logs that failed to parse. This feature is enabled by default. Enable: All logs that fail to parse will be uploaded to TLS, with the specified field as the key name and the original log content as the value. At this point, you need to specify the failure log key name to store logs that failed to parse. Disable: Logs with parsing failures will not be uploaded to TLS.
Failure log key name	The field used to store logs that failed to parse.
Time field	Choose the log time field. You can use the system time when the log service imports the data as the log time, or you can specify a time field, and the log service will use the value of that field as the log time. System points in time(default): If you use system time or Kafka message timestamps as log timestamps, you also need to configure a default time source, which defaults to the current system time. Specify time field: Time key name: The Key to the time field. Time regular: Use a regular expression to extract time from the log. Time conversion format: Time format used to parse the Value of the time field. For supported time format settings, see Time format. For example, if the time field value is 1651648409, set time conversion format as %s; if the value is 1651648416882, set time conversion format as %F; if the value is 2022-01-01T15:30:00.999+0800, set time conversion format as %Y-%m-%dT%H:%M:%S.%f%z. Time zone: Time zone configuration. It is the time zone of the machine by default. You can set it to GMT or UTC format.
Default time source	When you select Time field and choose System time in point, you can use System current time or Kafka timestamp as log time.

If you set the log extraction mode to single-line complete regular expression, you also need to configure the following parameters. You can choose different extraction modes to manually or automatically generate regular expressions. Confirm the extraction results in the extract results section to ensure the regular expression extracts the desired data from your log sample.

Extract mode	Description
Auto generation	If you enable Auto generation, the page automatically extracts the regular expression as you select words. In the Log sample section, select the content to be extracted by highlighting text from left to right with your mouse. Confirm the selected value in the pop-up text box, then click Generate regex. Once you have extracted all required key-value pairs, set the key values for each field in the Extract results section. Notice For fields that do not need to be collected, you can skip them while selecting words. The key in Extract results cannot be empty or duplicated. The Auto generation function only supports regular expression extraction for English characters; Chinese characters are not supported yet.
Manual set	To configure regular expressions manually, disable Auto generation feature. After configuration, click Verify now. TLS matches content in the log sample using the entered regular expression and extracts key-value pairs When Validation success is displayed, you can set the key values for each field in the Extract results section. Notice Use () to identify the regular expression for each kv. TLS regards () as capture groups and parse capture groups into key-value pairs. If a field does not need to be collected, you can omit it from the regular expression or delete it from the Extract result after extraction. The key in Extract results cannot be empty or duplicated.

Click Execution preview to view the content in the file to be imported, and then click Next step: check index configuration.

After completing the import configuration, TLS provides a data preview feature for the specified data source. You can check whether the data in the preview section matches the data to be imported. If the data is different, it means the data source configuration is incorrect. We recommend reconfiguring it.

Step 3 Check index configuration

Check index configuration.

Follow the prompts on the page to set or update the index as needed. You can also click Import existing index configuration to import indexes from other log topics.

If this log topic does not have an index configured, you can set one. For details, see Configure indexes.

If the index for this log topic is already configured, you can decide whether to update the index based on the log fields parsed by the collection rule.

Notice

You must enable the index to query and analyze the collected log data.

The updated index settings only apply to new data written afterward. For more instructions and limitations on query and analysis, see Query overview and Analysis overview.

Click Next step: check index configuration.

View import task details

After successfully creating a Kafka data import task, you can view relevant information in the TLS console, including details of the import configuration.

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

Then in the Data import tab under Task list, click the log import task name.

View task configuration information.

The details page of the log import task displays basic information and data source configuration.

Edit import task

After successfully creating a Kafka data import task, you can modify the task configuration at any time based on your business needs.

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

Then in the Data import tab's Task List section, click the import task to edit.

In the upper right corner of the page, click Edit configuration.

You can refer to the instructions in Create import task to modify related configurations.

Copy import task

TLS supports copying data import tasks. If a similar data import task exists, you can copy all the configurations from an existing task, and then adjust the settings as needed to create a new data import task.

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the Data import tab under the Task list section, click Copy under the Actions column for the task to be copied.

Refer to the steps for creating an import task to confirm or modify the relevant parameter settings.

Delete imported tasks

If there is a failure or anomaly with a data import task, you can delete the task and re-import data. If you have finished migrating historical log data, or no longer need to import logs to TLS, you can follow the steps below to delete log import tasks.

Notice

Deleting a log import task does not affect logs that have already been imported successfully.

A deleted log import task can not be restored. Proceed with caution.

Select the region where torch log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the task list section under data import tab, locate the log import task you need to delete, and click Delete in its Actions column.

Confirm the information for the task to be deleted in the popup dialog, and then click Confirm.

Last updated: 2026.07.30 17:29:24