Docs

Torch Log Service

Data import

Import data from Elasticsearch

TLS offers convenient and efficient data import features, supporting multiple data sources such as Elasticsearch. The data import function structures your Elasticsearch data and stores it in TLS for further query, analysis, and multi-platform processing. This topic describes how to import data from Elasticsearch to TLS.

Features overview

Elasticsearch is a distributed, RESTful search and data analytics engine. In migration scenarios such as moving services to the cloud or relocating applications, you can quickly import data from existing ELK systems into TLS. Currently supported data sources include Volcengine Elasticsearch, self-built Elasticsearch clusters, and Elasticsearch clusters from other cloud providers. When importing data from Elasticsearch, you can filter data with queries and configure time fields. After creating an import task, you can view the progress and status in TLS console, or use service log analysis for detailed progress and status data.

In addition, TLS will add the following metadata fields to imported log data:

Field	Description
__tag____index__	The source index to which the log belongs.
__tag____meta_id__	Document ID of the log.
__source__	URL corresponding to the imported index, i.e., the service address of the Elasticsearch cluster where the index is located.

Notice

Currently, only historical data import is supported. Data written to Elasticsearch after the import task is successfully created can not be imported into TLS. Once historical data import is complete, the task will automatically end.

You can only import data into TLS via the public network. If your data source is Volcengine Elasticsearch or a self-managed Elasticsearch cluster on Volcengine ECS, you need to enable public access for the cluster.

After creating an import task, it is recommended to enable indexing for fields to facilitate preview and query analysis of imported log data.

If a field key in Elasticsearch starts with an underscore (_), TLS will automatically add the tag prefix to it for subsequent indexing, log query, and analysis.

Limits

Items	Description
Individual data entry size	The maximum size for a single imported data entry is 1 MiB. If the data size exceeds this limit during import, the data will be discarded.
Number of data import configurations	Each project can have up to 100 different types of data import tasks.
Import data scope	Only historical data can be imported. Data written to Elasticsearch after a successful import task creation can not be imported to TLS. Once historical data import is complete, the task will automatically end.
Number of concurrent tasks	The import task uses Scroll mode to pull data from Elasticsearch. Because Elasticsearch limits the number of Scroll requests, the total number of shards for all import tasks in an Elasticsearch cluster must not exceed this limit. Otherwise, some import tasks will fail. This limit is specified by the search.max_open_scroll_context parameter in the Elasticsearch cluster and defaults to 500.
Edit import task	After creating a log import task, you can not modify the task's data source or other configurations. If the configuration is incorrect, delete it and create a new one.

Performance description

The service performance of the ElasticSearch data import feature depends on multiple factors. Main factors include:

Complexity of ES queries: You need to configure ES queries in the import task to filter data. The more complex the ES queries configured in the import task, the longer the ES server takes to process the request, which affects the data reading speed of the current import task.

Target topic write speed: The number of shards in the target log topic determines the data write speed. The maximum write rate per shard in a log topic is 5 MiB/s. We recommend that you evaluate the required number of shards in the target log topic based on the data volume, such as by enabling the auto split feature or adding shards in time when handling large volumes of data. For more information about shards, see Manage shards.

Billing description

Importing data from ElasticSearch generates TLS data storage fees and internal bandwidth fees. For detailed pricing information, see TLS Billing items based on pay-by-feature mode.

Billing item	Description
Write traffic	ElasticSearch writes data to TLS and this process incurs write traffic fees in TLS. Traffic fees are calculated based on the compressed data volume.
Log storage	After logs are stored in TLS, the backend automatically compresses them. Storage fees are calculated based on the actual compressed size.
Others	We recommend that you enable indexing for imported data to facilitate future queries, analysis, and data statistics. Enabling indexing incurs index storage fees. Storage fees are calculated based on the space occupied by indexes that are created from the log data before compression. If imported data is later read, transformed, or shipped, charges will apply according to the billing standards.

Prerequisites

An Elasticsearch cluster is created.

A log project and log topic are created. For detailed steps, see Create resources.

Before importing data from Alibaba Cloud or Tencent Cloud Elasticsearch, configure an IP address allowlist on the source node to allow the log collection data import task to access data on the source node. For the IP address list, see IP address whitelist.

Before using the log import or log shipping features for the first time, you must complete cloud resource access authorization to grant TLS permission to access and write resources. If authorization has not yet been granted, TLS will provide guidance when you create a log import task in the console. We recommend using the Volcanoengine primary account for authorization. For more information, see Cross-service access authorization.

Create an ES data import task

Step 1 Configure basic information

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the Data import section, select Elasticsearch.

Fill in Basic information configuration.

Configuration	Description
Log project	It is the project which stores Elasticsearch data.
Log topic	It is the log topic which stores Elasticsearch data. Notice Exclusive log topics such as cloud product logs and service logs do not support data import.

Step 2 Configure data source

Fill in the data source configuration.

Configuration	Description
Task name	Name of the log import task.
ES access URL	The service address of the Elasticsearch cluster, in URL format: http://host:port/.
User name	Elasticsearch user name. Only required to set when user authentication is enabled for the Elasticsearch cluster.
Password	Elasticsearch user password. Only required to set when user authentication is enabled for the Elasticsearch cluster.
Index list	Indexes to be imported into TLS. You can set multiple indexes. Indesxes should be separated by commas (","), for example indexA,indexB,indexC.
ES query statement	Use a query statement to filter data. Only data that meets the condition will be imported into TLS. Set to * or leave it blank to import all data without filtering. The query statement must follow the Elasticsearch query_string format. For example: project:tls and owner:zhangsan. For details, see Query string query.
Time field	Specify whether to use a field from Elasticsearch as the log time. By default, TLS uses the system time at data import as the log timestamp. You can also specify the column name in the Elasticsearch index that represents time, and TLS will use the value of this field as the log timestamp. After selecting Specify time field, configure the following settings: Time field name: The Key of the time field. Time field format: The format of the time field used to parse its Value. For supported time format settings, see Time format. For example, if the time field content is 1651648409, set the time field format as %s; if the time field content is 1651648416882, set the time field format as %F; if the time field content is 2022-01-01T15:30:00.999+0800, set the time field format as %Y-%m-%dT%H:%M:%S.%f%z. Import time range: The start time and end time of the data to be imported. Only data within the time range will be imported into TLS. Time zone: The time zone corresponding to the time field.
Import mode	Select a data import mode. Currently, only importing historical logs is supported. Data written to ES after successfully creating the import task can not be imported into TLS. Once historical data import is complete, the task will automatically end.
Preview	Click Preview to view the data import result based on the current configuration. If the previewed data does not meet expectations, modify your configuration and try again.

After confirming the configuration, click Next.

Step 3 Check index configuration

Check index configuration.

Follow the prompts on the page to set or update the index as needed. You can also click Import existing index configuration to import indexes from other log topics.

If this log topic does not have an index configured, you can set one. For details, see Configure indexes.

If the index for this log topic is already configured, you can decide whether to update the index based on the log fields parsed by the collection rule.

Notice

You must enable the index to query and analyze the collected log data.

The updated index settings only apply to new data written afterward. For more instructions and limitations on query and analysis, see Query overview and Analysis overview.

Click Next step: check index configuration.

Verify data import results

In the left navigation pane, select Common functions > Query and analysis

Select the log project and log topic that stores Elasticsearch data from the list on the left side of the Query and analysis page.

Preview data on the Query and analysis page.

About one minute after the data is written, you can preview it on the Query and analysis page in TLS. The latest log data from the past 5 minutes is displayed by default. You can also set other time ranges or query statements. If you successfully preview or query the data imported into the ES cluster, the import is successful.

You can also view the status of import tasks on the data import task list page. For details, see View import tasks.

View import tasks

View configuration details

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the Data Import tab's Task List section, click the name of the target import task.

View task configuration information.

The details page of the log import task displays basic information and data source configuration.

Display item		Description
Basic information	Task name	The name of the ES data import task.
		Task ID	The ID of the ES data import task.
		Import task status	The status of the import task. For more details, see Task status.
		Task type	The type of import task. Elasticsearch means importing ES data.
		TLS Topic ID	The log topic where the imported data is stored. You can configure the index based on the prompts on the page, so that you can perform query analysis and data statistics later.
		Create on	The creation time of the import task.
Data source configuration	ES access URL	The service address of the Elasticsearch cluster.
		User name	Elasticsearch user name.
		Password	Elasticsearch password.
		Index list	Indexes to be imported into TLS.
		ES query statement	Query statements used to filter data.
		Import mode	Mode for data import. Currently, only importing historical logs is supported. Data written to ES after the import task is created cannot be imported into TLS.
		Specified time period	Specify whether to use a column from the Elasticsearch index to represent time and use its value as the log timestamp.
		Name of the time field	Key of the time field Displayed only when Specified time period is enabled.
		Time field format	Format of the time field. Displayed only when Specified time period is enabled.
		Import time range	Start time and end time of the data to be imported. Displayed only when Specified time period is enabled.
		Time zone	The time zone corresponding to the time field. Displayed only when Specified time period is enabled.

Task status

The description of import task status:

Status	Description
Importing	The import task is in progress. You can monitor the progress of the current task by checking the BytesTransferred and BytesTotal fields in the service logs.
Import completed	Import task is completed. All data meeting the import rules has been successfully imported into the specified log topic in TLS.
Import exception	Import task exception. If any data import fails, the entire task will be marked as abnormal. For example, the index import failures caused by index deletion or network issues in ES clusters. After you resolve the exception, we recommend re-importing the problematic data. Additionally, you can check service logs for detailed information about the problem index. For log fields, see Service log overview.

Copy import task

TLS supports copying data import tasks. If a similar data import task exists, you can copy all the configurations from an existing task, and then adjust the settings as needed to create a new data import task.

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the Data import tab under the Task list section, click Copy under the Actions column for the task to be copied.

Refer to the steps for creating an import task to confirm or modify the relevant parameter settings.

Delete imported tasks

If there is a failure or anomaly with a data import task, you can delete the task and re-import data. If you have finished migrating historical log data, or no longer need to import logs to TLS, you can follow the steps below to delete log import tasks.

Notice

Deleting a log import task does not affect logs that have already been imported successfully.

A deleted log import task can not be restored. Proceed with caution.

Select the region where torch log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the task list section under data import tab, locate the log import task you need to delete, and click Delete in its Actions column.

Confirm the information for the task to be deleted in the popup dialog, and then click Confirm.

Troubleshooting and problem diagnosis

FAQs	Possible causes	How to fix it
An error occurs when previewing logs: time out.	The Elasticsearch URL in the import task is incorrect or invalid. No IP allowlist is configured, and the Elasticsearch cluster blocks access from Volcengine TLS.	Check whether the Elasticsearch URL used for import is correct. Check if the public IP address of TLS is properly configured in the Elasticsearch allowlist. For IP address list, see IP address whitelist.
Error shows when previewing logs: invalid argument.	The query_string statement used is invalid. The username and password used to access Elasticsearch are invalid.	Check whether the query statement follows the Elasticsearch specification. Check the Elasticsearch username and password configured in the task.
Log time does not match expectations during preview.	The log time field is not configured or the field format is incorrect.	Enable the specified time period configuration and set the time field format correctly. For supported time format settings, see Time format.
Some log entries can not be queried after data import.	If you create log indexes after importing data, indexes will not apply to logs imported before index settings were enabled.	Log indexes only take effect on logs collected after index configuration is enabled. New index configurations also only apply to subsequent logs. We recommend configuring an index for the target Topic before creating new import tasks.

Last updated: 2026.06.08 11:15:04