Docs

Torch Log Service

Data import

Import logs from SLS

TLS provides multiple cross-service data migration options. It supports exporting structured log data by consuming logs from Alibaba Cloud Simple Log Service (SLS) within a specified time frame and writing them into TLS.

Background information

In scenarios such as cross-service data migration, active/standby switch, and log archiving, you need to consume collected log data from Alibaba Cloud SLS and write it into TLS for log query, analysis, consumption, or long-term archiving. When importing logs from SLS, TLS pulls and consumes log data from SLS using the consumer group feature provided by SLS, uploads the data to TLS through the PutLogs API, and writes the data from the specified Logstore in SLS to the designated log topic. Log importing supports specifying the starting position for consumption, which means historical data or newly added logs in real time can be consumed.

When collecting logs in SLS, reserved fields such as __time__ added to the original logs are supported for import to TLS. Fields like __time__ and __source__ are imported directly by default. Other fields such as __tag__:_pod_name_ will be imported after renaming, while unsupported fields will not be imported.

The supported import fields include:

SLS metadata fields	VolcEngine TLS corresponding fields
__time__	__time__
__source__	__source__
__tag__:_pod_name_	__pod_name__
__tag__:_container_name_	__container_name__
__tag__:_pod_uid_	__pod_uid__
__tag__:_container_source_	__container_source__
__tag__:_container_ip_	__container_ip__
__tag__:_namespace_	__namespace__
__tag__:_image_name_	__image_name__
__tag__:__path__	__path__
content	__content__

Notice

To ensure data security, we recommend using Alibaba Cloud RAM sub-account keys for authentication when creating log import tasks. The RAM sub-account should have relevant permissions for consumer groups. You can refer to SLS RAM Authorization policy example to write authorization policies. For steps to grant permissions, see Grant permissions to RAM users. You can also customize RAM authorization policies based on your business needs. For consumer group-related actions, see Grant permissions to RAM users.

After creating a log import task, you can not modify the task's data source or other configurations. If the configuration is incorrect, delete it and create a new one.

Each log topic can only be linked to one SLS log import task. Plus, writing logs from multiple Logstores into the same log topic is not supported.

When importing logs, the shards in the SLS Logstore do not correspond one-to-one with the partitions of the log topic.

After you create an SLS log import task successfully, Volcengine TLS will create a consumer group on the SLS side to consume log data. Please note that after creation you can not delete or edit this consumer group in SLS, including operations like resetting the consumption position or deleting the group. Otherwise, duplicate consumption or consumption failures may occur.

Prerequisites

You have created a project and Logstore in Alibaba Cloud (SLS) and collected log data. For detailed instructions, see Quick Start.

You have created a project and log topic in Volcanoengine TLS. The number of shards in the log topic must match the number of shards in Logstore.

Before using the log import or log shipping features for the first time, you must complete cloud resource access authorization to grant TLS permission to access and write resources. If authorization has not yet been granted, TLS will provide guidance when you create a log import task in the console. We recommend using the Volcanoengine primary account for authorization. For more information, see Cross-service access authorization.

Import logs from SLS

The steps for creating an SLS log import task are as follows:

Step 1 Initial configuration

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log collection.

Under the SLS log migration tab, click Alibaba Cloud SLS log migration.

Enter Basic information.

Configuration	Description
Import type	The type of log import task. By default, it is cloud service log.
Log project	The project used to store SLS data.
Log topic name	The log topic used to store SLS data.

Step 2 Configure data source

Follow the page prompts to complete the configuration below.

Configuration	Description
Task name	Name of the log import task.
Data source	The data source of the log. By default it is Alibaba Cloud.
Network type	The network type for data import supports transmission via public or private network. Tip If you need to transfer data through a private network, contact technical support for your requirements via the Ticket system first.
Domain name	The domain name for Alibaba Cloud SLS. You can view it on the overview page of the log source project.
Project	Log source project name.
Logstore	Log source Logstore name.
Consumer group	Consumer group name. Each Logstore can create up to 30 consumer groups, and consumer group names must be unique.
Key	Alibaba Cloud account access keys, including AK and SK. Volcengine TLS will use this account to authenticate with Alibaba Cloud SLS, and read and consume log data from specified projects and Logstores. Tip We recommend using a RAM user for authentication. The user should have the necessary consumer group permissions. For an example authorization policy, see SLS RAM Authorization policy example.
Starting position for consumption	The starting position for first-time log consumption can be set as: Earliest data: (Default) Start consuming from the earliest log entry in the Logstore. Latest data: Start consuming from the latest log entry in the Logstore. Custom time data: Consume data starting from the specified time point. You can set log retention period in Logstore. Logs that exceed this period will be automatically deleted. The specified time point must be within the log retention period. Otherwise, data consumption will not work properly.
Consume data in order	Enable the ordered consumption feature. By default, it is enabled. After enabling, consume data in order within the Shard. After a Shard splits, the data from the original Shard will be consumed first. Then the data from the new Shards generated by the split will be consumed simultaneously. After Shards merge, the data from the original Shard will be consumed first. Then the data from the new Shard will be consumed. After disabling, data is consumed out of order. All logs will be consumed immediately, including those new Shards generated by Shard splitting or merging. At this time, the logs written to Volcengine TLS may be out of order.

After confirming the configuration, click Next.

Step 3 Check index configuration

Check index configuration.

Follow the prompts on the page to set or update the index as needed. You can also click Import existing index configuration to import indexes from other log topics.

If this log topic does not have an index configured, you can set one. For details, see Configure indexes.

If the index for this log topic is already configured, you can decide whether to update the index based on the log fields parsed by the collection rule.

Notice

You must enable the index to query and analyze the collected log data.

The updated index settings only apply to new data written afterward. For more instructions and limitations on query and analysis, see Query overview and Analysis overview.

Click Next step: check index configuration.

Notice

Once you create a log import task, TLS will begin consuming log data in SLS and writing it in real time to the specified log topic. You can enable and configure the index for log data to facilitate real-time query and analysis.

View log import tasks

After creating a log import task, you can view basic information and data source details for configured import tasks on the task details page. You can also check the consumption status of each shard in the TLS console.

View task information

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log collection.

In the Data import tab, click the name of the target import task in the Task List section.

View the basic information of the task.

The log import task details page displays the task name, task topic, import status, and other basic information, as well as data source, consumer group, network type, and other data source information.

Import status description:

Status	Description
Running	The log import task is running normally.
The SLS resource does not exist.	The Project or Logstore in SLS has been deleted. Consumption has stopped and cannot be recovered; you need to recreate the log import task.
SLS authentication failed.	SLS account or permission is abnormal, such as account arrears or deletion of related permissions. After the account or permissions are restored, consumption will resume shortly.
Unknown SLS exception.	SLS server exception. The server will resume to normal once the service is restored.
TLS resource does not exist.	The project or topic for TLS has been deleted Consumption has stopped and cannot be recovered; you need to recreate the log import task.
TLS authentication failed	The account or permissions for TLS are abnormal, such as account arrears or permissions related to account deletion. After the account or permissions are restored, consumption will resume shortly.

View consumption status and monitoring information

The Alibaba Cloud SLS console provides real-time display of consumption status. You can check the consumption progress in the following ways:

On the Data consumption page, view the progress of consumption data for each shard. For details, see View consumer group status in the console.

Check consumption status in the Consumer group monitoring dashboard, including Top 10 consumer group latency, consumption lag duration, consumer group list, and more. For details, see Consumer group monitoring and alarms.

Delete log import task

If there is a failure or anomaly with a data import task, you can delete the task and re-import data. If you have finished migrating historical log data, or no longer need to import logs to TLS, you can follow the steps below to delete log import tasks.

Notice

Deleting a log import task does not affect logs that have already been imported successfully.

A deleted log import task can not be restored. Proceed with caution.

Select the region where torch log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the task list section under data import tab, locate the log import task you need to delete, and click Delete in its Actions column.

Confirm the information for the task to be deleted in the popup dialog, and then click Confirm.

SLS RAM Authorization policy example

If you use RAM users to authenticate with Alibaba Cloud SLS and read or consume log data, you need to grant the RAM user permissions for the consumer group. The recommended authorization policy is as follows:

Tip

Replace ${projectName} and ${logstoreName} with the actual Project and Logstore names.

{​
  "Version": "1",​
  "Statement": [​
    {​
      "Effect": "Allow",​
      "Action": [​
        "log:GetCursorOrData",​
        "log:GetLogStore"​
      ],​
      "Resource": "acs:log:*:*:project/${projectName}/logstore/${logstoreName}"​
    },​
    {​
      "Effect": "Allow",​
      "Action": [​
        "log:CreateConsumerGroup",​
        "log:ListConsumerGroup",​
        "log:ConsumerGroupUpdateCheckPoint",​
        "log:ConsumerGroupHeartBeat",​
        "log:UpdateConsumerGroup",​
        "log:GetConsumerGroupCheckPoint"​
      ],​
      "Resource": "acs:log:*:*:project/${projectName}/logstore/${logstoreName}/consumergroup/*"​
    }​
  ]​
}​

Last updated: 2026.06.08 11:12:50