Docs

Torch Log Service

Data import

Import logs from TOS

TLS supports TOS data import, allowing you to structure the data stored in TOS and save it in TLS. This topic describes the steps to import data from TOS to TLS.

Background information

For bulk import of large volumes of log files, you can do the following: upload log files to a TOS bucket; use TLS's log import feature to structure; import, and save files in TLS. Then you can perform real-time query analysis and further processing to log data. When importing data from TOS, you can import all data from one or more objects in a specified bucket into TLS. Cross-region import is supported, meaning the TLS topic and source bucket can be located in different regions. After creating an import task, you can view the progress and status in TLS console, or use service log analysis for detailed progress and status data.

TLS also adds the following metadata fields to imported log data.

Field	Description
__content__	Original log data. Only add when log extraction type is set to Single-line full text.
__path__	Source file directory and file name for the log.
__source__	Name of the storage bucket where the log source is located.

Notice

The following are the limitations of the data import feature from TOS:

Limits	Description
Import performance	The maximum processing speed supported by TOS data import tasks is 80 MB/s. If you need to increase the processing speed, submit your business requirements through the ticketing system.
Number of data import configurations	You can create up to 100 different types of data import configurations in a single project.
Single file size	The file size limit varies by compression mode. Files that exceed the limit will not be imported into TLS. Snappy compression format (non-framing-format): The file size limit is 350 MiB. Other compression formats or uncompressed files: The file size limit is 5 GiB.
Individual data entry size	The maximum log entry length is 512 KiB. Logs exceeding this size will be truncated and discarded; if a log entry exceeds 2 MiB, it will be directly discarded.
Import data scope	Only full import is supported; incremental import is not available. After creating the import task, TLS will not collect incremental content from newly added files in the bucket that meet the import rules. If a file is updated and has not yet been collected, TLS will collect the latest version of the file.
Bucket type	Importing data from deep cold archive storage buckets is not supported.
Edit import task	After creating a log import task, you can not modify the data source configuration or other settings for the task. If the configuration is incorrect, delete it and create a new one.

Billing description

Importing data from TOS involves charges for TLS and TOS, including read traffic and write traffic. For detailed pricing information, see TLS Billing items based on pay-by-feature mode and TOS Billing Overview.

Billing side	Billing item	Description
TOS	Private network traffic outflow	When reading data from TOS via the Volcanoengine private network, private network traffic outflow is generated.
TOS		Number of requests	When reading data from TOS, billing is based on the number of requests for different storage types.
TLS	Write traffic	When TOS data is written to TLS, TLS write traffic fee is generated. Traffic fees are calculated based on the compressed data volume.
		Log storage	After logs are stored in TLS, the backend automatically compresses them. Storage fees are calculated based on the actual compressed size.
		Others	We recommend that you enable indexing for imported data to facilitate future queries, analysis, and data statistics. Enabling indexing incurs index storage fees. Storage fees are calculated based on the space occupied by indexes that are created from the log data before compression. If imported data is later read, transformed, or shipped, charges will apply according to the billing standards.

Prerequisites

Log files have been uploaded to a TOS bucket. For detailed steps, see Upload logs.

A log project and log topic are created. For detailed steps, see Create resources.

Before using the log import or log shipping features for the first time, you must complete cloud resource access authorization to grant TLS permission to access and write resources. If authorization has not yet been granted, TLS will provide guidance when you create a log import task in the console. We recommend using the Volcanoengine primary account for authorization. For more information, see Cross-service access authorization.

Steps

Step 1 Select a log space

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

. On the Data import tab, click Volcano Engine TOS.

Select a log space, then click Next step: configure import rules.

Configuration	Description
Log project	Name of the project used to store TOS data.
Log topic	Name of the log topic used to store TOS data. Exclusive log topics for cloud product logs and service logs do not support data import.

Step 2 Configure import rules

Enter the task name.

Set the name for the log import task.

Configure the data source.

Configuration	Description
Region where the bucket is located	Region where the TOS bucket is located. Supports cross-region data import.
IAM project of the bucket	IAM project to which the TOS bucket belongs.
Bucket	TOS bucket that contains log files.
Select file path	The path of the file to be imported in the TOS bucket. Select file path: Select a path or file directly from the dropdown list. Enter path manually: Enter the file prefix or the complete path. For example, if you set the file prefix as accesslog/, all files under the directory accesslog/ will be imported. Notice File prefixes do not support wildcards. Leaving the path blank means importing all files in the bucket. If too many files exist in the bucket, traversing the bucket may reduce data import efficiency.
Compression mode	The compression mode for data in the TOS bucket. TLS supports importing data in the following compression modes. Compression mode: No compression, snappy, gzip, lz4, tar.gz. Notice Before setting lz4, make sure the data source uses LZ4Frame compression; otherwise, decompression may fail during import. The tar.gz compression mode is only available for preview. If you have relevant business needs, contact your account manager for allowlist permission.

Configure the import method.

Configuration	Description
Log extraction type	TLS supports the following log parsing types: Single-line: Each line is parsed as a log; uses \n as the end mark. After import, logs are stored in the default field __content__ and are not further structured. Multi-line: When importing multi-line logs, you can specify a regular expression to match the first line of each log entry. JSON: Extract logs into multiple key-value pairs based on the JSON format. The top-level object keys serve as field names; top-level object values serve as field values; \n serves as the end of each log entry. The log attribute time is determined by the log collection time. CSV: Each log entry ends with \n as the end marker. Besides, each log entry can be structured using a specified separator. The log attribute time is determined by the log collection time. In this case, additional parameters are required. For details, see the table below.
Import archived files	If you are importing TOS files stored as Archive or Cold Archive, you need to enable this feature. After enabling, the system will first thaw the archived files. Once thawing is complete, preview the files. Importing Deep Cold Archive storage files is not currently supported. Enable: Only enable when importing Archive or Cold Archive files. Archive file: Thawing and restore takes about 1 minute. Cold Archive file: Fast retrieval mode is used for thawing and restore. If an error is reported during preview, you can choose to skip preview or try again later. Disable: Disable in other cases.
Upload failed parsing logs	Choose whether to upload logs that failed to parse. This feature is enabled by default. Enable: All logs that fail parsing will be uploaded to TLS, with the specified field as the key name and the original log content as the value. The key name can be specified via Failure log key name. It is defaulted as LogParseFailed. Disable: Logs with parsing failures will not be uploaded to TLS.
Failure log key name	You can configure this field after you enable uploading failed parsing logs. By default, it is LogParseFailed. All logs that fail parsing will be uploaded to TLS, with the specified field as the key name and the original log content as the key value.
Time field	This configuration determines whether to use the value from the specified field as the log timestamp. By default, TLS uses the system time at data import as the log timestamp. You can also specify a time field, and TLS will use the value of this field as the log timestamp. The related configuration is as follows: Time key name: The Key of the time field. Time regex: Use part of the value in the time field as the timestamp. When the log extraction type is set to single-line full text or multi-line full text, you must use a regular expression to extract the time from the log. When the log extraction type is set to CSV or JSON and you need to extract part of the time field, you can also use a regular expression for extraction. Time conversion format: Used to parse the value of the time field. For instructions on time formats, see Time format. Example: When the time field is 1651648409, configure it as %s. When the time field is 1651648416882, configure it as %F. When the time field is 2022-01-01T15:30:00.999+0800, configure it as %Y-%m-%dT%H:%M:%S.%f%z. Time zone: Time zone can be set here. Supports GMT or UTC. Notice If you need to support daylight saving time or standard time, select UTC; otherwise, select GMT.

If log extraction type is set to multi-line, you also need to configure the following additional parameters:

Parameters	Description
Log example	We recommend that you provide at least two log samples to ensure the first-line regular expression can match the first line of every log. We suggest that you use real samples from the production environment.
First-line regular expression	This regular expression is used to identify the first line in each log entry, and the matched part is identified as the start of the log. Supports automatic generation or manual input. After configuration, the page displays the number of first lines matched in the log examples. If this matches the number of actual log entries, the first-line regular expression is correct. Auto-generate: Automatically generates the first-line regular expression based on the entered log examples. The highlighted part in the example shows what the first-line regular expression matches. Manual input: Customize the first-line regular expression. TLS matches log samples using the specified first-line regular expression and identifies the matched part as the start of a log entry.

If log extraction type is set to CSV, you also need to configure the following additional parameters:

Parameters	Description
Separator	It is the separator between different fields in log data. Select the correct separator according to the log format to ensure every field can be accurately extracted. TLS supports configuring various common separators, such as spaces and commas. When your log data uses a separator that is not in the options, you can set separator to custom and specify the corresponding separator in custom separator. Notice Setting the separator as an invisible character is not supported yet. Only one separator can be specified, but a separator can contain multiple characters. For example, you can use double colons (::).
Quotation mark	If a log field value contains separators, enclose the entire field value within the specified quote character in the log; otherwise, the content may be incorrectly split by the separator. After you specify a separator and quote character, any content included in the quote character will be parsed by TLS as a complete field. For example, if a raw log is 20220511 debug "parse meet error" and the separator is set as space, then: If no quote character is set, the log will be split into key1: 20220511, key2: debug, key3: "parse, key4: meet, key5: error". If the quote character is set to double quotes, the log will be split into key1: 20220511, key2: debug, key3: "parse meet error". Supported quote characters include do not configure, single quote, double quotes, or custom. If you choose a custom character, you need to set a custom quote. Notice Setting the quote as an invisible character is not supported yet. If a field value contains a quote, you can use the escape character \ to escape it. The escaped quote will be treated as regular text. For example, if the quote character is a double quotation mark, but the log entry 2022"0511" contains double quotes, you can use 2022\"0511\" to escape them. Only one quote character can be specified, but the quote can contain multiple characters, such as double colons (::).
Field name	After TLS parses and extracts log content using the specified separator, you need to assign a different field name to each log entry. Content without an assigned field name will not be imported into TLS. Plus, field names must not be duplicated. Supported configuration methods: First line as field name: The first line of the CSV file is used as field names. Usually, the first line in a CSV file serves as the header for each data column. Custom field: If you select custom field, you can set field names manually. Multiple names can be separated with commas (,).
Skip line count	Specify the number of log lines to skip. For example, if you set it to 1, logs will be imported starting from the second line in the CSV file.

Click Execution preview to view the content of the file to be imported.

After configuring the import rules, TLS provides data preview from the specified data source so you can check whether the data in the preview area matches the data to be imported. If the data is different, it means the data source configuration is incorrect. We recommend reconfiguring it.

Click Next step: check index configuration.

Step 3 Check index configuration

Check index configuration.

Follow the prompts on the page to set or update the index as needed. You can also click Import existing index configuration to import indexes from other log topics.

If this log topic does not have an index configured, you can set one. For details, see Configure indexes.

If the index for this log topic is already configured, you can decide whether to update the index based on the log fields parsed by the collection rule.

Notice

You must enable the index to query and analyze the collected log data.

The updated index settings only apply to new data written afterward. For more instructions and limitations on query and analysis, see Query overview and Analysis overview.

Click Next step: check index configuration.

Error handling mechanism

Error item	Processing mechanism
Failed to read file	When reading a file, if the file is incomplete (due to issues like network issues or file corruption), the import task will automatically retry. If reading fails after three retries, the file will be skipped.
Large file	If a single file exceeds 5GB when reading, the import task will skip that file directly.
Compression format parsing error	If the file compression format is invalid or the wrong type is selected when decompressing, the import task will skip that file directly.
Data parsing error	When a data parsing error occurs, all logs that fail to parse are uploaded to TLS with the specified field as the key name and the original log content as the value. The key name can be specified via Failure log key name. It is defaulted as LogParseFailed.
No TOS bucket exists	If the TOS bucket specified in the import task does not exist, the import task will fail.

Related operations

After successfully creating the TOS data import task, you can view relevant information in the TLS console, including configuration details of the import setup and the progress of data import.

View configuration details

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

Then on the Data import tab under Task list, click the log import task name.

On the Configuration details tab, view the task's configuration information.

You can view the basic information, data source configuration, and log extraction mode for the import task.

View import progress

The Import task information tab displays the status and progress of the import task in detail as follows:

Display item	Description
Import task status	Status of the current import task. For detailed status descriptions, refer to the following table.
Import progress	Current task import progress is calculated as Import progress = (Number of imported resources + Number of skipped resources + Number of failed resources + Number of nonexistent resources) / Total number of resources.
Total number of resources	The number of files in the TOS bucket detected by TLS that meet the import rules.
Number of imported resources	Number of files imported by TLS.
Size of enumerated resources	Total size of files in the TOS bucket detected by TLS that meet the import rules, measured in GiB.
Size of imported resources	Size of files imported by TLS, measured in GiB.
Number of nonexistent resources	The number of files that TLS failed to locate due to file deletion during the import process.
Number of skipped resources during import	The number of files TLS skipped and did not import due to resource type mismatches, extraction failures, or unreadable files.
Number of failed resources	The number of files TLS failed to import.

Import task status description:

Status	Description
Importing	The import task is in progress. You can view the current progress of the task in Import progress.
Import completed	Import task is completed. All importable files that meet the import rules have been successfully imported into the specified log topic in TLS.
Import exception	Import task exception. If any file fails to import, the entire task will be marked as exception. File import may fail due to reasons such as incorrect decompression methods or data anomalies. We recommend that you resolve the exception and then re-import the affected files. Additionally, you can view detailed information about the problematic file in the service logs. For log fields, see Data import error logs.

Copy import task

TLS supports copying data import tasks. If a similar data import task exists, you can copy all the configurations from an existing task, and then adjust the settings as needed to create a new data import task.

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the Data import tab under the Task list section, click Copy under the Actions column for the task to be copied.

Refer to the steps for creating an import task to confirm or modify the relevant parameter settings.

Delete imported tasks

If there is a failure or anomaly with a data import task, you can delete the task and re-import data. If you have finished migrating historical log data, or no longer need to import logs to TLS, you can follow the steps below to delete log import tasks.

Notice

Deleting a log import task does not affect logs that have already been imported successfully.

A deleted log import task can not be restored. Proceed with caution.

Select the region where torch log service is located from the top navigation bar.

On the left navigation bar, select Common functions > Log Collection.

In the task list section under data import tab, locate the log import task you need to delete, and click Delete in its Actions column.

Confirm the information for the task to be deleted in the popup dialog, and then click Confirm.

Last updated: 2026.06.10 14:12:23