Docs

Torch Log Service

Ship logs to TOS

Ship data in Parquet format

TLS supports shipping logs to TOS. This topic uses Parquet format as an example and outlines the steps to configure log shipping.

Billing notes

Shipping logs in Parquet format is billed based on the advanced log shipping traffic mode. For details, see Billing items based on pay-by-feature mode.

Prerequisites

TLS has been enabled and log data has been successfully collected. For more details, see Overview.

Volcengine Object Storage has been enabled, and a bucket has been created in the same region as the specified log topic. For more details, see Creating bucket.

TLS needs to assume the ServiceRoleForTLS role to enable cross-service access to other cloud resources. You must grant TLS permission to access and write to TOS resources.

If you have not yet created the ServiceRoleForTLS role, TLS will guide you through the process when you create the shipping configuration. We recommend authorizing the Volcanoengine account. For details, see Cross-service access authorization.

Steps

Select the region where log service is located from the top navigation bar.

On the left navigation bar, select Data processing > Log shipping.

On the Log shipping page, click Create TOS shipping configuration.

On the Create TOS shipping configuration page, complete the following settings.

Authorization settings.

If shipping to the same account, keep the default settings.

If shipping across accounts, configure authorization information.

Parameters	Description
Delivery target	Select Deliver to another account.
IAM Role (TRN)	Enter the TRN for a custom role to grant permission for the shipping task to assume this role and deliver data across accounts to the TOS Bucket. This custom role must have permission to deliver data. For instructions on creating a role and granting permissions, see Configure cross-account delivery permissions.

Configure basic information.

Parameters	Description
Configuration name	Name of the log shipping configuration.
Source project	Select the project where the data source is located from the dropdown list.
Source topic	Select the topic where the data source is located from the dropdown list.

Configure log shipping target information.

Parameters	Description
Target bucket	Select a TOS bucket, which must be in the same region as the source topic. If you do not have a suitable bucket, you can follow the prompts on the page to create a new one.
Shipping directory	The top-level directory name of the bucket. All log data shipped according to this configuration will be shipped to this directory.
Shard format	Subdirectory naming format in the bucket. TLS dynamically generates subdirectories in the specified shipping directory based on the creation time of the shipping configuration. The subdirectory naming format supports strftime syntax.

The following shows examples of shard formats and corresponding TOS bucket directory structures:

Shard format values	Corresponding TOS bucket directory structure (using 2022/06/30/14/00 as an example)
%Y/%m/%d/%H/%M (default value)	tos://bucket-name/top-level-directory-name/2022/06/30/14/00/
%Y%m%d%H	tos://bucket-name/top-level-directory-name/2022063014/
%Y%m%d	tos://bucket-name/top-level-directory-name/20220630/
ds=%Y%m%d/%H	tos://bucket-name/top-level-directory-name/ds=20220630/14/
year=%Y/mon=%m/day=%d/log_%H%M	tos://bucket-name/top-level-directory-name/year=2022/mon=06/day=30/log_1400/

(Optional) Configure advanced settings for log shipping target.

Parameters

Description

Set the shard based on the value

Defines the naming format for subdirectories in the bucket. This directory will appear as either the parent or child of the time shard subdirectory.

Disabled by default. After enabling, select the level for this directory and set the corresponding key and value regular expressions.

Key (required): Select the key used for shard shipping of the value.

Value regular expression (optional):

Set the regular expression to extract the value of the corresponding key. The extracted value will serve as the name of the subdirectory in the bucket.

If not set, all fields for the corresponding key will be automatically extracted, and each field will be used as a subdirectory name in the bucket.

Notice

This may generate too many small files. Up to 100 shards are supported. Extra files will be shipped to the subdirectory named others. Configure with caution.

If the configured key does not exist or regular expression extraction fails, the setting will not take effect.

Example:

Original log: {"name":"Li Si","address":"Guangdong Province, Shenzhen XXX Science Park","phone":"13xxxxxx321"}

Configuration example:

Shipping result:

In the TOS destination bucket, the data will be shipped to the corresponding directory based on whether the regular expression (.*?province) is configured.

Set regular expression: tos://if-tls-test/No1-test/Guangdong Province/2022/06/30/14/00/.

No regular expression configured: tos://if-tls-test/No1-test/Guangdong Province Shenzhen xxx Technology Park/2022/06/30/14/00/.

Configure log shipping formats.

Parameters	Description
Compression type	Log data can be compressed using a specified compression format before being shipped to object storage TOS. This can help to save read traffic costs. Supports uncompressed, snappy, gzip, and lz4.
Shipping format	The format in which data is stored after being shipped to TOS. This topic uses the Parquet format as an example. After successful shipping, the Parquet file header contains all shipping fields.
Shipping fields	When shipping in Parquet format, you can select the fields to be delivered.

Configure shipping rules.

Shipping rules specify the trigger conditions for log shipping actions. After you create the shipping configuration, TLS continues to record the interval since the last shipping. When the specified interval is reached, TLS packages and ships log data from each shard to TOS separately. If the volume of new data in a shard exceeds the specified file size for shipping, TLS automatically splits the files to meet the size requirement.

Tip

Each delivered file corresponds to an object in TOS.

The delivered file size refers to the volume of original data before compression. If compression is enabled, the files saved in TOS after shipping will be smaller than the delivered file size, reducing storage costs.

Parameters	Description
Delivered file size	The original file size to be delivered for each shard, which means the uncompressed log file size. The unit is MiB, and the value range is 5–256.
Time interval of log shipping	The interval between log shipping actions. The unit is seconds. The value ranges from 60 to 900.
Shipping time range	Deliver log data within the specified time range. Start time: The earliest write time of the log data to be delivered. The default start time is the current moment. You can choose a historical time, which means the log data that collected before shipping rule was created can also be shipped to TOS. End time: The latest write time of the log data to be shipped. TLS does not ship deliver log data collected after this time. If the end time is left empty, the most recent log data will continue to be shipped. Notice Once the delivery configuration is created, the delivery time range can not be modified.

Click create.

Next steps

Once the shipping configuration is created successfully, TLS automatically sets the task status to enabled and triggers a log shipping task. You can view details such as the shipping configuration name on the log shipping page. For more information, see View shipping configuration and task details; you can also view shipped log data directly in TOS console.

Last updated: 2026.04.27 11:28:01