Torch Log Service
Torch Log Service
Torch Log Service
Document
Product News
Product Overview
Product Billing
User Guide
TLS Agent
AgentLoop
Resource management
Host group
Data collection
LogCollector collection
Introduction to LogCollector
Installation and upgrade
Collect host logs
Collect container logs
LogCollector plugins
Manage LogCollector
general reference
Data import
Cloud product log collection
TOS log collection
DBW log collection
VPN Log collection
VKE log collection
CLB log collection
CloudTrail log collection
CFW log collection
NTA log collection
ALB log collection
APMPlus log collection
veFaaS log collection
VPC log collection
veImageX log collection
TrafficRoute DNS log collection
DDoS log collection
WAF log collection
DCDN log collection
Multi-cloud security platform log collection
APIG log collection
Machine learning task log collection
CR log collection
Volcano Ark application lab log collection
FileNAS log collection
LLM Application Firewall Service log collection
vePFS log collection
Cloud security center log collection
EFS log collection
Data storage
Query and analysis
Configure log indexes
Query log
Analyze logs
Query and analyze logs using third-party tools
Associate external data source for log query
SQL function
SQL syntax
Visualization
Query and analysis charts
Summaries of query and analysis charts
Create dashboard
Use dashboard
Add dashboard filters and variables
Use other systems to analyze logs
Log application
Log applications for product observability
Log applications for large models observability
Coding Agent log app
ModelProvider log app
No-coding Agent log app
Log applications for basic service monitoring
Data processing
Data transformation
Syntax and functions
Global operation function
expression function
Data processing best practices
general reference
Data write processor
Data consumer processor
Data consumption and shipping
Ship logs to TOS
Ship logs to Kafka
Ship logs to Splunk
Alarm
Alarm policy
Alarm notification content
Alarm notification content template
alarm notification channel
Tracing service
Monitor Log Service
API reference
Project management API
Topic management API
Shard management API
Index management API
Log management API
Host group management API
Collection configuration management API
Data import API
Data processor API
Data transformation API
Data backflow API
SQL analysis API
Alarm management API
Consumer group management API
Kafka protocol consumption API
Tag management API
Trace storage API
Account management API
SDK reference
Go SDK
Java SDK
Python SDK
iOS SDK
C++ SDK
CLI reference
Best practices
Data collection
Query and analysis
permission management
Consumption and delivery
Visualization
Alarm
FAQs
Billing and cost FAQ
Data collection FAQ
Data storage FAQ
Index configuration FAQ
Query and analysis FAQ
Scheduled SQL tasks FAQ
Alarm FAQ
- Documentation
Torch Log ServiceUser GuideData processingData transformationCreate data transformation task
Copy page
Download PDFData transformation
Create data transformation task
Copy pageDownload PDFCreate data transformation task
TLS provides flexible and convenient data transformation capabilities, supporting data cleansing, filtering, content masking, data splitting, and data distribution for collected log data. After you create a data transformation task, TLS automatically reads the data from the source log topic and processes it according to the specified processing rules, and then writes the results to different log topics.
This topic describes how to create a data transformation task through the console.
Prerequisites
- Log data has been collected. For detailed steps, see Overview.
- A log project and log topic are created.
- Before creating a data transformation task, you can read Data transformation overview to know the feature description and limitations.
Steps
- Log in to TLS console.
- Select the region where log service is located from the top navigation bar.
- In the left navigation bar, select Data processing > Data transformation.
- On the Data transformation - Task list page, click Create data transformation task.
- On the data transformation task creation page, complete the following configuration.
- Fill in the basic settings for the data transformation task, and then click Next: processing rule configuration.
- ParametersDescriptionParametersDescriptionTask nameName of the data transformation task.DescriptionA brief description of the data transformation task.Source projectThe project that contains the logs to be transformed.Source topicThe topic that contains the logs to be transformed.Target topicInformation about the target topic, which refers to the topic where the transformed logs are stored.You can specify up to 20 different log topics. Only one log topic can be selected or one topic ID can be entered for each target topic. If multiple topics are specified, the transformed data is saved to multiple log topics.TLS supports data transformation for both same-account and cross-account scenarios. You need to provide the following information for each output destination:
- Same account:
- Shipping type: Select This account.
- Target name: The customized name for the output target. This name needs to be used in the data processing rules to refer to the output target.
- Target Region: The region where the transformed log data will be stored.
- Configuring cross-region log data storage incurs cross-region transfer traffic fees. For more details, see Billing overview.
- After configuring cross-region log data storage, only the Topic ID field is required.
- Currently available in North China 2 (Beijing), South China 1 (Guangzhou), and East China 2 (Shanghai) regions.
- Project: The project used to store the transformed log data.
- Log topic: The log topic where the transformed log data will be stored.
- Topic ID: After configuring cross-region storage of log data, fill in the log Topic ID to store the processed log data.
- Cross-account:
- Shipping type: Select Other accounts.
- IAM role: For cross-account data transformation, you need to configure an IAM role by entering a custom role's TRN. This authorizes the transformation task to assume the custom role, process the data, and then write it to TLS in another account.
- This custom role must have query and write permissions for the target topic. For instructions on creating a role and granting permissions, see Configure cross-account data transformation permissions.
- Target name: The customized name for the output target. This name needs to be used in the data processing rules to refer to the output target.
- Target Region: The region where the transformed log data will be stored.
- Configuring log data storage across regions incurs cross-region transfer traffic fees. For details, see Billing overview.
- Currently available in North China 2 (Beijing), South China 1 (Guangzhou), and East China 2 (Shanghai) regions.
- Topic ID: The log topic ID where the processed log data is stored.
Tip- Set the shard count for the target log topic based on the shard count of the source log topic. If the source log topic receives high write traffic and the target log topic has fewer shards, data transformation performance may be affected.
- If you specify multiple target log topics, the data transformation results are written to the first log topic by default. You can also use the e_output function to distribute the processed data to different log topics when editing transformation rules. For details, see e_output function.
Processing time rangeThe processing time range for the data to be transformed.This time range refers to the period during which log data is stored on the TLS server.- All logs: Process all data stored within the specified log topic, including historical data and newly ingested logs.
- From certain time: Process data collected after the specified time, including newly ingested logs. If no time is specified, all logs will be processed.
- Specify time range: Process historical data within the specified time period.
Tip- The time range is a left-closed, right-open interval.
- If the start time is not specified, processing begins with the earliest log. If the end time is not specified, newly written data will continue to be processed.
- Write transformation rules.
- Follow the onscreen instructions to enter transformation rules in the input box. TLS supports a variety of DSL syntax functions. On the transformation functions tab, you can click a function name to view syntax descriptions and examples, and insert specified syntax with one click to quickly write transformation rules. For the list of supported syntax functions, see Functions overview.
- Click Preview execution to test transformation rules.
- Follow the page instructions to test by using log data from the source log topic or specified test data.
- If the page prompts DSL code compilation failed, it means the transformation statement contains errors. You can refer to the error information provided and fix the syntax issue in the specified line, and then retest the transformation rule.
- If you confirm the data transformation results are correct, click Complete task creation.
- In the Enable immediately after task creation dialog, select whether to enable the task right away.
- Enable immediately: Start the data transformation task right after completing task configuration.
- Enter pause state: The data transformation task will be paused after configuration is completed and needs to be manually started later.
- Cancel: Cancel task creation for data transformation.
Tip
When a data transformation task is stopped or has completed, TLS does not automatically delete its associated consumer group instances. It is recommended to promptly clean up data transformation tasks that are stopped or completed and confirmed to be unused, reducing unnecessary consumer group instances.
Related operations
- After the data transformation task starts, TLS records the execution status and progress of the task in the service logs. You can view and analyze the relevant log data in the corresponding log topic. For more details, see Service log fields.
- After you successfully create a data transformation task, you can modify or delete the task as needed for your business requirements. For detailed instructions, see Manage data transformation tasks.
- We recommend enabling indexing for the target log topic to facilitate query and analysis of the processed data. Refer to the steps Configure indexes.
Last updated: 2026.07.30 17:29:25
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
