Docs

Torch Log Service

Data transformation

Data transformation overview

TLS provides flexible and convenient data transformation capabilities, supporting data cleansing, filtering, content masking, data splitting, and data distribution for collected log data. This topic describes the principles, capabilities, and limitations of data transformation.

Features overview

In big data and stream computing scenarios, logs often need to be processed into structured data. For example, extracting fields from logs with mixed formats, formatting log content, or distributing logs to multiple downstream data processing systems for storage or computation for different business systems. TLS offers real-time data transformation services, delivering data cleansing, filtering, content masking, data splitting, and data distribution for log data. After creating a data transformation task, TLS establishes a consumer group to extract log data from the source log topic. And then TLS processes these logs according to the specified transformation rules and DSL (Domain Specific Language) functions, and writes the processed data into the designated log topic for subsequent log analysis and data archiving.

The data transformation feature includes a range of built-in DSL functions, supporting complex scenarios for data structuring, cleansing, and masking. For supported transformation syntaxes, see data processing syntax.

In addition, TLS provides service logs with the data transformation feature. You can view task progress and processing rates. For more details, see Service log overview.

Use cases

The data transformation feature supports processing capabilities for various scenarios:

Structured processing: If logs can not be structured during collection, you can collect them to the server first and then use the data transformation feature for complex structuring tasks, such as field extraction and format conversion, which helps with subsequent SQL statistics and analysis. Suitable for stream processing, data warehouse computation, and similar scenarios.

Cleaning and filtering: Uniformly filter diverse and complex logs, retaining only key information that requires attention. Before filtering, you can set a short data retention period for logs, reducing storage volume and lowering costs.

Content desensitization: Apply desensitization to sensitive information in logs, such as mobile numbers, ID numbers, passwords, and other business log details.

Data splitting and distribution: After unified log transformation, split one log into multiple records and deliver them to different downstream systems.

Billing description

Data transformation services incur billing costs from TLS. For detailed pricing and billing methods, see TLS Billing items based on pay-by-feature mode.

Notice

Concurrent unit data load: The concurrent unit's data load affects the performance of data transformation.

During data transformation, the data load of a single concurrent unit depends on the amount of data stored in its corresponding source log topic shard. If log data written to the source log topic is unevenly distributed across multiple partitions, it will cause hot partitions during the execution of data processing jobs, resulting in delayed processing progress on some partitions.

If you use the HashKey routing shard mode to write log data to the source log topic, allocate shard traffic reasonably when writing to minimize uneven distribution of data across shards. For more information regarding data writing, see PutLogs.

TLS writes transformed data to the target log topic using the PutLogs API. The data writing rate depends on the number of shards in the target log topic.

When writing the processing results to the target log topic, you must comply with the data write specifications of the log topic. If the target log topic has fewer shards and a large amount of data is written, it may cause processing delay. We recommend that you valuate the number of shards for the target log topic or enable the auto split feature based on the data volume written to the target log topic. For details, see basic resources and data read and write. For information about enabling the shard auto split feature, see Manage shards.

When a write limitation error occurs during data transformation to the target log topic, the system performs unlimited retries to ensure data integrity. However, this affects transformation progress and causes delays for jobs handling the current source shard.

Limits

Limit items	Description	Note
Number of tasks	A maximum of 100 data transformation tasks can be created per log project. A maximum of 30 data transformation tasks can be created per log topic. All tasks that are displayed on the page are included, regardless of their status. Notice Tasks that are in suspended or completed statuses also occupy the quota. Therefore, if you ensure that you no longer need some task, clear the task list promptly to release quota quantities. For more information, see the following documents: Manage data transformation tasks.	For quota adjustments, refer to Apply for resource quota.
Target log topic number	You can configure up to 20 different log topics in one data transformation task to store processed data.	For quota adjustments, refer to Apply for resource quota.
Execution preview	The execution preview function for data transformation is used for online debugging of data transformation code. Note the following restrictions when conducting execution preview: In a single request, test data or raw logs can not exceed 20 entries. Otherwise, the preview request will return an error. For per request, a maximum of 100 transformation results can be returned.	/
Transformation performance	The overall transformation task rate is calculated as the number of shards read and written by the source log topic * 1MiB/s (uncompressed traffic). Meanwhile, the overall transformation rate is also limited by factors such as transformation rule complexity, source log topic concurrency, source log topic consumption rate, and target log topic write speed. For more details, see Factors affecting performance.	/

Basic concepts

Concepts	Description
ETL	ETL (Extract, Transform, Load) refers to the process of extracting, transforming, and loading data from source business systems to the target, enabling integration, standardization, and unification of data. TLS supports loading log data from the specified source topic and outputting the processed results to the designated target topic.
Events, data, logs	In data transformation, both events and data refer to log data; for example, one event corresponds to one log entry. The event function e_keep is used to retain specific logs.
Source topic	In data processing, the source log topic refers to the location where the log data to be processed is stored. The log service consumes data from it and performs data processing. Only one source topic can be set for a transformation task, but multiple data transformation tasks can be added for a single source topic.
Target topic	The target topic is the log topic where processed data is written. A transformation task can be configured with up to 20 target log topics.
TLS DSL	TLS DSL (Domain Specific Language) is a Python-like scripting language provided by TLS for specifying the data processing logic during data transformation. TLS DSL offers more than forty function syntaxes to support various data processing needs in big data analytics and data warehouse scenarios. For the function list, see Function overview.
Processing rules	The execution script for data transformation is a collection of logical code organized by TLS DSL.
Transformation task	The scheduling unit for data transformation, which defines multiple configuration items such as source log topic, target log topic, and processing rules.

Last updated: 2026.04.16 11:21:29