Docs

Torch Log Service

Data consumer processor

Consumer processor overview

TLS supports regular real-time data consumption and rule-based data consumption processed through the consumer processor. This topic describes the features and usage limitations of the TLS consumer processor.

Notice

Consumer processor is an invite-only feature. If you have related business needs, contact your account manager to enable the whitelist.

Billing notes

Data processor is in closed beta and is free of charge for now. We will release billing notes in the upcoming announcement.

Features overview

Rule-based data consumption

The TLS data consumer processor is a powerful tool that pre-processes and cleans semi-structured log data on the server side using data transformation syntax. For example, before consuming log data, you can do the following to log data: row filtering, column trimming, regex extraction, and JSON field extraction, etc. This is ideal for scenarios in stream and real-time computing where data preprocessing is required.

You can create a consumer processor directly in the TLS console and write processing rules using SQL-like DSL (data processing language). With the processors and rules, you can use the following features such as string masking, encrypting fields, renaming fields, conditional judgment, flow control, and structured handling of JSON fields.

Use DSL to define data processing rules. For details, see Data transformation overview.

Supported data processing functions. For details, see Functions overview.

Common data consumption

TLS offers the following common data consumption methods. The ConsumeLogs API supports rule-based data consumption through consumer processors.

Data consumption types	Reference
Consume via Kafka protocol	Consume logs via Kafka protocol Consume logs with Spark Streaming Consume logs with Flink Consume logs with ByteHouse
Consume with ConsumeLogs API	Consume logs using ConsumeLogs
Consume with consumer group	Consume data via consumer group

Features

Data write processors, consumer processors, and data transformation tasks all support real-time, lightweight log data cleansing, filtering, content masking, and data conversion and data enrichment capability. The main differences are compared as follows:

Features comparison	Write processor	Consumer processor	Data transformation task
Processing stage	Data before writing to TLS	Data collected in TLS	Data collected in TLS
Applicability	Exists in the project and applies to log topics within the project. Cross-region, cross-account, and cross-project operations are not supported.	Exists in the project and applies to log topics within the project. Cross-region, cross-account, and cross-project operations are not supported.	An independent processing task that applies to the selected source log topic and target log topic Supports cross-region, cross-account, and cross-project usage.
Billing notes	Currently free of charge. The billing schedule will be announced later.	Currently free of charge. The billing schedule will be announced later.	Data transformation fees and cross-region transmission costs.
Processing rules	You can use built-in functions provided by TLS DSL to write processing rules. The following functions are not supported: e_output, e_auto_output, and res_rds_mysql.	You can use built-in functions provided by TLS DSL to write processing rules.	You can use built-in functions provided by TLS DSL to write processing rules.

Limits

Items	Specifications
Number of consumer processors	You can create up to 100 consumer processors in each project. For quota adjustments, see Apply for resource quota.
DSL limitations	The length of each DSL can not exceed 4,000 characters. Supports only row-level processing commands. Operations like aggregation and logic judgments are not supported.
Effective time	Becomes effective within 1 minute after creating or editing a consumption processor.
DSL rules and performance impact	For details, see Data transformation overview. Notice If there are DSL syntax errors or missing source data fields, data obtained may be incomplete or fail.

Last updated: 2026.05.27 18:28:03