Docs

Torch Log Service

Product Overview

Basic concepts

This topic introduces common basic concepts of Torch Log Services(TLS).

Basic resources

Basic concepts	Description
Logging	Logs record events during the operation of systems or programs, including user actions, interface access, and runtime errors. Each log type has a consistent structure and format.
Link tracking data	Trace data records the entire service call chain information, including call relationships, latency, and results.
Log project	A log project is a data management unit used for tenant resource isolation and control. Generally, one log project corresponds to one application, business, or product.
Log topic	A log topic is a unit for data collection and storage. A project can include multiple log topics. Each log topic belongs to one log project. You can configure different data sources and collection rules for the same log topic.
Shard	A log shard is the basic unit for data read and write. All data is stored within shards. Each shard provides specific read and write capacity. We recommend that you plan the number of log shards based on actual business needs.

Data collection

Basic concepts	Description
LogCollector	LogCollector is a log collection tool developed by the TLS team based on the open-source tool Filebeat. It receives collection configurations and reports logs to the server in real time.
Collection configuration	Collection configuration is a rule collection for LogCollector to collect data. It specifies log file locations, collection methods, etc.
Host group	A host group is a virtual configuration unit in TLS. One host group can contain multiple servers. TLS distributes LogCollector configurations at the host group level to collect logs in batches from multiple servers within the group.

Data query and analysis

Basic concepts	Description
Query	TLS provides real-time query capability and supports the query of hundreds of billions of data entries at the second level. You can specify query rules in query statements to quickly retrieve target logs.
Analysis	Based on the query feature, TLS provides real-time SQL computing capability. Supports real-time analysis and calculation of massive logs. Analysis results are displayed in charts.
Query and analysis statements	Query and analysis statements consist of query conditions and SQL analysis statements. The format is &{query conditions} \| &{SQL analysis statements}.
Index	An index is a storage structure that sorts one or more column values in a data table. An index is like a directory for the data. It allows you to quickly find target data rows based on keywords. TLS supports creating full-text index and key-value index.

Visualization

Basic concepts	Description
Dashboard	A dashboard is a real-time data analysis panel provided by TLS to centrally display data query and analysis results.
Statistical charts	Statistical charts visualize query and analysis results. Charts include tables, line charts, pie charts, single-value charts, flow charts, and more.
Dashboard filter	Filters can interact with statistical charts. Based on field names and field values, you can filter statistical charts.
Dashboard variables	Variables can interact with statistical charts. Based on variable names and variable values, you can filter statistical charts.

Data transformation

Basic concepts	Description
ETL	ETL (Extract, Transform, Load) refers to the process of extracting, transforming, and loading data from the source to the target. Via this process, data integration, standardization, and unification can be achieved. TLS supports processing logs within the source log topic and outputs the processed data to the specified target log topic.
TLS DSL	TLS DSL (Domain Specific Language) is a Python-like scripting language provided by TLS. It is used for defining data transformation logic during processing. TLS DSL provides dozens of transformation functions to support data processing needs in various big data analysis and data warehouse scenarios.
Processing rules	Transformation rules are scripts for data transformation. The rules consist of logical code organized by TLS DSL.
Transformation task	A transformation task is a scheduling unit for data transformation. It defines configurations such as source log topics, target log topics, and transforming rules.

Scheduled SQL analysis

Basic concepts	Description
Scheduled SQL analysis task	Scheduled SQL analysis task is a collection of SQL analysis rules. It defines configuration items such as the source/target log topics, scheduling period, and SQL time window.
Scheduled SQL analysis task instance	After starting the scheduled SQL analysis task, TLS creates instances on time based on the specified scheduling cycle. Each instance performs a scheduled SQL analysis once on the source log topic. The results are written into the specified log topic.

Log shipping and consumption

Basic concepts	Description
Consumer group	A consumer group is a virtual set composed of multiple consumers. When consuming data by consumer group, all consumers in the group subscribe to the same log topic and jointly consume data from that topic. Each consumer consumes data from one or more shards of a consumer log topic. Data will not be consumed repeatedly among different consumers.
Consumer	A consumer is a client that consumes data in TLS. It is a part of a consumer group. Within the same consumer group, consumer names are unique. At any given time, a shard in a log topic is assigned to a consumer in the consumer group. One consumer may handle multiple shards. During the running of consumer groups, the relationship between consumers and shards may change.
Consumption offset	During consumption, the consumer program continuously records the consumption offset (cursor progress) of current shard and reports it to the server. Thereafter, starting consumption cursor is recorded when the program restarts. This helps to prevent data from being consumed repeatedly.

Alarm

Basic concepts	Description
Alarm policy	The alarm policy defines data monitoring methods and alarm rules. You need to specify the monitoring task, alarm rules, alarm notification methods, and other configurations in the policy.
Notification group	Notification groups are collections of alarm notification policies that define notification types, delivery channels, and receiving objects.
Alarm triggered	Alarm triggering is a type of alarm notification. An alarm notification is sent when the monitoring result meets the alarm triggering expression.
Alarm recovery	An alarm recovery notification is a type of alarm notification. A recovery notification is sent when the monitored object returns to normal.
Content template	An alarm notification content template is a predefined text template that specifies the content and format of alarm notifications. The content template supports adding variables. When the notification is sent, variables are replaced with actual values.
Webhook integration	TLS supports managing alarm notification channels such as Feishu, DingTalk, and WeCom through Webhook integration.

Last updated: 2025.12.03 15:13:24