Docs

Torch Log Service

Timed SQL Analysis

Scheduled SQL analysis overview

Log Service supports scheduled SQL analysis, enabling retrieval and analysis of log data within specified ranges according to preset time windows and scheduling cycles. The results are saved to designated log topics for scenarios such as data analysis, log aggregation, and information filtering. This topic describes the features and usage limitations of scheduled SQL analysis.

Background information

In scenarios where business log data is highly precise and log content is complex, long-term storage of large volumes of data incurs high costs, and real-time queries of massive data may cause performance issues. TLS provides scheduled SQL analysis. This feature allows periodic query and analysis of specified log data ranges and saves the results to designated log topics. Scheduled SQL analysis supports all query and analysis syntax in TLS, enabling analysis tasks to run based on preset time windows and scheduling cycles.

After using scheduled SQL analysis, you can set a relatively short storage period for source log topics that store large volumes of data. Log data can be deleted after analysis, reducing the cost of storing logs and indexes. The target log topic where the scheduled SQL analysis results are saved can be set to a longer storage period, such as permanent retention.

In addition, TLS provides logs for scheduled SQL analysis, which display the running information of scheduled SQL analysis tasks. For more details, see Service log overview.

Features

Scheduled SQL analysis provides the following capabilities:

Scheduled log query and analysis: Based on a specified time window and scheduling cycle, TLS regularly executes query and analysis statements on logs and saves the analysis results to a designated log topic. Scheduled retrieval and analysis can be used for periodic data statistics scenarios, such as regularly summarizing log data, extracting key results, and generating data reports.

Log aggregation: This feature aggregates detailed raw log data, which allows you to summarize large volumes of logs into appropriately precise data and reduce storage costs. Log aggregation is ideal for scenarios like storage cost control. For example, aggregating logs by time window to reduce log precision and save storage space.

Log filtering: This feature filters raw log content according to specified rules. It retains only important fields and saves them to the specified log topic. Log filtering is suitable for data cleansing scenarios, such as filtering out error logs in operations and maintenance.

Notice

Schedule SQL tasks in TLS are currently in public beta, and there are no service charges. However, other TLS resources involved in schedule SQL analysis are billed according to standard items. For example, the target log topic used to store processed log data will incur storage fees. For detailed pricing information, see Billing items based on pay-by-feature mode.

After creating a schedule SQL analysis task, TLS automatically adds the following reserved fields to each piece of data when writing, to record metadata information for the task. For descriptions of basic concepts such as scheduling time, see Basic concepts.

Field	Description
__task_id__	ID of the scheduled SQL analysis task.
__process_id__	The instance ID that executes this scheduled calculation in timed SQL analysis tasks.
__process_start_time__	The start time of the SQL time window for timed SQL analysis tasks.
__process_end_time__	SQL time window end time for scheduled SQL analysis tasks.
__process_time__	The scheduling time of an instance, i.e., the time when the backend service starts to create and run instances by default.
__schedule_time__	The execution time of the instance indicates when the scheduled SQL analysis task actually starts running.

Limits

Category	Limit items	Description
Resource management	Number of scheduled SQL analysis tasks	A maximum of 100 SQL analysis tasks can be created per log project. A maximum of 15 scheduled SQL analysis tasks can be created per source log topic.
		Delete scheduled SQL analysis task	The trace functionality of TLS depends on the normal running of the scheduled SQL analysis task. To ensure the availability of trace storage, do not conduct any manual changes to SQL analysis tasks that correspond to trace storage, such as changing configurations or deleting tasks. If you need to modify the trace storage configuration or disable trace storage, proceed on pages that relate to trace storage.
		Execution history	Task execution record can be retained for up to 7 days. We recommend that you add scheduled SQL analysis tasks to identify issues promptly.
		Task scheduling concurrency	A single account supports 10 concurrent executions.
Data writing	Data delay	When data delay happens, the risk that data fails to be retrieved may exist. That is to say, if data for a certain time arrives after the corresponding scheduling instance finishes, it will not be queried in the next scheduling instance.
		Instance concurrency	Only one scheduling instance per task can be in the scheduling state at a time.
		Write failure retry	If data writing fails, the system retries 3 times thereafter. The interval between every two retries is 60 seconds. After 3 consecutive write failures, the instance status will change to execution failed. We recommend that you add scheduled SQL analysis tasks to promptly detect issues.

Last updated: 2026.03.16 14:05:01