Docs

Torch Log Service

Service log

Service log fields

TLS provides log information generated during the execution of various features, including TOS data import runtime logs. This topic describes the fields corresponding to each log type.

Service log types

TLS offers the following service log types:

Log type	Related features	Storage location (Topic)	Description
Data import runtime logs	TOS data import Kafka data import	tls_import-task	TLS automatically records this log after a TOS or Kafka data import task is completed. TLS records the operation and import details for each data import task.
Data import error logs	TOS data import Kafka data import	tls_import-task	Exceptions are reported only when there are anomalies or failures during TOS or Kafka data import tasks. A log entry is recorded for each imported file with an exception.
LogCollector status logs	LogCollector host group	tls_log-collector-status	Logs are only recorded after manually enabling the LogCollector service log feature. When enabled, each LogCollector client records its running status once per minute in log format.
LogCollector business logs	LogCollector host group	tls_log-collector-business	Logs are only recorded after manually enabling the LogCollector service log feature. After enabling, LogCollector reports the collection progress of each file once per minute.
Data transformation runtime logs	Data transformation	tls_etl-log	When running a data transformation task, TLS automatically records process and progress information at regular intervals, and prints the information in service logs. If a runtime issue occurs during data transformation, the service log records error details and labels the logging.level field as ERROR.
Log shipping runtime logs	Ship data to TOS or Kafka	tls_shipper-log	After log shipping is enabled, TLS automatically records the execution status of scheduled shipping tasks and displays the status in service logs. If any runtime issues occur during shipping, detailed error information will be recorded in the runtime log.
Consumer group latency log	Consumer group	tls_consumergroup-log	After a user uses the SDK consumer group feature, TLS records the consumer lag in seconds for each Shard in the consumer group every 2 minutes. Then, TLS prints this information in service logs.
Scheduled SQL runtime logs	SQL analysis	tls_schedule-sql-log	After the instance of a scheduled SQL analysis task starts running, TLS periodically logs the execution details of each instance in the service logs.
Execution logs for alarm policies	Log alarm	tls_alarm-log	After the alarm policy starts running, TLS periodically logs the execution details of each monitoring task, such as execution time and results, and prints them in the service log.
Alarm event logs	Log alarm	tls_alarm-log	Each time an alarm policy is triggered, TLS prints an alarm event log to record relevant metadata when the alarm is triggered.

Data import runtime logs

Data import runtime logs are stored in the log topic tls_import-task. For each import task, TLS records one entry of runtime and import details.

Field	Example value	Description
TaskID	123-xxx	Import task ID.
TaskStatus	Finished	The status of imported tasks, including: Finished: The import task is successfully finished. Transferring: The data is importing. Abnormal: Task exception. Stopping: Task pausing. Stopped: Task paused. Restarting: The task is restarting.
SourceType	tos	The types of data source imported, including: tos: TOS data. Kafka: Kafka data.
Transferred	5004	Number of imported records.
Failed	0	Number of data entries that failed imports.
Skipped	0	Number of records that are skipped during import.
NotExist	0	Number of failed cases caused by non-existent TOS file. If the data source is Kafka, this field is always 0.
Total	5004	Total number of records in the task.
BytesTransferred	20372186	The size of imported data, measured in bytes.
BytesTotal	20372186	Total size of all data, measured in bytes.

Data import error logs

Data import error logs are stored in the log topic tls_import-task. These logs are only reported when exceptions or failures occur during TOS or Kafka data import tasks. TLS will record one log entry for each abnormal file import.

Field	Example value	Description
TaskID	123-xxx	Import task ID.
SourceType	tos	The types of data source imported, including: tos: TOS data. Kafka: Kafka data.
Object	abc.log	Storage path for the imported object in TOS. If the data source is Kafka, this field is empty.
Status	NotExist	Status of imported data. including: Success: Import succeeded. Failed: Import failed. For details, see the Error field. Skip: skip the file. The TOS file was skipped due to reasons such as exceeding the TOS object size limit or decompression failure. NotExist: TOS file does not exist. Due to file deletion during the import process, TLS can not locate the TOS file.
Error	Obj not exist	Reasons for import failure, including: Obj not exist: TOS import object does not exist. Due to file deletion during the import process, TLS can not locate the file. Other reasons such as TOS access failure and Kafka access failure.

LogCollector status logs

LogCollector status logs are stored in the log topic tls_log-collector-status. Each LogCollector client records its running status every minute as a log.

Field	Description
InstanceId	Unique identifier of LogCollector client.
Label	Host identification array.
IP	IP address of the host.
Version	Version number of LogCollector.
HeartbeatStatus	Status of LogCollector's latest heartbeat: Normal: Heartbeat status is normal. Abnormal: Heartbeat status is abnormal.
MemoryUsed	Memory usage, measured in bytes.
CpuUsage	CPU usage percentage.
DuplicateFilePathCnt	Number of collection configurations to which the file is repeatedly bound. If field value is 1, it means the log file is bound to two collection configurations. For related instructions, see Monitoring LogCollector.
DuplicateFilePaths	Log file paths with duplicate bindings in the collection configuration.
SendTotalLogSize	Total logs sent since LogCollector was last started, measured in bytes.
SendTotalLogCount	Total number of logs sent.
SendSuccessLogSize	Total size of successfully sent logs, measured in bytes.
SendSuccessLogCount	Number of logs sent successfully.
SendDropLogSize	Total amount of logs discarded due to failed delivery, measured in bytes.
SendDropLogCount	Number of logs discarded due to failed delivery.
SendTotalReqs	Total number of requests sent to the server.
SendSuccessReqs	Number of requests sent successfully.
SendFailReqs	Number of requests that failed to send.
SendLimitReqs	Number of requests blocked due to rate limiting.
SendDeleteReqs	Number of requests that are blocked because the target Topic was deleted.
QueueCurrSize	Number of logs currently cached by the LogCollector client.
QueueCurrBytes	Amount of logs currently cached by the LogCollector client, measured in bytes.

LogCollector business logs

LogCollector business logs are stored in the log topic tls_log-collector-business. LogCollector reports the collection progress for each file being collected once per minute.

Field	Description
InstanceId	Unique identifier of LogCollector client.
Label	Host identification array.
IP	IP address of the host.
Version	Version number of LogCollector.
TopicId	Destination topic for log writing.
FilePath	The path of the log file currently being collected.
RealFilePath	The actual path of the log file currently being collected on the physical machine.
FileSize	The size of the log file currently being collected, measured in bytes.
ReadOffset	The offset for reading the file, measured in bytes.
SendSuccessOffset	Offset of successfully sent data, measured in bytes.
StartReadTimestamp	Timestamp for starting to read the file.
LastReadTimestamp	Timestamp of the most recent read from the file.
ParseSuccessSize	Size of successfully parsed logs in bytes.
ParseSuccessCount	Number of successfully parsed log entries.
ParseFailSize	Volume of logs that failed to parse, measured in bytes.
ParseFailCount	Number of logs that failed to parse.
FormatTimeSuccessSize	Size of logs successfully parsed by the specified time format, measured in bytes.
FormatTimeSuccessCount	Number of logs successfully parsed when parsing the time field according to the specified time format.
FormatTimeFailSize	Size of logs that failed to parse by the specified time format, measured in bytes.
FormatTimeFailCount	Number of logs failed to parse when parsing the time field according to the specified time format.
FilterSize	The size of logs filtered according to the specified filter configuration, measured in bytes.
FilterCount	The number of logs filtered according to the specified filter configuration.
HarvesterStatus	The status of the file collector includes: started: The collector is started. waiting: The collector is ready. running: The collector is running. closed: Collection ends.

Data transformation runtime logs

Data transformation runtime logs are stored in the log topic tls_etl-log. TLS periodically records the runtime logs for each data transformation task, including input, filter, failure, and output information at the shard level of the source log topic.

Field	Example value	Description
task_id	123-xxx	The task ID for log processing.
etl_context.project	abc-xxx	The Project to which the source Topic of processing task belongs.
etl_context.topic	edf-xxx	Source log topic ID of the data transformation task.
etl_context.shard_id	0	Shard ID of the source log topic consumed by the data transformation task.
logging.level	INFO	Log levels include: DEBUG: Debug log INFO: Normal log WARN: Warning log ERROR: Error log
logging.message	division zero	Information related to the data transformation task. In normal logs (logging.level is INFO), this field is empty. In error logs (logging.level is ERROR), this field contains the error message. For example: invalid literal for int() with base 10: 'ss': Failed to convert the string "ss" to a decimal number. You need to handle such non-numeric fields specifically during conversion. Division by zero: The dividend can not be set to 0. It is recommended to check the processing rule and perform a zero-value check on the denominator before division.
process_info.accept	100	This is the number of logs consumed during the data processing task, i.e., the number of logs that were read from the source log topic.
process_info.dropped	0	This is the number of logs filtered during the data processing task, i.e., logs that are deleted by the e_drop function or logs that are removed due to failed processing.
process_info.delivered	100	The number of logs output during processing, i.e., the logs that have been successfully processed and are ready to be written to the target log topic.
process_info.failed	0	The number of logs that failed to output during processing, such as failures caused by the deletion of the target log topic.

Log shipping runtime logs

Log shipping runtime logs are stored in the log topic tls_shipper-log. TLS regularly records operation information for each log shipping task, including task ID, task status, shipped quantity, traffic, and other details.

Field	Example value	Description
LogCount	10	The number of logs processed in this shipping task.
LogStartTime	1678781706	The timestamp of the first log processed in this shipping task.
LogEndTime	1678781706	The timestamp of the last log processed in this shipping task.
ProcessSpeed	1000	The shipping speed for this task, measured in Byte/s.
SendStartTime	1678781706	The start timestamp of this shipping task.
SendEndTime	1678781706	The end timestamp of this shipping task.
ShardId	0	The shard of the topic to be shipped for this shipping task.
ShipperId	411-xxx	The ID of the shipping configuration to which the shipping task belongs.
TaskId	task-xxx	The ID of this shipping task.
TopicId	b67-xxx	The ID of the topic shipped in this shipping task.
TopicName	testtopic	The name of the topic shipped in this shipping task.
WriteTraffic	1000	The amount of write traffic generated by this shipping task, measured in bytes.
ReadTraffic	1000	The amount of read traffic generated by this shipping task, measured in bytes.
Message	Task shipper success.	Execution response details for this shipping task are as follows: StatusCode: status code. If StatusCode is 1, it means the shipping task successed; you do not need to review ErrorCode and ErrorMessage. If StatusCode displays other value, it means the shipping task failed; you can check ErrorCode and ErrorMessage for details about the failure. ErrorCode: Error code that specifies the failure type. ErrorMessage: Error message that provides details about the failure. Common response information includes: Task shipper success: Task completed successfully. No log data: Task completed successfully, but no logs needed to be delivered. Shipper tos bucket not exist: Task failed because the specified TOS bucket does not exist. Shipper Task is cancelled: Task execution failed because the shipping task was cancelled.
Status	Success	Result of this shipping task, including: Success: Execution succeeded. All log entries processed this time have been delivered. Failed: Execution failed. See the Message field for details.

Consumer group latency log

Consumer group latency logs are stored in the log topic tls_consumergroup-log. TLS records one consumer latency progress log in the service logs every two minutes for each active ConsumerGroup. Mainly includes information such as consumer group ID, topic ID, shard ID, and latency.

Field	Example value	Description
ConsumerGroupName	test	Consumer group name.
ProjectId	xxx	The project ID to which the consumer group belongs.
TopicId	xxx	The log topic ID associated with this delay information.
ShardId	0	The shard ID associated with this delay information.
FallBehindSecond	10	The consumption delay in seconds of the current consumer group on the current shard, i.e., the difference between the timestamp of the latest consumed data and the timestamp of the actual latest data in this shard.

Scheduled SQL runtime logs

Scheduled SQL runtime logs are stored in the log topic tls_schedule-sql-log. Scheduled SQL runtime logs record the detailed execution of each scheduling instance for SQL analysis tasks in TLS, including the scheduled SQL analysis task ID, scheduled SQL execution instance ID, scheduled SQL scheduling time, and more.

Field	Example value	Description
task_id	0c554a3a-a307-43ec-857a-3743f8a0****	ID of the scheduled SQL analysis task.
process_id	0c554a3a-a307-43ec-857a-3743f8a0****	Scheduled SQL analysis task instance ID.
task_start_time	1692180957000	Task execution start time, measure in milliseconds.
task_end_time	1692181022000	Task execution end time, in milliseconds.
task_running_time	65	Task execution duration, measure in milliseconds.
process_time	1692180883	Instance scheduling time, displayed as a timestamp in seconds.
process_start_time	1692180780	Scheduling start time, displayed as a timestamp in seconds.
process_end_time	1692180840	Scheduling end time, displayed as a timestamp in seconds.
process_log_nums	1	Number of processed logs.
process_result	{status:"success","message":"success"}	Instance scheduling result. Note: status: Scheduling result, including: success: Scheduling succeeded. error: Scheduling failed. Message: Additional details, i.e., the specific state information. When the instance status is error, this field returns the error details, such as Dest topic xxx doesn't exist, indicating the destination log topic does not exist. When the instance status is success, this field returns success.

Execution logs for alarm policies

Execution logs for alarm policies are stored in the log topic tls_alarm-log. After an alarm policy starts running, TLS records the execution details of each monitoring session, such as execution time and results, and prints them in the service logs.

Field	Example value	Description
AccountId	200000****	Volcanoengine account ID.
ProjectId	a2026e02-91d8-460e-887f-19db4da9****	Project ID.
ProjectName	tls_service-project	Project name.
AlarmName	rewrwer-copy	Alarm name.
AlarmId	1482fce9-6655-4aa2-bf6b-9a1d4e20****	Alarm policy ID.
TopicIds	ebee41c6-ba2f-4dba-84be-b310ce4c****	Log topic ID list.
TopicNames	tls_etl-log	Log name list.
HappenThreshold	$1.count>=100	Trigger conditions configured in the alarm policy.
AlarmFrequency	30	Alarm frequency.
ReachTrigger	FALSE	Whether the alarm is triggered.
Notify	NotSend	Execution result. Not Send: Not send SendFailed: Delivery failed PartialSuccess: Partial delivery AllSuccess: All delivered
NotifyGroup	[{"NotifyGroupId":"87527591-06b2-4e40-bcee-c92416ba****","Name":"yyj-tsy-001"}]	Alarm group list. NotifyGroupId: Alarm group ID. Name: Alarm group name.
ProcessResult	AlarmConditionEvaluateError	Execution result. The following are common execution results. SendFail: Failed to send notification. QueryError: Analysis statement execution exception. Check your analysis statement and the index configuration for the log topic. PartialSent: Partial alarms sent successfully.
ProcessMsg	Alarm evaluate error: result of '$1.count' does not exist	Execution result details.
NotifyType	AlarmRecover	Alarm type.
Severity	notice	Alarm level.
AlarmDuration	0	Alarm cycle.
SmsBillingCount	0	Number of SMS chargeable messages sent.
AlarmVersion	2	Alarm version.
SmsSuccessCount	0	Number of alert SMS messages sent successfully.
SmsFailedCount	0	Number of alert SMS messages that are failed to send.
VmsSuccessCount	0	Number of successful alarm calls.
VmsFailedCount	0	Number of failed alarm calls.
EmailSuccessCount	0	Number of alarm emails that are successfully sent.
EmailFailedCount	0	Number of alarm emails that failed to sent.
WebHookSuccessCount	0	Number of successful WebHook requests.
WebHookFailedCount	0	Number of failed WebHook requests.
Env	{"AlarmStartTime":"","FireResults":"","FireResultsCount":0,"Results":"[{\"TopicId\":\"ebee41c6-ba2f-4dba-84be-b310ce4c***\",\"TopicName\":\"tls_etl-log\",\"Query\":\" AND task_id:\\\"7975efd7-e63b-4cdd-b51c-06d9cf08***\\\" \| select sum(process_info.failed) AS failed_count\",\"StartTime\":1701678164,\"EndTime\":1701679064,\"RawResults\":[{\"failed_count\":\"0\"}],\"RawResultsCount\":1,\"FireResult\":{}},{\"TopicId\":\"ebee41c6-ba2f-4dba-84be-b310ce4c5b9f\",\"TopicName\":\"tls_etl-log\",\"Query\":\" AND task_id:\\\"7975efd7-e63b-4cdd-b51c-06d9cf0**\\\" \| select sum(process_info.failed) AS failed_count\",\"StartTime\":1701678164,\"EndTime\":1701679064,\"RawResults\":[{\"failed_count\":\"0\"}],\"RawResultsCount\":1,\"FireResult\":{}},{\"TopicId\":\"ebee41c6-ba2f-4dba-84be-b310ce4c*\",\"TopicName\":\"tls_etl-log\",\"Query\":\" AND task_id:\\\"7975efd7-e63b-4cdd-b51c-06d9cf0****\\\" \| select sum(process_info.failed) AS failed_count\",\"StartTime\":1701678164,\"EndTime\":1701679064,\"RawResults\":[{\"failed_count\":\"0\"}],\"RawResultsCount\":1,\"FireResult\":{}}]"}	Common variable field values used during alert execution.
LogType	Alarm	Type of log. Alarm: Alarm policy execution log Incident: Alarm event log
QueryUrl	https://console-stable.volcanicengine.com/tls/region:tls+cn-chongqing-sdv/project/a2026e02-91d8-460e-887f-19db4da9**/search?codebase64=KiBBTkQgIHRhc2tfaWQ6Ijc5NzVlZmQ3LWU2M2ItNGNkZC1iNTFjLTA2ZDljZjA4N2VmMiIgfCBzZWxlY3Qgc3VtKHByb2Nlc3NfaW5mby5mYWlsZWQpIEFTIGZhaWxlZF9j==&hideIndexBtn=false&hideShareBtn=false&hideTitle=false&hideTopic=false&time=2023-12-04%2016:22:44%2C2023-12-04%2016:37:44&topicId=ebee41c6-ba2f-4dba-84be-b310ce4**	URL for the query and analysis page of the first alarm query statement.

Alarm event logs

Alarm event logs are stored in the log topic tls_alarm-log. Each time an alarm policy is triggered, TLS prints an alarm event log to record relevant metadata when the alarm is triggered.

Field	Example value	Description
AccountId	200000****	Volcanoengine account ID.
InstanceId	94285baf-7568-48dc-983f-26402bb****	Alarm instance ID.
IncidentId	06a80b71-f578-4732-aeab-f215a7e8****	Alarm event ID.
AlarmAnnotations	{"desc":"aaa001 alarm triggered","title":"aaa001 alarm triggered"}	Description of alarm metadata.
ProjectId	a2026e02-91d8-460e-887f-19db4da****	Project ID.
ProjectName	tls_service-project	Project name.
AlarmName	rewrwer-copy	Alarm name.
AlarmId	1482fce9-6655-4aa2-bf6b-9a1d4e20****	Alarm policy ID.
Env	{"AlarmStartTime":"","FireResults":"","FireResultsCount":0,"Results":"[{\"TopicId\":\"ebee41c6-ba2f-4dba-84be-b310ce4c***\",\"TopicName\":\"tls_etl-log\",\"Query\":\" AND task_id:\\\"7975efd7-e63b-4cdd-b51c-06d9cf08***\\\" \| select sum(process_info.failed) AS failed_count\",\"StartTime\":1701678164,\"EndTime\":1701679064,\"RawResults\":[{\"failed_count\":\"0\"}],\"RawResultsCount\":1,\"FireResult\":{}},{\"TopicId\":\"ebee41c6-ba2f-4dba-84be-b310ce4c5b9f\",\"TopicName\":\"tls_etl-log\",\"Query\":\" AND task_id:\\\"7975efd7-e63b-4cdd-b51c-06d9cf0**\\\" \| select sum(process_info.failed) AS failed_count\",\"StartTime\":1701678164,\"EndTime\":1701679064,\"RawResults\":[{\"failed_count\":\"0\"}],\"RawResultsCount\":1,\"FireResult\":{}},{\"TopicId\":\"ebee41c6-ba2f-4dba-84be-b310ce4c*\",\"TopicName\":\"tls_etl-log\",\"Query\":\" AND task_id:\\\"7975efd7-e63b-4cdd-b51c-06d9cf0****\\\" \| select sum(process_info.failed) AS failed_count\",\"StartTime\":1701678164,\"EndTime\":1701679064,\"RawResults\":[{\"failed_count\":\"0\"}],\"RawResultsCount\":1,\"FireResult\":{}}]"}	Common variable field values used during alert execution.
LogType	Alarm	Type of log. Alarm: Log of alarm policy execution. Incident: Log of alarm events.

Last updated: 2026.04.14 16:37:51