Docs

Torch Log Service

Document

Product News

New feature release notes

Product announcement

Announcement on the cancellation of the free quota for TOS log management

TLS Architecture upgrade notification

Product Overview

What is Torch Log Service

Product features

Application scenarios

Product Billing

Billing items

Billing items based on pay-by-feature mode

Billing items based on pay-by-ingested-data mode

Billing examples

Billing examples based on pay-by-feature mode

Billing examples based on pay-by-ingested data mode

Getting Started

Query and analyze logs

User Guide

TLS Agent

Smart assistant copilot

AgentLoop

AgentLoop overview

Quick Start with AgentLoop

Data integration

LogBackFlow

Data backflow overview

Create data backflow task

Manage data backflow task

Create data backflow task for Volcanoengine Ark high-code application

Resource management

Host group

Create host group (IP address)

Create host group (Host identification)

Tag management

Filter resources by tags

Edit or unbind tags

Apply for resource quota

Console overview

Data collection

Collection overview

LogCollector collection

Introduction to LogCollector

LogCollector limits

LogCollector versions

Installation and upgrade

Install LogCollector (host machine)

Install LogCollector (DaemonSet deployment)

Upgrade LogCollector (host machine)

Install LogCollector (Sidecar deployment)

Install LogCollector CRD

Upgrade LogCollector (Kubernetes cluster)

Collect host logs

Single-line full text mode

Multi-line full text mode

JSON mode

Separator mode

Single-line complete regex mode

Multi-line complete regex mode

NGINX configuration mode

Collect container logs

Container collection overview

LogCollector CRD collection mechanism

Collect container text logs (DaemonSet - Console method)

Collect container standard output (DaemonSet - Console method)

Collect container logs (DaemonSet - Pod environment variable method)

Collect container logs (DaemonSet - CRD method)

Collect container logs (Sidecar - Console method)

Collect container logs (Sidecar - CRD method)

Collect Docker container logs

Collect metric topic data

Collect host monitoring

Collect Syslog logs

LogCollector plugins

Plugin overview

Process logs using LogCollector plugins

Plugin execution conditions

Field extraction plugin

JSON parsing plugin

Add field plugin

Data desensitization plugin

String replacement plugin

IP address conversion plugin

JSON extraction plugin

Field splitting and reassembly plugin

URL parsing plugin

Field retention plugin

Manage LogCollector

LogCollector collection configuration management

Configure LogCollector startup parameters

Create anonymous identifiers to collect logs without AKSK

general reference

Time format

Container log path placeholder

Obtain container labels and environment variables

Aggregation rules for multi-line full text logs

Set IP addresses for hosts or containers

Collect multiple log files or standard output simultaneously

Relation between container standard output and log levels

Data import

Import logs from SLS

Import logs from TOS

Import data from Kafka

Import data from Elasticsearch

Cloud product log collection

Cloud product log collection overview

TOS log collection

Collect TOS logs

Manage TOS log resources

DBW log collection

Collect DBW logs

Manage DBW log resources

VPN Log collection

Collect VPN logs

Manage VPN log resources

VKE log collection

Collect VKE logs

Manage VKE log resources

CLB log collection

Collect CLB logs

Manage CLB log resources

CloudTrail log collection

Collect CloudTrail logs

Manage CloudTrail log resources

CFW log collection

Collect CFW logs

Manage CFW log resources

NTA log collection

Collect NTA logs

Manage NTA log resources

ALB log collection

Manage ALB log resources

Collect ALB logs

APMPlus log collection

Collect APMPlus logs

Manage APMPlus log resources

veFaaS log collection

Collect veFaaS logs

Manage veFaaS log resources

VPC log collection

Collect VPC logs

Manage VPC log resources

veImageX log collection

Collect veImageX logs

Manage veImageX log resources

TrafficRoute DNS log collection

Collect TrafficRoute DNS logs

Manage TrafficRoute DNS log resources

DDoS log collection

Manage DDoS log resources

Collect DDoS logs

WAF log collection

Collect WAF logs

Manage WAF log resources

DCDN log collection

Collect DCDN logs

Manage DCDN log resources

Multi-cloud security platform log collection

Collect multi-cloud security platform logs

Managing multi-cloud security platform log resources

APIG log collection

Collect APIG logs

Manage APIG log resources

Machine learning task log collection

Collect machine learning task logs

Manage machine learning task log resources

CR log collection

Collect CR logs

Manage CR log resources

Volcano Ark application lab log collection

Collect Volcano Ark Application Lab logs

Manage Volcano Ark Application Lab log resources

FileNAS log collection

Collect FileNAS logs

Manage FileNAS log resources

LLM Application Firewall Service log collection

Collect LLM Application Firewall Service logs

Manage LLM Application Firewall Service log resources

vePFS log collection

Collect vePFS logs

Manage vePFS log resources

Cloud security center log collection

Collect Cloud Security Center logs

Manage Cloud Security Center log resources

EFS log collection

Collect EFS logs

Manage EFS log resources

Upload logs using the Kafka protocol

Use WebTracking to collect logs

Upload logs using Logback Appender

Upload logs using Log4j2 Appender

Upload logs using Syslog protocol

Import SLS collection configuration

Data storage

Storage types

Data tamper-proof

Data encryption

Query and analysis

Query and analysis workflow

Configure log indexes

Query log

Single-topic log query

Multi-topic log query

Query logs via phrase query

Query logs via phrase query (Index edition)

Query the context of specified log

Analyze logs

Analysis overview

Analysis syntax

Log analysis

Asynchronous query and analysis of logs

Quick analysis for specified log fields

Query and analyze logs using third-party tools

TLS Elasticsearch-compatible API

Query logs using Kibana

Associate external data source for log query

Associate to MySQL data source

MySQL join query and analysis

Query and analyze metric topic

Quickly set up query and analysis statements using tap mode

Scroll logs in real time using LiveTail

Analyze data using log clustering

Download logs

SQL function

Functions overview

Aggregate functions

Date and time functions

String functions

Regular expression functions

Year-on-year and period-on-period functions

Mathematical calculation functions

Mathematical statistics functions

Color processing functions

Phone number function

Unit conversion functions

Window funnel functions

Window functions

Type conversion functions

HyperLogLog functions

Lambda expressions

Arithmetic operation syntax

Comparison operators

Logical operators

Conditional expressions

SQL syntax

Machine learning function

Overview of machine learning functions

Prediction and anomaly detection function

Time series clustering functions

Anomaly comparison function

Threshold recommendation function

Difference pattern statistical functions

Visualization

Query and analysis charts

Overview of query and analysis charts

Configure data analysis charts

Configure data classification charts

Configure data aggregation chart

Summaries of query and analysis charts

Create dashboard

Create a blank dashboard

Quickly create a dashboard using a preset template

Import dashboard

Use dashboard

Common dashboard actions

Add charts to dashboard

Configure drill-down events for dashboard chart

Subscribe to dashboard messages

Password-free dashboard sharing

Export dashboards in Grafana format

Dashboard JSON Model description

Log field description for preset dashboard

Add dashboard filters and variables

Introduction to dashboard filters and variables

Add dashboard filter

Add custom variables to dashboard

Add dashboard data source variables

Use preset system variables

Create variable cascade relationships

Add time series filter

Use other systems to analyze logs

Embed externally built systems into TLS console

Embedded parameter details in TLS Console

Visualize and analyze logs via Grafana

Log application

Overview of log applications

Log applications for product observability

CloudLens for TOS

Integrate TOS log application

Configure log analysis dashboard

Configure pivot dashboard

Log analysis fields

Data pivot fields

CloudLens for VKE

Log applications for large models observability

Coding Agent log app

Coding Agent log application overview

Claude observability

Codex observability

Cursor observability

DeepSeek Harness observability

OpenCode observability

Pi observability

Trae observability

WorkBuddy Observability

ModelProvider log app

Langfuse observability

TLS LLM observability

No-coding Agent log app

HermesAgent observability

OpenClaw Observability

Framework log app

Eino observability

LangGraph observability

LLM application observability (LangChain/Eino)

Log applications for basic service monitoring

JuiceFS access analysis

MQTT monitoring

NGINX access monitoring

Trace call chain analysis

Data processing

Data transformation

Data transformation overview

How data transformation works

Configure cross-account data transformation permissions

Create data transformation task

Manage data transformation tasks

View data transformation progress and status

Syntax and functions

Data transformation syntax

Functions overview

Global operation function

Flow control function

Event handler function

Event check function

Field processing function

Field check function

Key-value extraction function

Enrichment mapping function

expression function

Date value processing function

String handler

Type conversion functions

Logical expression functions

Arithmetic expression function

Regular expression functions

Dictionary functions

Encoding and decoding function

IP address parsing functions

Parsing functions

Data structuring functions

list functions

resource function

Data transformation performance guide

Data processing best practices

general reference

Data write processor

Write processor overview

Create write processor

Associate write processor

Manage write processor

Data consumer processor

Consumer processor overview

Create consumer processor

Manage consumer processor

Timed SQL Analysis

Scheduled SQL analysis overview

Principles of work

Create schedule SQL analysis and store data in topic

Create scheduled SQL analysis and store data in managed Prometheus

Manage scheduled SQL analysis tasks

Select time zone

SQL time window syntax

Data consumption and shipping

Data consumption and shipping overview

Consume logs using ConsumeLogs

Consume data via consumer group

Consume logs via Kafka protocol

Consume log data through Function Service

Ship logs to TOS

Overview of shipping logs to TOS

Configure cross-account log shipping permissions

Ship logs in JSON format

Ship logs in CSV format

Ship data in Parquet format

Manage shipping configurations

Decompress Snappy-compressed files

Ship logs to Kafka

Overview of shipping logs to Kafka

Configure cross-account log shipping permissions

Ship logs to Message Queue for Kafka

Manage Kafka shipping rules

Ship logs to Splunk

Ship logs to Splunk using SDK

Ship logs to Splunk using Add-On plugin

Alarm

Alarm overview

Quickly configure log alarm

Alarm policy

Create alarm policy

Manage alarm policy

Monitor alarm policy

Multi-statement set operation

Triggering condition expression

Timeliness of alarm monitoring task

Alarm policy testing

Alarm notification content

View alarm execution results

View alarm history

Alarm notification content template

Notification content overview

Create content template

Manage content template

Content syntax

Content variable

Content functions

alarm notification channel

Create notification group

Manage notification group

Create Webhook integration configuration

Tracing service

Trace feature overview

Trace data format

Create a universal trace instance

Create trace instance for VolcanoEngine Ark high-code application

Manage trace instances

Write data via OpenTelemetry SDK

View trace data via Grafana

Monitor Log Service

View cloud monitoring data

Service log

Service log overview

Enable or disable LogCollector service logs

Service log fields

Service log dashboard

Permission management

IAM overview

Authorizable actions

Authorizable resources

Examples of custom permission policy

Cross-service access authorization

API reference

Invocation method

Calculating a signature

Appendix

Project management API

Topic management API

Shard management API

Index management API

Log management API

DescribeDownloadTasks

DescribeDownloadUrl

CancelDownloadTask

Host group management API

CreateHostGroup

DeleteHostGroup

ModifyHostGroup

DescribeHostGroup (Deprecated)

DescribeHostGroupV2

DescribeHostGroups (deprecated)

DescribeHostGroupsV2

DescribeHosts

DeleteHost

DescribeHostGroupRules

ModifyHostGroupsAutoUpdate

DeleteAbnormalHosts

Collection configuration management API

CreateRule

DeleteRule

ModifyRule

DescribeRule (deprecated)

DescribeRuleV2

DescribeRules

ApplyRuleToHostGroups

DescribeBoundHostGroups

DeleteRuleFromHostGroups

Data import API

Data processor API

DescribeTopicsByProcessor

DescribeProcessorFunctions

DescribeProcessorByTopic

DescribeProcessorBindings

Data transformation API

Data shipping API

Data backflow API

CreateLogBackFlowTask

DeleteLogBackFlowTask

DescribeLogBackFlowTasks

ModifyLogBackFlowTask

SQL analysis API

CreateScheduleSqlTask

ModifyScheduleSqlTask

DeleteScheduleSqlTask

DescribeScheduleSqlTask

DescribeScheduleSqlTasks

Alarm management API

CreateAlarmNotifyGroup

DeleteAlarmNotifyGroup

ModifyAlarmNotifyGroup

DescribeAlarmNotifyGroups

CreateAlarmContentTemplate

DeleteAlarmContentTemplate

ModifyAlarmContentTemplate

DescribeAlarmContentTemplates

CreateAlarmWebhookIntegration

DeleteAlarmWebhookIntegration

DescribeAlarmWebhookIntegrations

ModifyAlarmWebhookIntegration

Consumer group management API

CreateConsumerGroup

DeleteConsumerGroup

DescribeConsumerGroups

Kafka protocol consumption API

OpenKafkaConsumer

CloseKafkaConsumer

DescribeKafkaConsumer

Tag management API

AddTagsToResource

RemoveTagsFromResource

Trace storage API

DescribeTraceInstances

DescribeTraceInstance

DescribeTrace

SearchTraces

Account management API

ActiveTlsAccount

GetAccountStatus

SDK reference

SDK overview

Configure identity authentication

Go SDK

Go SDK overview

Install TLS SDK for Go

Getting started

Use TLS SDK for Go to create basic resources

Use TLS SDK for Go to write logs

Use TLS SDK for Go to consume logs

Use TLS SDK for Go to query and analyze logs

Use TLS SDK for Go consumer groups to consume logs

Import the TOS data via Go SDK

Import Kafka data using the Go SDK

Send logs to TOS using Go SDK

Using the Go SDK to send logs to Kafka

Java SDK

Java SDK overview

Install TLS SDK for Java

Getting started

Use TLS SDK for Java to create basic resources

Use TLS SDK for Java to write logs

Use TLS SDK for Java to consume logs

Use TLS SDK for Java to query and analyze logs

Use TLS SDK for Java consumer groups to consume logs

Python SDK

Python SDK overview

Install TLS SDK for Python

Getting started

Use TLS SDK for Python consumer groups to consume logs

Android SDK

SDK for Android overview

Install TLS SDK for Android

Getting started

iOS SDK

SDK for iOS overview

Install TLS SDK for iOS

Getting started

Obtain SDK responses

Configure timeout and retry policies

C++ SDK

SDK for C++ Overview

Install TLS SDK for C++

Getting started

Get SDK response information

Configure timeout and retry policies

Node.js SDK

SDK for Node.js overview

Install TLS SDK for Node.js

Getting started

Browser JavaScript SDK

.NET SDK

CLI reference

Install and use TLS CLI

Best practices

Data collection

Upload logs using Flume

Upload logs using Logstash

Enable LogCollector high precision time

Monitor LogCollector collection

Configure network connections for cross-region log collection

Query and analysis

Query and analyze O&M logs

Query and analyze JSON logs

Query and analyze Nginx access logs

permission management

Access TLS via STS

Set TLS access permissions through IAM

Enable cross-account access to TLS based on IAM roles

Consumption and delivery

Consume logs via ByteHouse

Consume logs using Spark Streaming

Consume logs using Flink

Synchronize TLS data to EMR Hive in real time

Visualization of Kafka protocol consumption metrics

Visualization

Create a line chart with double Y-axes

Alarm

Trigger alerts based on log keywords

Common methods for controlling the time window for sending alert notifications

Log-based JuiceFS observability best practices

FAQs

Billing and cost FAQ

Product billing FAQ

How to view TLS resource usage

How to view TLS billing details

How to reduce TLS usage costs

How to stop TLS billing

How to handle receiving a bill for TLS when the service was not used

Data collection FAQ

How to install, upgrade, and configure LogCollector

How to handle LogCollector maintenance failures

How to troubleshoot log collection exceptions (host machine)

How to troubleshoot log collection exceptions (container)

How to troubleshoot host group heartbeat status Exception (host machine)

How to troubleshoot host group heartbeat exceptions (container)

How to handle high load on the K8S API Server caused by numerous ListWatch requests

Data storage FAQ

How to view the log storage period

How to delete log projects and log topics

How to handle log topics that cannot be deleted

Index configuration FAQ

How to configure indexes for logs

How to distinguish between JSON automatic index and manual index

Query and analysis FAQ

Common errors in query and analysis

What are the usage scenarios for quotes in retrieval analysis statements?

How to accurately query phrases?

How to perform a fuzzy query on logs?

How to improve the speed of search analysis?

Why is the log time inconsistent with its generation time?

How to handle inaccurate results

Scheduled SQL tasks FAQ

How to distinguish between scheduled SQL tasks and data transformation tasks

How to trace the data written by scheduled SQL tasks

How to handle scheduled SQL tasks that have not started scheduling

Alarm FAQ

Alert Policy FAQ

Alarm notification channel FAQ

Alarm service log FAQ

Alarm notification content FAQ

How to set up Feishu, DingTalk, and WeCom alarm notifications

API and SDK FAQ

Other FAQ

How to view IP address allowlist

What is the difference between the logging service and Datasail?

Torch Log Service

日志服务是火山引擎提供的针对日志类数据的一站式服务，提供日志采集、海量存储、检索分析、监控告警、数据可视化等功能，适用于应用运维、服务监控、等保合规等场景，全方位提升研发与运维效率。

立即使用产品动态产品简介

Learning Path

Step-by-step guide to help you quickly learn product features

了解

产品介绍

什么是日志服务功能特性相关概念使用限制

计费指南

计费概述计费项欠费说明

入门

新手指南

创建资源采集日志数据配置索引检索分析日志

进阶

开发者资源

API参考

使用前必读如何调用 APIAPI 概览通用错误码

Public security network record in Beijing: No. 11010802032137

Torch Log Service

Torch Log Service

Learning Path

了解

产品介绍

计费指南

入门

新手指南

进阶

资源管理

日志采集

检索分析

数据加工

可视化

告警

消费与投递

开发者资源

API参考