Docs

Torch Log Service

Data consumption and shipping

Data consumption and shipping overview

TLS provides real-time log consumption and shipping features. You can use the SDK to consume data collected in TLS to third-party tools and systems, or ship it to other products such as Volcanoengine TOS for subsequent log processing and big data analysis.

Features overview

TLS provides real-time log consumption and shipping features for big data real-time computing, offline analysis, and long-term archiving scenarios.

Log consumption refers to the process in which applications or systems such as various third-party tools and stream computing frameworks consume data in log topics in real time through SDKs or other methods provided by the log service. The log consumption feature supports reading data from the log service in write order. This enables TLS to act as a data pipeline for information flow and connect to various downstream consumption systems.

Log shipping refers to delivering data collected by TLS to products such as Volcanoengine TOS for long-term storage of infrequently accessed data, integration with other system consumption, and data analysis scenarios.

The comparison between log consumption and log shipping features is as follows:

Key features	Consumption	Shipping
Use cases	Real-time computing, data analysis	Long-term data storage and offline analysis
Using method	Implement with your own code	Configure using the console wizard
Data timeliness	Real-time	Offline
Original format	Raw log	Raw logs, JSON, CSV
Data compression format	No compression, Snappy, Gzip, LZ4	No compression, Snappy, Gzip, LZ4
Usage options	Standard consumption, consumer group consumption, Kafka protocol consumption	Volcengine TOS, Kafka
Use cases	Stream processing: log data is consumed in real time and sent to downstream stream processing systems. Data warehouse/offline processing: log data is consumed and sent to downstream big data components or data warehouses.	Data archiving: for example, log data is delivered to low-frequency storage, enabling audit compliance or data archiving. Stream processing: log data is consumed in real time and sent to downstream stream processing systems.

Data consumption and shipping solutions

Feature	Solution	Description	Related documentation
Data consumption	Consume logs via Kafka protocol	Consume data from log topics in real time to third-party tools or systems using the Kafka protocol in the stream computing framework.	Consume logs via Kafka protocol Spark Streaming Spend Logs via Flink Spend Logs via ByteHouse
		General consumption	Consume the data in the log topic using the ConsumeLogs API provided by TLS.	Consume logs via ConsumeLogs
		Consume with consumer group	Use multi-language applications as consumer groups to consume data from TLS.	Consumption data by consumption group
		Consume with function service	Use the TLS trigger provided by Volcanoengine function service to consume log data in real time. Perform any custom data processing, analysis, and routing to meet log handling, cleaning, and shipping needs.	Consume log data through function services
Log shipping	Ship data to TOS	Ship log data to Volcanoenngine Torch Object Storage based on defined shipping rules for long-term storage and archiving.	An overview of shipping logs to TOS
		Ship logs to Kafka	Ship log data to Volcanoengine Message Queue Kafka Edition according to shipping rules for downstream third-party systems consumption.	Deliver logs to the Kafka message queue
		Ship logs to Splunk	Ship log data to Splunk based on defined shipping rules for data analysis on the Splunk platform.	Send logs to Splunk

Billing description

The billing items related to log consumption and log shipping are as follows. For detailed billing rules, see Billing items based on pay-by-feature mode.

The log consumption feature itself does not incur service fees. However, if you access log data via the TLS public domain, the corresponding public network data transfer fees will be incurred. If you call the data read API of TLS, service request fees will be incurred.

Log shipping charges for data transfer are based on the amount of data delivered to TOS.

Last updated: 2026.03.23 11:52:09