Docs

Torch Log Service

Data collection

Upload logs using Flume

Flume is a distributed, highly reliable system for massive log collection, aggregation, and transmission. It supports collecting data from various applications and uploading it to a storage system. This topic explains how to use Flume's Kafka Sink to upload data to TLS.

Background information

When Flume is used as a data collection tool, its Kafka Sink can send data from the Flume Channel to Kafka. Since TLS supports receiving data via the Kafka protocol, Flume can upload data to a TLS log topic via the Kafka Sink.

Prerequisites

The log project and log topic are created. For detailed instructions, see Create project and Create topic.

We recommend using IAM users for access authorization. Before using an IAM user, ensure that the IAM user has been created and has permission to upload logs via the Kafka protocol, specifically the PutLogs permission. For more details, see Authorizable operations.

Flume configuration

When uploading data to TLS via Flume, you need configure the Kafka Sink provided by Flume. This section lists only the parameters required for uploading data to TLS and parameters affecting upload performance. For other parameters, see Kafka Sink.

Notice

This documentation uses Flume version 1.9.0 as an example. Configuration parameters may vary for different Flume versions. For details, see Flume official documentation.

Name	Required	Example value	Description
type	Yes	org.apache.flume.sink.kafka.KafkaSink	Must be configured as org.apache.flume.sink.kafka.KafkaSink.
kafka.bootstrap.servers	Yes	tls-cn-beijing.volces.com:9094	The initial cluster connection address is formatted as service address:port, where: The service address is the entry point for TLS. Select the correct service entry point based on region and network type. For details, see Service address. The port number is fixed at 9094. Notice https:// is not required to be specified in service address.
kafka.producer.security.protocol	Yes	SASL_SSL	To ensure data transmission security, you must set the connection protocol to SASL_SSL.
kafka.producer.sasl.mechanism	Yes	PLAIN	Set the authentication method for uploading data to TLS. Set this to PLAIN.
kafka.producer.sasl.jaas.config	Yes	None	Specify the authentication settings for uploading data to TLS, including username and password. Set username to the TLS project ID. Set password to the Volcengine account access key. You can view and copy the Access Key ID and Secret Access Key. Just go to the Volcengine console Key Management page and follow the instructions. The configuration here uses the format ${access-key-id}#${access-key-secret}. org.apache.kafka.common.security.plain.PlainLoginModule required username="Log service project ID" password="Volcengine account secrect key";
kafka.topic	No	f93fc61xxxxefec8a5	Kafka topic. Set this to the topic ID for TLS. When Flume uploads data, Flume uploads data to this log topic in TLS.
flumeBatchSize	No	100	Number of records uploaded each time. Larger values increase throughput but also raise latency. The default value is 100 and modification is not recommended.
kafka.producer.acks	No	1	The message acknowledgment mechanism for Kafka producer. The default value is 1, which means the Leader broker writes successfully. Modification is not recommended.
kafka.producer.batch.size	No	1048576	The batch data size for Kafka producers, measured in bytes. Notice We recommend setting it to 1048576, which equals 1MB.
kafka.producer.linger.ms	No	100	The waiting time for Kafka producers before sending messages in batches, measured in milliseconds. Notice We recommend setting it to 100.
kafka.producer.compression.type	No	lz4	The compression method used by Kafka producer for sending messages. We recommend using lz4.

Configuration example

The following configuration shows how Flume uploads data from the /var/log/test1/example.log log file to TLS.

a1.sources = r1​
a1.sinks = k1​
a1.channels = c1​
​
a1.sources.r1.type = TAILDIR​
a1.sources.r1.channels = c1​
a1.sources.r1.filegroups = f1​
a1.sources.r1.filegroups.f1 = /var/log/test1/example.log #Replace this with the actual log file​
​
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink​
a1.sinks.k1.kafka.bootstrap.servers = tls-cn-beijing.ivolces.com:9094​
a1.sinks.k1.channel = c1​
a1.sinks.k1.kafka.topic = 5f93fcxxxxfec8a5​
a1.sinks.k1.kafka.producer.acks = 1​
a1.sinks.k1.kafka.producer.batch.size = 1048576​
a1.sinks.k1.kafka.producer.linger.ms = 100​
a1.sinks.k1.kafka.producer.compression.type = lz4​
a1.sinks.k1.kafka.producer.security.protocol = SASL_SSL​
a1.sinks.k1.kafka.producer.sasl.mechanism = PLAIN​
a1.sinks.k1.kafka.producer.sasl.jaas.config = org.apache.kafka.common.security.plain.PlainLoginModule required username="20fxxxx8ad7af" password="AKxxxx#T0RNME5EUTFZVxxxx";​
​
a1.channels.c1.type = memory​

Last updated: 2026.05.26 11:31:26