Docs

Torch Log Service

Data collection

Upload logs using Logstash

Upload logs using Logstash

TLS supports uploading logs collected by Logstash using the following two Logstash plugins. This topic describes how to upload logs to TLS with the ve-tls-logstash-output plugin.

Plugin	Description
logstash_output_kafka	Kafka output plugin built into Logstash that uploads log data to TLS using the Kafka protocol. For details, see Upload logs using Logstash.
ve-tls-logstash-output	Logstash output plugin developed for TLS, enabling output of logs to TLS.

Prerequisites

The log project and log topic are created. For detailed instructions, see Create project and Create topic.

We recommend using IAM users for access authorization. Before using IAM users, ensure that an IAM user has been created and has the permission to upload logs, specifically the PutLogs permission. For more details, see Authorizable operations.

Install plugin

Obtain the ve-tls-logstash-output plugin package.

Run the following command to install the plugin package locally.

logstash-plugin install --local /xxxxxx/ve-tls-logstash-output-0.0.1.gem

Run the following command to view the installed plugins and their versions.

logstash-plugin list --verbose ve-tls-logstash-output

Configure the following output in Logstash's pipeline configuration file, such as logstash.conf.

output {
vetls {
endpoint => "your project endpoint"
region => "your region"
topic => "your topic id"
access_key_id => "your access id"
access_key_secret => "your access key"
max_send_retry => 3
max_batch_count => 4096
max_batch_size_bytes => 2097152
linger_ms => 2000
total_size_in_bytes => 104857600
time_key => "@timestamp"
}
}

Parameters	Parameter type	Required	Note
endpoint	string	Yes	The service address for TLS. For details, see Service address. Example: Private network service address: https://tls-cn-hongkong.ivolces.com Public network service address: https://tls-cn-hongkong.volces.com
region	string	Yes	The region where TLS is located. This region must match the endpoint. Example: cn-hongkong.
topic	string	Yes	Log topic ID for TLS.
access_key_id	string	Yes	Access Key ID for the Volcengine account or IAM user. We recommend using IAM user credentials. Ensure this user has the required permissions for operations.
access_key_secret	string	Yes	AccessKey Secret for the VolcanoEngine account or IAM user We recommend using IAM user credentials. Ensure this user has the required permissions for operations.
source	string	No	The source field in the uploaded logs.
max_send_retry	number	No	The number of retries after a failed upload
max_batch_count	number	No	The maximum number of cached log entries sent at once. The default is 4096.
max_batch_size_bytes	number	No	The upper limit of bytes for cached logs sent at once. The default is 2,097,152 (2MB).
linger_ms	number	No	The maximum time limit for sending cached logs is 2 seconds by default.
total_size_in_bytes	number	No	The maximum total byte size of cached logs is 104,857,600 (100MB) by default.
time_key	string	No	The timestamp field in the uploaded logs defaults to "@timestamp".

Last updated: 2026.05.26 11:31:26