Torch Log Service
Torch Log Service
Torch Log Service
Document
Product News
Product Overview
Product Billing
User Guide
TLS Agent
AgentLoop
Resource management
Host group
Data collection
LogCollector collection
Introduction to LogCollector
Installation and upgrade
Collect host logs
Collect container logs
LogCollector plugins
Manage LogCollector
general reference
Data import
Cloud product log collection
TOS log collection
DBW log collection
VPN Log collection
VKE log collection
CLB log collection
CloudTrail log collection
CFW log collection
NTA log collection
ALB log collection
APMPlus log collection
veFaaS log collection
VPC log collection
veImageX log collection
TrafficRoute DNS log collection
DDoS log collection
WAF log collection
DCDN log collection
Multi-cloud security platform log collection
APIG log collection
Machine learning task log collection
CR log collection
Volcano Ark application lab log collection
FileNAS log collection
LLM Application Firewall Service log collection
vePFS log collection
Cloud security center log collection
EFS log collection
Data storage
Query and analysis
Configure log indexes
Query log
Analyze logs
Query and analyze logs using third-party tools
Associate external data source for log query
SQL function
SQL syntax
Visualization
Query and analysis charts
Summaries of query and analysis charts
Create dashboard
Use dashboard
Add dashboard filters and variables
Use other systems to analyze logs
Log application
Log applications for product observability
Log applications for large models observability
Coding Agent log app
ModelProvider log app
No-coding Agent log app
Log applications for basic service monitoring
Data processing
Data transformation
Syntax and functions
Global operation function
expression function
Data processing best practices
general reference
Data write processor
Data consumer processor
Data consumption and shipping
Ship logs to TOS
Ship logs to Kafka
Ship logs to Splunk
Alarm
Alarm policy
Alarm notification content
Alarm notification content template
alarm notification channel
Tracing service
Monitor Log Service
API reference
Project management API
Topic management API
Shard management API
Index management API
Log management API
Host group management API
Collection configuration management API
Data import API
Data processor API
Data transformation API
Data backflow API
SQL analysis API
Alarm management API
Consumer group management API
Kafka protocol consumption API
Tag management API
Trace storage API
Account management API
SDK reference
Go SDK
Java SDK
Python SDK
iOS SDK
C++ SDK
CLI reference
Best practices
Data collection
Query and analysis
permission management
Consumption and delivery
Visualization
Alarm
FAQs
Billing and cost FAQ
Data collection FAQ
Data storage FAQ
Index configuration FAQ
Query and analysis FAQ
Scheduled SQL tasks FAQ
Alarm FAQ
- Documentation
Torch Log ServiceUser GuideData consumption and shippingShip logs to SplunkShip logs to Splunk using SDK
Copy page
Download PDFShip logs to Splunk
Ship logs to Splunk using SDK
Copy pageDownload PDFShip logs to Splunk using SDK
This topic introduces the basic concepts and sample code for shipping Syslog logs to Splunk using TLS.
Features overview
TLS provides the capability to ship Syslog log data to Splunk. After consuming data with the ConsumeLogs API of TLS, the data is parsed into the corresponding Syslog format and then written to Splunk.
Notice
- Currently, RFC5424 and RFC3164 Syslog protocols are supported.
- When sending Syslog log data to the Splunk platform via TCP, the Splunk platform does not remove the original priority information in the event, that is, the priority information included in the Syslog data (such as the priority defined by the PRI field) is retained. For details, see the Splunk website.
- An available Splunk endpoint is required to receive data in advance.
Code examples
Refer to the following examples to obtain consumer group configurations, configure TLS topics, set up the syslog client, and use RFC5424 Syslog as an example to print each key-value pair of every consumed log entry one by one.
package tls
import (
"context"
"crypto/tls"
"fmt"
tlsConsumer "github.com/volcengine/volc-sdk-golang/service/tls/consumer"
"github.com/volcengine/volc-sdk-golang/service/tls/pb"
"github.com/volcengine/volc-sdk-golang/service/tls/siem"
"log"
"os"
"strconv"
"time"
)
func ConsumerLogsToSyslogClient() error {
// Get the default configuration for the consumer group
consumerCfg := tlsConsumer.GetDefaultConsumerConfig()
// Configure your Endpoint, Region, AccessKeyID, AccessKeySecret, and other essential information
consumerCfg.Endpoint = os.Getenv("VOLCENGINE_ENDPOINT")
consumerCfg.Region = os.Getenv("VOLCENGINE_REGION")
consumerCfg.AccessKeyID = os.Getenv("VOLCENGINE_ACCESS_KEY_ID")
consumerCfg.AccessKeySecret = os.Getenv("VOLCENGINE_ACCESS_KEY_SECRET")
// Configure your project ID and log topic ID list
consumerCfg.ProjectID = "<YOUR-PROJECT-ID>"
consumerCfg.TopicIDList = []string{"<YOUR-TOPIC-ID>"}
// Configure the name of your consumer group (if you haven't created one, the SDK will automatically create a consumer group with the specified name)
consumerCfg.ConsumerGroupName = "<CONSUMER-GROUP-NAME>"
// Configure the consumer name (consumers in the same consumer group must use unique names)
consumerCfg.ConsumerName = "<CONSUMER_NAME>"
// Set up the syslog client
syslogClient := siem.NewSyslogClient()
// Set client configurations
syslogClient.Host = os.Getenv("SPLUNK_HOST")
syslogClient.Port, _ = strconv.Atoi(os.Getenv("SPLUNK_PORT_UDP"))
// For SSL authentication, configure as needed. Note: Do not confuse the TLS here with the logging TLS.
syslogClient.SslSettings = &tls.Config{}
// Start the client
err := syslogClient.Connect()
if err != nil {
log.Fatalf("syslog client connect error %v:", err)
}
// Define the log consumption function. You can implement a function to process LogGroupList based on your business needs
// Using RFC5424 syslog as an example, the following code demonstrates how to print each key-value pair of every log consumed, one by one.
var handleLogsFunc = func(topicID string, shardID int, l *pb.LogGroupList) {
for _, logGroup := range l.LogGroups {
for _, tlsLog := range logGroup.Logs {
// If using RFC3164, use NewSyslogFormatRFC3164
syslogRFC5424 := siem.NewSyslogFormatRFC5424()
for _, content := range tlsLog.Contents {
switch content.Key {
// Update the fields to match the actual fields in the logs
case "pri":
syslogRFC5424.Priority = content.Value
// Update the fields to match the actual fields in the logs
case "version":
syslogRFC5424.Version = content.Value
// Update the fields to match the actual fields in the logs
case "timestamp":
syslogRFC5424.Timestamp, _ = time.Parse(time.RFC3339, content.Value)
// Update the fields to match the actual fields in the logs
case "hostname":
syslogRFC5424.Hostname = content.Value
// Update the fields to match the actual fields in the logs
case "appName":
syslogRFC5424.AppName = content.Value
// Update the fields to match the actual fields in the logs
case "pid":
syslogRFC5424.Pid = content.Value
// Update the fields to match the actual fields in the logs
case "msgId":
syslogRFC5424.MsgId = content.Value
// Update the fields to match the actual fields in the logs
case "message":
syslogRFC5424.Message = content.Value
}
}
syslogRFC5424Format, err := syslogRFC5424.FormatSyslogMessageWithRFC5424()
if err != nil {
// Handle formatting errors here
log.Printf("NOTICE: syslogRFC5424.FormatSyslogMessageWithRFC5424() failed: %v", err)
}
// When sending fails, it will automatically retry once
err = syslogClient.Send(syslogRFC5424Format)
if err != nil {
// Handle logic for send failure here
return
}
fmt.Println("Send successfully")
}
}
}
tlsConsumerIns, err := tlsConsumer.NewConsumer(context.TODO(), consumerCfg, handleLogsFunc)
if err != nil {
log.Fatalf("NewConsumer error %v", err)
}
// Start consuming logs
if err := tlsConsumerIns.Start(); err != nil {
return fmt.Errorf("start consumer failed, error is %v", err)
}
// Control log consumption exit based on your actual scenario
time.Sleep(300 * time.Second)
_ = syslogClient.Close()
tlsConsumerIns.Stop()
return nil
}
Last updated: 2026.05.06 14:26:39
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
