Torch Log Service
Torch Log Service
Document
- Documentation
Torch Log ServiceSDK referenceGo SDKUse TLS SDK for Go to consume logs
Copy page
Download PDFGo SDK
Use TLS SDK for Go to consume logs
Copy pageDownload PDFUse TLS SDK for Go to consume logs
Torch Log Service (TLS) allows you to use an SDK to consume log data collected to the server. This topic offers sample code that demonstrates how to use TLS SDK for Go to consume logs.
Before you begin
- Install TLS SDK for Go. For more information, refer to Install TLS SDK for Go.
- Run the following command to install the proto dependency:
- go get -u github.com/gogo/protobuf/proto
- Add environment variables such as VOLCENGINE_ACCESS_KEY_ID. For information on how to configure environment variables, refer to Configure identity authentication.
warning
We recommend that you use environment variables to dynamically get identity authentication information such as Volcano Engine AccessKey. Otherwise, hard-coded AccessKey information leads to data security risks.
Consuming logs
Scenario description
This topic offers sample code that demonstrates how to use TLS SDK for Go to consume log data. TLS SDK for Go supports the following methods to write log data:
Method | Description |
Consumer | Recommended. In the actual production environment, use Go Consumer to consume log data to improve efficiency. Go Consumer consumes the data of all shards in a topic in a balanced way. Go Consumer features asynchronous consumption, high performance, retry on failure, and graceful shutdown. To get the sample code, refer to Using Go Consumer to consume logs. For information on how to use consumer groups to consume logs, refer to Use ConsumeLogs to consume logs and Use TLS SDK for Go consumer groups to consume logs. |
ConsumeLogs | Not recommended. TLS allows you to call ConsumeLogs to consume logs using synchronous requests. The progress of consuming logs is limited by the read and write performance of a single shard and must be maintained manually. In addition, the consumption logic and process are cumbersome in automatic sharding scenarios. If you use the HashKey Routing Shard mode when you call PutLogs, log data is written to the specified shard in sequence. In this case, you can call ConsumeLogs to consume the log data in a specific shard. To get the sample code, refer to Using the synchronous ConsumeLogs API action to consume logs. |
Use Go Consumer to consume logs
The following sample code demonstrates how to use Go Consumer to consume log data.
package tls
import (
"context"
"fmt"
"os"
"time"
"github.com/pkg/errors"
log_consumer "github.com/volcengine/volc-sdk-golang/service/tls/consumer"
"github.com/volcengine/volc-sdk-golang/service/tls/pb"
)
func launchConsumer() error {
// Get the default configurations of a consumer group.
consumerCfg := log_consumer.GetDefaultConsumerConfig()
// Specify basic information such as Endpoint, Region, AccessKeyID, and AccessKeySecret.
consumerCfg.Endpoint = os.Getenv("VOLCENGINE_ENDPOINT")
consumerCfg.Region = os.Getenv("VOLCENGINE_REGION")
consumerCfg.AccessKeyID = os.Getenv("VOLCENGINE_ACCESS_KEY_ID")
consumerCfg.AccessKeySecret = os.Getenv("VOLCENGINE_ACCESS_KEY_SECRET")
// Specify your project ID and topic ID list.
consumerCfg.ProjectID = "<YOUR-PROJECT-ID>"
consumerCfg.TopicIDList = []string{"<YOUR-TOPIC-ID>"}
// Specify the name of the consumer group. If you did not create a consumer group, the SDK creates a consumer group with the specified name by default.
consumerCfg.ConsumerGroupName = "<CONSUMER-GROUP-NAME>"
// Specify the name of the consumer. The name must be unique in the consumer group.
consumerCfg.ConsumerName = "<CONSUMER_NAME>"
// Implement a log consumption function to handle LogGroupList as needed.
// The following sample code prints the key-value pair of each log entry that is consumed.
var handleLogs = func(topicID string, shardID int, l *pb.LogGroupList) {
fmt.Printf("received new logs from topic: %s, shard: %d\n", topicID, shardID)
for _, logGroup := range l.LogGroups {
for _, log := range logGroup.Logs {
for _, content := range log.Contents {
fmt.Printf("%s: %s\n", content.Key, content.Value)
}
}
}
}
// Create a consumer.
consumer, err := log_consumer.NewConsumer(context.TODO(), consumerCfg, handleLogs)
if err != nil {
return errors.Wrap(err, "get new consumer failed: ")
}
// Start the consumer.
if err := consumer.Start(); err != nil {
return errors.Wrap(err, "start consumer failed: ")
}
// Wait for log data to be consumed.
<-time.After(time.Second * 60)
// Stop the consumer.
consumer.Stop()
return nil
}
func main() {
if err := launchConsumer(); err != nil {
fmt.Println(err.Error())
}
}
Use the synchronous ConsumeLogs API action to consume logs
The following sample code demonstrates how to use the synchronous ConsumeLogs API action to consume log data.
package tls
import (
"fmt"
"github.com/volcengine/volc-sdk-golang/service/tls"
"github.com/volcengine/volc-sdk-golang/service/tls/pb"
)
func main() {
// When you initialize the client, we recommend that you use environment variables to dynamically get identity authentication information such as Volcano Engine AccessKey. Otherwise, hard-coded AccessKey information leads to data security risks.
// If you use Security Token Service (STS), use a temporary AccessKey ID and AccessKey secret, and set VOLCENGINE_TOKEN. If you do not use STS, leave VOLCENGINE_TOKEN empty.
client := tls.NewClient(os.Getenv("VOLCENGINE_ENDPOINT"), os.Getenv("VOLCENGINE_ACCESS_KEY_ID"),
os.Getenv("VOLCENGINE_ACCESS_KEY_SECRET"), os.Getenv("VOLCENGINE_TOKEN"), os.Getenv("VOLCENGINE_REGION"))
// Specify TopicID and shardID from which you want to consume logs.
topicID := "TopicId"
shardID := 0
// Get the start cursor of log consumption.
describeCursorResp, err := client.DescribeCursor(&tls.DescribeCursorRequest{
TopicID: topicID,
ShardID: shardID,
From: "begin",
})
if err != nil {
// Handle an error.
fmt.Println(err.Error())
}
beginCursor := describeCursorResp.Cursor
// Consume log data.
// Specify parameters such as TopicId, ShardId, Cursor, LogGroupCount, and Compression as needed. We recommend that you use LZ4 for compression.
// Call DescribeCursor again to get the end cursor of log consumption as the value of EndCursor in the ConsumeLogs API request.
logGroupCount := 1000
compression := "lz4"
resp, err := client.ConsumeLogs(&tls.ConsumeLogsRequest{
TopicID: topicID,
ShardID: shardID,
Cursor: beginCursor,
LogGroupCount: &logGroupCount,
Compression: &compression,
})
if err != nil {
// Handle an error.
fmt.Println(err.Error())
}
handleLogs(topicID, shardID, resp.Logs)
}
// Implement a log consumption function to handle LogGroupList as needed.
// The following sample code prints the key-value pair of each log entry that is consumed.
func handleLogs(topicID string, shardID int, l *pb.LogGroupList) {
fmt.Printf("received new logs from topic: %s, shard: %d\n", topicID, shardID)
for _, logGroup := range l.LogGroups {
for _, log := range logGroup.Logs {
for _, content := range log.Contents {
fmt.Printf("%s: %s\n", content.Key, content.Value)
}
}
}
}
Related documents
- After you use the SDK to send an API request, the server returns a response. If the response contains a non-200 status code, the API call fails. To view the error code and error message, refer to API reference.
- For more information on Go Consumer, refer to Go Consumer.
- To get the complete sample code of consuming logs using Go Consumer, refer to TLS Go SDK Demo on GitHub.
Last updated: 2025.10.11 16:44:07
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
