Torch Log Service
Torch Log Service
Torch Log Service
Document
Product News
Product Overview
Product Billing
User Guide
TLS Agent
AgentLoop
Resource management
Host group
Data collection
LogCollector collection
Introduction to LogCollector
Installation and upgrade
Collect host logs
Collect container logs
LogCollector plugins
Manage LogCollector
general reference
Data import
Cloud product log collection
TOS log collection
DBW log collection
VPN Log collection
VKE log collection
CLB log collection
CloudTrail log collection
CFW log collection
NTA log collection
ALB log collection
APMPlus log collection
veFaaS log collection
VPC log collection
veImageX log collection
TrafficRoute DNS log collection
DDoS log collection
WAF log collection
DCDN log collection
Multi-cloud security platform log collection
APIG log collection
Machine learning task log collection
CR log collection
Volcano Ark application lab log collection
FileNAS log collection
LLM Application Firewall Service log collection
vePFS log collection
Cloud security center log collection
EFS log collection
Data storage
Query and analysis
Configure log indexes
Query log
Analyze logs
Query and analyze logs using third-party tools
Associate external data source for log query
SQL function
SQL syntax
Visualization
Query and analysis charts
Summaries of query and analysis charts
Create dashboard
Use dashboard
Add dashboard filters and variables
Use other systems to analyze logs
Log application
Log applications for product observability
Log applications for large models observability
Coding Agent log app
ModelProvider log app
No-coding Agent log app
Log applications for basic service monitoring
Data processing
Data transformation
Syntax and functions
Global operation function
expression function
Data processing best practices
general reference
Data write processor
Data consumer processor
Data consumption and shipping
Ship logs to TOS
Ship logs to Kafka
Ship logs to Splunk
Alarm
Alarm policy
Alarm notification content
Alarm notification content template
alarm notification channel
Tracing service
Monitor Log Service
API reference
Project management API
Topic management API
Shard management API
Index management API
Log management API
Host group management API
Collection configuration management API
Data import API
Data processor API
Data transformation API
Data backflow API
SQL analysis API
Alarm management API
Consumer group management API
Kafka protocol consumption API
Tag management API
Trace storage API
Account management API
SDK reference
Go SDK
Java SDK
Python SDK
iOS SDK
C++ SDK
CLI reference
Best practices
Data collection
Query and analysis
permission management
Consumption and delivery
Visualization
Alarm
FAQs
Billing and cost FAQ
Data collection FAQ
Data storage FAQ
Index configuration FAQ
Query and analysis FAQ
Scheduled SQL tasks FAQ
Alarm FAQ
- Documentation
Torch Log ServiceSDK referenceGo SDKImport Kafka data using the Go SDK
Copy page
Download PDFGo SDK
Import Kafka data using the Go SDK
Copy pageDownload PDFImport Kafka data using the Go SDK
TLS supports importing Kafka data via the SDK. This document demonstrates how to import data from Kafka into TLS using sample code with the Go SDK.
Prerequisites
- TLS Go SDK has been installed. For more information, see Install Go SDK.
- The following commands have been executed to install the proto dependency package.
- go get -u github.com/gogo/protobuf/proto
- The VOLCENGINE_ACCESS_KEY_ID and other environment variables have been added. For details on the environment variable configuration method, please refer to Configure authentication information.
warning
It is recommended to use environment variables to dynamically obtain Volcengine keys and other authentication information in order to avoid data security risks caused by hardcoding AccessKey.
Scenario description
This document uses sample code to demonstrate how to use the GO SDK to create a Kafka data import task in a specified project. For related feature overview, important notes, and cost details, please refer to Import data from Kafka.
Sample code
package demo
import (
"github.com/volcengine/volc-sdk-golang/service/tls"
"os"
)
func CreateImportKafkaTaskDemo() (*tls.CreateImportTaskResponse, error) {
// When initializing the client, it's recommended to dynamically obtain the Volcano Engine key and other authentication information through environment variables to avoid data security risks caused by hard-coding the AccessKey. For detailed instructions, see https://www.volcengine.com/docs/6470/1166455.
// When using STS, both ak and sk use temporary keys and set VOLCENGINE_TOKEN; when not using STS, the VOLCENGINE_TOKEN part is passed empty.
client := tls.NewClient(os.Getenv("VOLCENGINE_ENDPOINT"), os.Getenv("VOLCENGINE_ACCESS_KEY_ID"), os.Getenv("VOLCENGINE_ACCESS_KEY_SECRET"), os.Getenv("VOLCENGINE_TOKEN"), os.Getenv("VOLCENGINE_REGION"))
//Create a data import task.
//For a description of the CreateImportTask API request parameters, see https://www.volcengine.com/docs/6470/1333757.
createImportTaskResp, err := client.CreateImportTask(&tls.CreateImportTaskRequest{
TopicID: "fa**",
TaskName: "task-name-1",
SourceType: "kafka",
Description: "this is a task",
ImportSourceInfo: &tls.ImportSourceInfo{
KafkaSourceInfo: &tls.KafkaSourceInfo{
Host: "kafka-cnngc7an0qpv****.kafka.ivolces.com:9092",
Group: "mygroup",
Topic: "test",
Encode: "UTF-8",
Password: "",
Protocol: "",
Username: "",
Mechanism: "",
InstanceID: "",
IsNeedAuth: "false",
InitialOffset: 0,
TimeSourceDefault: 0,
},
},
TargetInfo: &tls.TargetInfo{
Region: "cn-shanghai",
LogType: "json_log",
ExtractRule: &tls.ImportExtractRule{
TimeZone: "GMT",
SkipLineCount: 0,
TimeExtractRegex: "",
ExtractRule: &tls.ExtractRule{
Quote: "\"",
},
},
},
})
if err != nil {
return nil, err
}
return createImportTaskResp, nil
}
Related documentation
- After you send an API request through the SDK, you will receive a response from the server. If the response contains a status code other than 200, it indicates that the API call has failed. You can refer to the documentation for each API to view the corresponding error code information.
- The APIs related to data import tasks are as follows. You can refer to the API documentation to view detailed information about the corresponding interface.
- APIDescriptionAPIDescriptionCall the CreateImportTask API to create a data import task.Call the DeleteImportTask API to delete the specified data import task.Call the ModifyImportTask API to modify the specified data import task.Call the DescribeImportTask API to obtain information about a specific data import task.Call the DescribeImportTasks API to obtain the list of data import tasks.
Last updated: 2025.10.11 16:44:08
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
