This topic explains how to upload logs to TLS using Syslog protocol.

Background information

Syslog is a standardized log delivery protocol used to collect, forward, and store system logs. It supports transmission protocols such as UDP and TCP, but traditional implementations offer relatively basic features. For example, on Linux, it typically runs as the default log service. ‌

TLS supports using the SYSLOG-NG tool to upload Syslog logs by configuring the syslog-ng.conf file. You can use TLS's query and analysis, processing, consumption, delivery, monitoring and alerting, and visualization features to implement business operation monitoring and data analytics scenarios.

Limits

The Syslog protocol must comply with the standard RFC 5424 specification.

Time can be accurate to the millisecond.

Prerequisites

SYSLOG-NG is installed.

A log project and log topic are created. To further query and analyze logs, you need to configure indexes for the log topic.

The IAM role is created, and the Access Key ID and Secret Access Key are obtained. The user is granted TLSFullAccess permissions for the log topic. For more details, see IAM Overview.

Configuration steps

Create and run the following script to obtain AUTHORIZATION for TLS.

Replace self.ak and self.sk with the actual Access Key ID and Secret Access Key based on your actual environment.

import hashlib
import hmac
import base64

class Util:
@staticmethod
def hmac(key, string):
return hmac.new(key, string.encode('utf-8'), hashlib.sha256).digest()

@staticmethod
def sha256(data):
return hashlib.sha256(data.encode('utf-8')).digest()

class tlsSigners:
def __init__(self,ak,sk):
self.service_name = "TLS"
self.region = "cn-north-1"
self.algorithm = "HMAC-SHA256"
self.v4Identifier = "request"
self.ak = ak
self.sk = sk
self.body = "{}"
self.path = "/adapter/"
self.method = "POST"
self.query = ""
self.datetime = "20250101T000000Z"
self.request = {"headers": ["x-content-sha256","x-date"]}

def authorization(self):
cred_string = self.create_scope()
parts = [
f"HMAC-SHA256 Credential={self.ak}/{cred_string}",
f"SignedHeaders={self.signed_headers()}",
f"Signature={self.signature()}"
]
token = ', '.join(parts)
return token

def signature(self):
signing_key = self.get_signing_key()
sign_str = self.get_sign_str()
return Util.hmac(signing_key,sign_str).hex()

def get_signing_key(self):
k_date = Util.hmac(f"{self.sk}".encode('utf-8'),self.datetime[:8])
region = self.region
k_region = Util.hmac(k_date,region)
k_service = Util.hmac(k_region,self.service_name)
sign_key = Util.hmac(k_service,self.v4Identifier)
return sign_key

def get_sign_str(self):
parts = [
self.algorithm,
self.datetime,
self.create_scope(),
Util.sha256(self.get_canonical_string()).hex()
]
return '\n'.join(parts)

def get_canonical_string(self):
parts = [
self.method,
self.path,
self.query,
"x-content-sha256:"+Util.sha256(self.body).hex(),
"x-date:" + self.datetime,
"",
self.request["headers"][0]+";"+self.request["headers"][1],
Util.sha256(self.body).hex(),
]
return '\n'.join(parts)

def signed_headers(self):
headers = sorted(k.lower() for k in self.request['headers'])
return ';'.join(headers)

def create_scope(self):
return '/'.join( [self.datetime[0:8], self.region, self.service_name, self.v4Identifier])

if __name__ == '__main__':
t = tlsSigners("your_ak","your_sk")
print(t.authorization())

Configure the SYSLOG-NG syslog-ng.conf file (typically located in the /etc/syslog-ng/ directory).

Replace some parameters with values appropriate for your environment:

{AUTHORIZATION}: Replace with the AUTHORIZATION value obtained in the previous step.

{TLS_ENDPOINT}: Replace with the ENDPOINT of the TLS. For details, see Service address.

{TOPIC_ID}: Replace with the log topic ID in TLS.

# Data source definition
source s_network {
default-network-drivers(
flags(syslog-protocol)
);
};

# Destination, TLS service
destination d_http {
http(
url("https://{TLS_ENDPOINT}:10009/PutSysLogs?TopicId={TOPIC_ID}")
headers(
"Content-Type:text/plain",
"Authorization:{AUTHORIZATION}"
)
content-compression("gzip")
accept-encoding("all")
body("<${PRI}>1 ${ISODATE:--} ${HOST:--} ${PROGRAM:--} ${PID:--} ${MSGID:--} ${SDATA:--} ${MESSAGE}")
batch-bytes(65536) # The size of the message data sent for a single request
batch-timeout(1000) # Maximum waiting time for batch
);
};

# Data forwarding configuration
log {
source(s_network);
destination(d_http);
};

Restart SYSLOG-NG.

sudo /etc/init.d/syslog-ng restart

Generate test logs in the SYSLOG-NG container environment.

# -n specifies the maximum expected number of messages to send; the actual number sent may be less than this value.
loggen -i -S -P localhost 601 -n 10000

Log in to the TLS console and check the logs.

Log in to TLS console.

Select the region where TLS is located from the top navigation bar.

On the left navigation bar, select Common functions > Query and analysis.

On Query and analysis page, a list is displayed in the left area. Select the specified project and topic.

On the query and analysis page for the specified log topic, set the time range.

By default, the time range is 5 minutes. You can also customize the time range for query. If log data does not fall within the specified query time range, it will not be retrieved.

In search box, enter the query and analysis statement and click Search. For details, see Log analysis.

Torch Log Service