Compatibility principle

Elasticsearch is an open-source, distributed query and analytics engine based on Apache Lucene. It supports real-time storage of structured, unstructured, and vector data, providing fast hybrid and vector query capabilities.

TLS provides compatibility with the Elasticsearch API by translating standard Elasticsearch DSL queries into TLS index and SQL analytic queries, and returning results in the standard Elasticsearch API format to ensure protocol compatibility.

ES-compatible API domain name

Elasticsearch-compatible API access domain for TLS is as follows:

Format: https://${tls-endpoint}/es/

Example: https://tls-cn-beijing.volces.com/es/

Account and password for ES-compatible API access

Example of account information for TLS Elasticsearch-compatible API access:

import hashlib

import hmac

import base64

class Util:

@staticmethod

def hmac(key, string):

return hmac.new(key, string.encode('utf-8'), hashlib.sha256).digest()

@staticmethod

def sha256(data):

return hashlib.sha256(data.encode('utf-8')).digest()

class tlsSigners:

def __init__(self,ak,sk):

self.service_name = "TLS"

self.region = "cn-north-1"

self.algorithm = "HMAC-SHA256"

self.v4Identifier = "request"

self.ak = ak

self.sk = sk

self.body = "{}"

self.path = "/es/"

self.method = "POST"

self.query = ""

self.datetime = "20250101T000000Z"

self.request = {"headers": ["x-content-sha256","x-date"]}

def authorization(self):

cred_string = self.create_scope()

parts = [

f"HMAC-SHA256 Credential={self.ak}/{cred_string}",

f"SignedHeaders={self.signed_headers()}",

f"Signature={self.signature()}"

]

token = ', '.join(parts)

tokenBytes = "{}".format(token).encode('utf-8')

tokenStr = base64.b64encode(tokenBytes).decode('utf-8')

authorizationStr = base64.b64encode((self.ak + ":" + tokenStr).encode('utf-8')).decode('utf-8')

return authorizationStr

def signature(self):

signing_key = self.get_signing_key()

sign_str = self.get_sign_str()

return Util.hmac(signing_key,sign_str).hex()

def get_signing_key(self):

k_date = Util.hmac(f"{self.sk}".encode('utf-8'),self.datetime[:8])

region = self.region

k_region = Util.hmac(k_date,region)

k_service = Util.hmac(k_region,self.service_name)

sign_key = Util.hmac(k_service,self.v4Identifier)

return sign_key

def get_sign_str(self):

parts = [

self.algorithm,

self.datetime,

self.create_scope(),

Util.sha256(self.get_canonical_string()).hex()

]

return '\n'.join(parts)

def get_canonical_string(self):

parts = [

self.method,

self.path,

self.query,

"x-content-sha256:"+Util.sha256(self.body).hex(),

"x-date:" + self.datetime,

"",

self.request["headers"][0]+";"+self.request["headers"][1],

Util.sha256(self.body).hex(),

]

return '\n'.join(parts)

def signed_headers(self):

headers = sorted(k.lower() for k in self.request['headers'])

return ';'.join(headers)

def create_scope(self):

return '/'.join( [self.datetime[0:8], self.region, self.service_name, self.v4Identifier])

if __name__ == '__main__':

t = tlsSigners("your_ak","your_sk")

print("Authorization: Basic "+t.authorization())

ES-compatible API endpoints

Elasticsearch-compatible API for TLS supports the following standard Elasticsearch APIs.

Request method	ES API path	Description
Post	/${index}/_search	Query a specified Index. The Index corresponds to the TLS project and topic, in the format: ${project}.${topic_id}.
Post	/${index}/_async_search	Batch search.
Post	/_msearch	Batch search.
Post	/${index}/_bulk	Write data to the specified Index. Updating data is not supported; only data writing is allowed.
Get	/${index}/_mapping	Retrieve the mapping for a specified index.

ES-compatible DSL syntax

Elasticsearch-compatible API for TLS supports the following standard Elasticsearch DSL.

DLS categories	DSL syntax compatible with TLS
Query types	bool: Boolean operations match_all: Match all logs match: Index match match_phrase: Phrase match (translated as SQL LIKE syntax in TLS, returns fields configured with indexes) multi_match: Multi-field match Prefix: prefix query Range: range query Exists: field existence query Term: keyword query (translated into TLS index query syntax) Terms: multiple keyword queries (translated into TLS index query syntax) Wildcard: wildcard match (translated into fuzzy match query, usage restrictions are consistent with TLS fuzzy query) Exists: field existence query (translated as key:* query)
Aggregation	Group statistics: Date histogram: generate histogram statistics based on @timestamp field. Histogram: generate statistics based on value intervals Range: group and analyze data based on value ranges Terms: group and analyze data based on unique field values Filter: group and analyze data based on a single filter condition Filters: group and analyze data based on multiple filter conditions Metric statistics: Min: calculate the minimum value Max: calculate the maximum value Avg: calculate the average value Sum: calculate the total sum Count: Total statistics Return logs: Top hits: Returns grouped logs

Differences between ES-compatible API and ES-standard API

Differences between ES-compatible API for TLS and ES-standard API are as follows:

Differences	Standard Elasticsearch API	Elasticsearch-compatible API for TLS
Update data	Supported	Not supported
Query statements containing Script	Supported	Not supported
Automatically create index	Supported	Not supported It can be created manually via API or in the TLS console.
PB-level data volume Topic	Not supported	Supported No need for index rolling.
Field mapping	@timestamp	__time__
Field mapping		_id	The _id field already exists in the TLS data: Yes: use _id. No: use uuid or _pack_meta.

Torch Log Service