Torch Log Service
Torch Log Service
Torch Log Service
Document
Product News
Product Overview
Product Billing
User Guide
TLS Agent
AgentLoop
Resource management
Host group
Data collection
LogCollector collection
Introduction to LogCollector
Installation and upgrade
Collect host logs
Collect container logs
LogCollector plugins
Manage LogCollector
general reference
Data import
Cloud product log collection
TOS log collection
DBW log collection
VPN Log collection
VKE log collection
CLB log collection
CloudTrail log collection
CFW log collection
NTA log collection
ALB log collection
APMPlus log collection
veFaaS log collection
VPC log collection
veImageX log collection
TrafficRoute DNS log collection
DDoS log collection
WAF log collection
DCDN log collection
Multi-cloud security platform log collection
APIG log collection
Machine learning task log collection
CR log collection
Volcano Ark application lab log collection
FileNAS log collection
LLM Application Firewall Service log collection
vePFS log collection
Cloud security center log collection
EFS log collection
Data storage
Query and analysis
Configure log indexes
Query log
Analyze logs
Query and analyze logs using third-party tools
Associate external data source for log query
SQL function
SQL syntax
Visualization
Query and analysis charts
Summaries of query and analysis charts
Create dashboard
Use dashboard
Add dashboard filters and variables
Use other systems to analyze logs
Log application
Log applications for product observability
Log applications for large models observability
Coding Agent log app
ModelProvider log app
No-coding Agent log app
Log applications for basic service monitoring
Data processing
Data transformation
Syntax and functions
Global operation function
expression function
Data processing best practices
general reference
Data write processor
Data consumer processor
Data consumption and shipping
Ship logs to TOS
Ship logs to Kafka
Ship logs to Splunk
Alarm
Alarm policy
Alarm notification content
Alarm notification content template
alarm notification channel
Tracing service
Monitor Log Service
API reference
Project management API
Topic management API
Shard management API
Index management API
Log management API
Host group management API
Collection configuration management API
Data import API
Data processor API
Data transformation API
Data backflow API
SQL analysis API
Alarm management API
Consumer group management API
Kafka protocol consumption API
Tag management API
Trace storage API
Account management API
SDK reference
Go SDK
Java SDK
Python SDK
iOS SDK
C++ SDK
CLI reference
Best practices
Data collection
Query and analysis
permission management
Consumption and delivery
Visualization
Alarm
FAQs
Billing and cost FAQ
Data collection FAQ
Data storage FAQ
Index configuration FAQ
Query and analysis FAQ
Scheduled SQL tasks FAQ
Alarm FAQ
- Documentation
Torch Log ServiceUser GuideData processingData transformationData transformation performance guide
Copy page
Download PDFData transformation
Data transformation performance guide
Copy pageDownload PDFData transformation performance guide
The service performance of data transformation depends on factors such as data volume and rule complexity. This topic describes various factors that may affect data transformation performance to help troubleshoot issues during the transformation process.
Factors affecting performance
According to the transformation principles, the overall rate of a data transformation task mainly depends on the following factors:
- Target topic write speed: The number of shards in the target log topic determines the data write speed.
- Source topic transformation concurrency: The read and write shard count of the source log topic determines its overall concurrency capability for transformation. Usually, you can plan with 1 MiB/s (before compression) per shard. For example, if the source log topic's write speed is 10 MiB/s, you need to split the number of shards in the source log topic to 10 MiB/1 MiB = 10.
- Rule complexity: The logic and complexity of transformation rules configured by users will affect the data transformation speed. Mainly reflected in the following aspects:
Influencing factors | Description |
Influencing factors | Description |
Log output volume | The output mode of log data after processing can be specified in the processing rules. The size and number of logs output affect the data transformation rate. If the transformation involves data splitting, the more event fields are output, the longer the output content will be. This increases the computing power and network bandwidth consumed by output requests, resulting in a slower data transformation speed. |
Number of output targets | You can specify which target endpoints to ship the transformed data to in transformation rules. The more target endpoints you output to, the slower the data transformation rate. The more target endpoints you output to, the more LogGroups are included in each data packet. Then, the network interaction will increase and processing speed will be slower. |
Processing logic | TLS supports a variety of data transformation function syntax and logic. The complexity of processing logic affects the data transformation rate. The more complex the configured processing logic is, the greater the computational resource consumption in TLS and the slower the processing speed. |
Performance improvement methods
In addition to reducing the complexity of data transformation rules, you can also improve processing speed by modifying log topic configurations.
- Source topic: Increase the number of shards in read/write state to enhance TLS data transformation processing speed.
- Target topic:
- We recommend evaluating the number of shards in the target topic to prevent write limits from affecting the overall transformation rate. The number of shards in the target topic is mainly determined by the data write rate. The maximum write rate per shard in a log topic is 5 MiB/s. You can estimate the required number of shards in the target topic based on the source topic's shard count and processing concurrency. For example, if the source topic has 10 shards, the target topic must have at least 2 shards.
- You can also enable the auto split feature for the target topic. If write limits are frequently triggered, shards will be split automatically. For details about auto split, see Automatic split shards.
Last updated: 2026.03.16 11:00:11
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
