Torch Log Service
Torch Log Service
Torch Log Service
Document
Product News
Product Overview
Product Billing
User Guide
TLS Agent
AgentLoop
Resource management
Host group
Data collection
LogCollector collection
Introduction to LogCollector
Installation and upgrade
Collect host logs
Collect container logs
LogCollector plugins
Manage LogCollector
general reference
Data import
Cloud product log collection
TOS log collection
DBW log collection
VPN Log collection
VKE log collection
CLB log collection
CloudTrail log collection
CFW log collection
NTA log collection
ALB log collection
APMPlus log collection
veFaaS log collection
VPC log collection
veImageX log collection
TrafficRoute DNS log collection
DDoS log collection
WAF log collection
DCDN log collection
Multi-cloud security platform log collection
APIG log collection
Machine learning task log collection
CR log collection
Volcano Ark application lab log collection
FileNAS log collection
LLM Application Firewall Service log collection
vePFS log collection
Cloud security center log collection
EFS log collection
Data storage
Query and analysis
Configure log indexes
Query log
Analyze logs
Query and analyze logs using third-party tools
Associate external data source for log query
SQL function
SQL syntax
Visualization
Query and analysis charts
Summaries of query and analysis charts
Create dashboard
Use dashboard
Add dashboard filters and variables
Use other systems to analyze logs
Log application
Log applications for product observability
Log applications for large models observability
Coding Agent log app
ModelProvider log app
No-coding Agent log app
Log applications for basic service monitoring
Data processing
Data transformation
Syntax and functions
Global operation function
expression function
Data processing best practices
general reference
Data write processor
Data consumer processor
Data consumption and shipping
Ship logs to TOS
Ship logs to Kafka
Ship logs to Splunk
Alarm
Alarm policy
Alarm notification content
Alarm notification content template
alarm notification channel
Tracing service
Monitor Log Service
API reference
Project management API
Topic management API
Shard management API
Index management API
Log management API
Host group management API
Collection configuration management API
Data import API
Data processor API
Data transformation API
Data backflow API
SQL analysis API
Alarm management API
Consumer group management API
Kafka protocol consumption API
Tag management API
Trace storage API
Account management API
SDK reference
Go SDK
Java SDK
Python SDK
iOS SDK
C++ SDK
CLI reference
Best practices
Data collection
Query and analysis
permission management
Consumption and delivery
Visualization
Alarm
FAQs
Billing and cost FAQ
Data collection FAQ
Data storage FAQ
Index configuration FAQ
Query and analysis FAQ
Scheduled SQL tasks FAQ
Alarm FAQ
- Documentation
Torch Log ServiceUser GuideData collectionLogCollector collectionManage LogCollectorConfigure LogCollector startup parameters
Copy page
Download PDFManage LogCollector
Configure LogCollector startup parameters
Copy pageDownload PDFConfigure LogCollector startup parameters
To prevent LogCollector from excessively consuming server resources and affecting the normal operation of other services, TLS has imposed limits on LogCollector’s collection performance. If you need to increase LogCollector’s collection performance, you can adjust the CPU and memory thresholds. This topic introduces how to configure LogCollector startup parameters.
Scenarios to configure
Modify LogCollector resource settings in the following scenarios.
- A large number of log files need to be collected (such as collecting more than 100 files simultaneously) and it leads to high memory usage.
- Large log data traffic leads to high LogCollector CPU usage.
- There is a collection delay when querying logs.
- Certain LogCollector restrictions can not meet your business requirements.
Recommended parameter values
Based on practical experience, we recommend the following parameter configuration for collecting regular JSON files. The performance of complete regular expression and separator modes is similar to JSON mode, while single-line full text and multi-line full text modes offer twice the performance of JSON mode. The complexity of data and the number of files to be collected affect CPU and memory consumption. Please adjust as needed based on the table below and your actual situation.
- Host environment
- ParametersDefault collection rateCollection rate exceeds 20 MB/sCollection rate exceeds 40 MB/sCollection rate exceeds 60 MB/sParametersDefault collection rateCollection rate exceeds 20 MB/sCollection rate exceeds 40 MB/sCollection rate exceeds 60 MB/smax_procs1234max_mem1GiB2GiB3GiB4GiB
- Container environment
- ParametersDefault collection rateCollection rate exceeds 20 MB/sCollection rate exceeds 40 MB/sCollection rate exceeds 60 MB/sParametersDefault collection rateCollection rate exceeds 20 MB/sCollection rate exceeds 40 MB/sCollection rate exceeds 60 MB/smax_procs1234max_mem1GiB2GiB3GiB4GiBresources.limits.cpu1000m2000m3000m4000mresources.limits.memory1GiB2GiB3GiB4GiB
- LogCollector performance limits are shown in the table below.
Description
Due to differences between the testing and production environments, actual collection performance may vary.
- Collection modePerformance limitCollection modePerformance limitSingle-line full text mode300 MB/sMulti-line full text mode200 MB/sJSON mode150 MB/sSeparator mode120 MB/sSingle-line complete regex mode100 MB/sMulti-line complete regex mode100 MB/s
Set startup parameters
- Modify the configuration file.
- The file path details are shown in the table below:
- LogCollector versionInstallation scenarioFile pathLogCollector versionInstallation scenarioFile pathLogCollector v2.2.0 and later versionsHost scenario/usr/local/logcollector/etc/logcollector.ymlContainer scenarioModify the LogCollector configuration file by updating the configmap.kubectl edit configmap -n kube-syetem logcollector-configLogcollector versions before v2.2.0Host scenario/usr/local/filebeat-7.12.0/etc/filebeat.ymlContainer scenarioModify the LogCollector configuration file by updating the configmap.kubectl edit configmap -n kube-syetem logcollector-config
Description
- Set startup parameters based on your needs.
- Startup parameters example:
- daemonset_mode: trueautodiscover:providers:- type: kubernetescleanup_timeout: 15smax_procs: 2max_mem: 2GiBlog_max_bytes: 512KiB...
Description
The table below lists the commonly used startup parameters you should pay attention to; for those not listed, keep the default configuration.
- Configuration itemTypeDescriptionExampleConfiguration itemTypeDescriptionExamplemax_procsintCPU threshold allowed for LogCollector.
- Value range: from one to the number of CPU cores on this machine.
- Default value: 2.
max_procs is a soft limit, and the actual CPU usage by LogCollector may exceed this value.max_procs: 2max_memstringMemory threshold for LogCollector- Value range: 256 MiB – 8 GiB
- Default value: 2GiB
max_mem is a soft limit, and the actual memory used by LogCollector may exceed this value.max_mem: 2GiBlog_max_bytesstringMaximum length for each log entry- Value range: 16 KiB – 8 MiB
- Default value: 512KiB
log_max_bytes: 512KiBautodiscoverautodiscoverContainer discovery module. For more details, see the table below.autodiscover:providers:- type: kuberbetescleanup_timeout: 15sinput.log.enable_inotifyboolThis configuration determines whether to enable or disable inotify mechanism.- true (default): Enable inotify to monitor logs.
- false: Disable the inotify mechanism. This may result in log files with a survival time of less than 20 seconds being missed for collection.
input.log.enable_inotify:false
- autodiscover
- ParametersTypeDescriptionParametersTypeDescriptionProvidersArray of ObjectSpecific provider for container discovery. Currently, only Kubernetes is supported. For more details, see the table below.
- kubernetes
- ParametersTypeDescriptionExampleParametersTypeDescriptionExamplecleanup_timeoutstringTime delay before ending collection after Pod terminates. The default value is 15s.cleanup_timeout: 15s
Last updated: 2026.05.26 11:12:11
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
