Torch Log Service
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
RuleId | String | Yes | fa****** | ID of the collection configuration. |
RuleName | String | No | testname | Unique within a log topic. |
Paths | Array of String | No | / | Collection path list.
|
LogType | String | No | delimiter_log | Collection mode.
|
ExtractRule | No | / | Log extraction rules. If you configure log types other than minimalist_log or json_log for collection, you must also set extraction rules. | |
ExcludePaths | No | / | Collection denylist.
| |
LogSample | String | No | / | Log sample.
|
UserDefineRule | No | / | User-defined collection rules. | |
InputType | Integer | No | 2 | Collection type.
|
ContainerRule | No | / | Container collection rules. | |
Pause | Integer | No | 1 | Pause log collection configuration.
|
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
Keys | Array of String | No | [ "time", "", "level", "msg" ] | List of log field names (Key).
|
Quote | String | No | " | Quotation mark Content enclosed by quotation marks is parsed as a complete field and not split. Valid only when LogType is set to delimiter_log. |
TimeKey | String | No | request_time | Name of the log timestamp field. If you use a specified time field from the log as the log timestamp, you need to provide the TimeKey and TimeFormat fields. TimeKey and TimeFormat must be provided together. |
LogRegex | String | No | [(\d+-\d+-\w+:\d+:\d+,\d+)]\s[(\w+)]\s(.*) | The entire log must match the regular expression.
|
TimeZone | String | No | GMT+08:00 | Time zone; supports host time zone (default) and custom time zone. The custom time zone supports GMT and UTC.
|
Delimiter | String | No | # | Log separator. Effective only when LogType is delimiter_log. |
BeginRegex | String | No | [(\d+-\d+-\w+:\d+:\d+,\d+)]\s[(\w+)]\s(.*) | Regular expression required for the first line of the log.
|
TimeFormat | String | No | %Y-%m-%dT%H:%M:%S,%f | Parsing format of the time field If you use the specified time field in the log as the log timestamp, you need to enter both TimeKey and TimeFormat
|
LogTemplate | No | { "Type": "Nginx", "Format": "format main '$remote_addr - $remote_user [$time_local] \"$request\" $request_time $request_length $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\"';" } | Automatically extract log fields based on the specified log template. | |
UnMatchLogKey | String | No | LogParseFailed | When uploading logs that fail to parse, this is the key name for the logs with parsing errors. UnMatchUpLoadSwitch=true and UnMatchLogKey must appear together. |
FilterKeyRegex | No | / | Filter rule list.
| |
TimeExtractRegex | String | No | [0-9]{0,2}\/[0-9a-zA-Z]+\/[0-9:,]+ | Regular expression for extracting time, used to extract and parse time values from the TimeKey field as the collection time. |
UnMatchUpLoadSwitch | Boolean | No | true | Upload logs that failed to parse. UnMatchUpLoadSwitch=true and UnMatchLogKey must be used together.
|
EnableNanosecond | Boolean | No | true | Enable nanosecond precision time. When enabled, log timestamps are parsed and reported with nanosecond precision.
|
TimeSample | String | No | 2006-01-02T15:04:05Z07:00 | Time sample. Used to verify whether the entered time parsing format is correct. |
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
Type | String | No | Path | Collection path type
|
Value | String | No | /accesslog | Collection path. Must specify an absolute path.
|
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
Fields | JSON Map | No | {"ClusterID":"dabaad5f-7a10-4771-b3ea-d821f73e****"} | Add constant fields to logs. Constant fields must follow these restrictions:
|
Plugin | No | {"processors":[{"json":{"field":"__content__","trim_keys":{"mode":"all","chars":"#"},"trim_values":{"mode":"all","chars":"#"},"allow_overwrite_keys":true,"allow_empty_values":true} | LogCollector plugin configuration. Once plugin configuration is enabled, you can add one or more LogCollector processor plugins to parse logs with complex or varying structures. | |
Advanced | No | { "CloseInactive": 10 } | LogCollector Extended configuration. | |
RawLogKey | String | No | __raw__ | Original log field name. This is only effective when EnableRawLog is set to true. RawLogKey defaults to __raw__, which means the original log data will be encapsulated in the __raw__ field and uploaded to TLS along with the parsed log data. |
TailFiles | Boolean | No | true | LogCollector collection policy specifies whether LogCollector collects incremental logs or full logs. The default value is false, indicating collection of full logs.
|
HostnameKey | String | No | hostname | hostname field name. Required only when EnableHostname is set to true. |
EnableRawLog | Boolean | No | true | Choose whether to upload the original log.
|
ShardHashKey | No | { "HashKey": "3C" } | Rule for routing log shard.
| |
ParsePathRule | No | { "PathSample": "/data/nginx/log/dabaad5f-7a10/tls/app.log", "Regex": "\\/data\\/nginx\\/log\\/(\\w+)-(\\w+)\\/tls\\/app\\.log", "Keys": ["instance-id", "pod-name"] } | Rules for parsing the collection path. After setting the rule, fields in the collection path will be extracted using the regular expression specified in the rule and added as metadata to the log data. tip When collecting container standard output, this parameter is not supported. | |
EnableHostname | Boolean | No | true | You can choose whether to upload the hostname field. By default, it is disabled.
|
HostGroupLabelKey | String | No | host_group_label | Field name used to store host group label information. |
EnableHostGroupLabel | Boolean | No | true | You can select whether to upload the label information of the host group to TLS. By default, it is disabled.
|
TailSizeKb | Integer | No | 1024 | Incremental collection rollback threshold, measured in KiB. When the LogCollector collection strategy is incremental, for the initial collection of log files:
|
IgnoreOlder | Integer | No | 7 | Ignore log files that haven't been updated for a certain period; the unit is hours. |
MultiCollectsType | String | No | RuleID | Allow collecting log files multiple times.
|
Parameter | Category | Required | Example value | Description |
Parameter | Category | Required | Example value | Description |
EnvTag | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | This button determines whether to set environment label as log tag and add it to raw log data. If enabled, TLS will add related container environment variable fields to the logs. If you set multiple key-value pairs, it means you add multiple fields. For example, set the Key to source and the Value to data_source. If the container includes the environment variable source=DC, a new field __tag__data_source__: DC will be added to the log. |
Stream | String | No | all | Collect information.
|
KubernetesRule | No | { "NamespaceNameRegex": ".*test.*", "WorkloadType": "Deployment", "WorkloadNameRegex": ".*test.*", "IncludePodLabelRegex": { "Key1": "Value1", "Key2": "Value2" }, "ExcludePodLabelRegex": { "Key1": "Value1", "Key2": "Value2" }, "PodNameRegex": ".*test.*", "LabelTag": { "Key1": "Value1", "Key2": "Value2" } } | Collection rules for Kubernetes containers. | |
ContainerNameRegex | String | No | .*test.* | Name of container to collect. If the container name is not specified, all containers in the host group will be collected. Supports regex matching. For example, if you set container name to ^(container-test)$, all containers named container-test are collected. |
ExcludeContainerEnvRegex | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | The container environment variable denylist specifies which containers are excluded from data collection. If the denylist is disabled, all containers will be collected. If you enable the container environment variable denylist, Key is required, while Value is optional.
|
IncludeContainerEnvRegex | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | Specify which containers to collect by using the container environment variable allowlist. If the allowlist is not enabled, all containers will be collected. To enable the container environment variable allowlist, enter key-value pairs. Key is required; value is optional.
|
ExcludeContainerLabelRegex | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | The container label denylist is used to specify which containers should not be collected. If the denylist is disabled, all containers are collected. If the container label denylist is enabled, the Key field is required and the Value field is optional.
|
IncludeContainerLabelRegex | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | You can specify which containers to collect using container label allowlists. When the allowlist is not enabled, all containers will be collected. When enabling the container label allowlist, you need to fill in key-value pairs. Key is required; value is optional.
|
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
Type | String | No | Nginx | Type of log template. Supported types:
|
Format | String | No | log_format main '$remote_addr - $remote_user [$time_local] "$request" $request_time $request_length $status $body_bytes_sent "$http_referer" "$http_user_agent"'; | Format of the log template. |
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
Key | String | No | __content__ | Name of the filter field. |
Regex | String | No | .*ERROR.* | Regular expression that the log content of the filter field must match. |
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
processors | Array of JSON Map | No | { "processors":[ { "json":{ "field":"__content__", "trim_keys":{ "mode":"all", "chars":"#" }, "trim_values":{ "mode":"all", "chars":"#" }, "allow_overwrite_keys":true, "allow_empty_values":true } } ] } | LogCollector plugin. For the list of supported plugins and parameter details, see LogCollector Plugin Overview. |
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
CloseEOF | Boolean | No | true | After reaching the end of the log file, release the handle for that log file. Default is false. |
CloseRemoved | Boolean | No | true | Whether to release the file handle after the log file is removed Default is false. |
CloseRenamed | Boolean | No | true | After the log file is renamed, release the handle of the log file. The default value is false. |
CloseTimeout | Integer | No | 1800 | Maximum monitoring duration for log files by LogCollector. Measured in seconds. The default is 0 seconds, which means LogCollector will not restrict the monitoring duration for log files. LogCollector starts timing when monitoring a log file. Once the specified duration is exceeded, the log file handle is released immediately and monitoring ends, regardless of whether the log file has been fully read. |
CloseInactive | Integer | No | 10 | Wait time before releasing the log file handle. Release the handle of the log file if no new logs are written over the specified period. Unit: seconds. Allowed values: 1 to 300 seconds. Default is 60 seconds. |
NoLineTerminatorEOFMaxTime | Integer | No | 5 | Maximum wait time when LogCollector does not read a newline character. Measured in seconds; default is 5s. LogCollector starts timing when it begins reading log file contents. If no newline is read within the specified time, it will send the logs in the buffer. If there is a long interval between log writes, a complete log entry may be split and written in two parts. You can adjust this parameter based on the interval between log writes. |
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
HashKey | String | No | 3C | The HashKey of the log group, used to specify the shard to which the current log group writes. This parameter must be within the range [00000000000000000000000000000000-ffffffffffffffffffffffffffffffff). |
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
Keys | Array of String | No | ["instance-id","pod-name"] | Field name list. TLS parses the path sample into multiple fields based on regular expressions (Regex). Keys specify the name of each field.
|
Regex | String | No | /var/logs/([a-z]*)_any_([a-z]*)/test.log | Regular expression for extracting path fields. It must match the sample collection path; otherwise, extraction fails. |
PathSample | String | No | /var/logs/instanceid_any_podname/test.log | Sample collection path from an actual scenario.
|
Parameter | Type | Required | Example value | Description |
Parameter | Type | Required | Example value | Description |
LabelTag | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | This button determines whether to add Kubernetes Labels as log tags to the raw log data. If enabled, TLS will add Kubernetes Pod Label related fields to the logs. If you set multiple key-value pairs, multiple fields are added. For example, set the Key to source and the Value to data_source. Suppose the Pod includes the Label source=DC, a new field __tag__data_source__: DC will be added to the log. |
PodNameRegex | String | No | .*test.* | Pod name specifies the container to be collected. If no Pod name is specified, all containers will be collected. Pod names support regular expressions. For example, set the Pod name to ^(http.*)$ to collect all containers under Pods that start with "http". |
WorkloadType | String | No | Deployment | Specify the container for collection based on workload type. Only one type can be selected. If no type is specified, containers of all types are collected. Supported workload types:
|
AnnotationTag | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | This button determines whether to set Kubernetes Annotation to log tags and add it into the raw log data. If enabled, TLS will add Kubernetes Pod annotation fields to the logs. If you set multiple key-value pairs, multiple fields are added. If you set the key to "sink" and the value to "data_sink", and the Pod contains the annotation sink=ck, a new field __tag__data_sink__: ck will be added to the log. |
WorkloadNameRegex | String | Not supported | .*test.* | By configuring the workload name, you can specify the container to collect. If no workload name is specified, all containers are collected. Workload names support regular expression matching. For example, if you set the workload name to ^(http.*)$, all containers under workloads starting with http will be collected. |
NamespaceNameRegex | String | No | .*test.* | Name of the Kubernetes namespace to be collected. If no Namespace is specified, all containers will be collected. Namespace names support regular expression matching. For example, setting the namespace name to ^(tcp|udp)$ means collecting all containers in the tcp and udp namespaces. |
ExcludePodLabelRegex | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | Specify containers to exclude from collection using a Pod Label denylist. If disabled, all containers will be collected. If you need to set a Pod Label denylist, the Key field is required, while the Value field is optional.
tip
|
IncludePodLabelRegex | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | The Pod Label allowlist is used to specify the containers to be collected. If the Pod Label allowlist is not enabled, all containers will be collected. If you need to set the Pod label allowlist, the key field is required.
tip
|
EnableAllLabelTag | Boolean | No | true | This button determines whether to add all Kubernetes labels as log tags to the raw log data. When enabled, TLS will add all related Kubernetes Pod label fields to logs. For example, if the Pod contains the label source=DC and destination=CS, the log will include the fields __tag__source__: DC and __tag__destination__: CS. |
ExcludePodAnnotationRegex | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | Pod Annotation denylist specifies containers that are excluded from collection. If you need to set a Pod Annotation denylist, the Key is required and the Value is optional.
By default, Value uses string matching. Containers only match if the value is exactly the same as the value of the Pod annotation. If the value starts with^and ends with$, it uses regular expression matching. For example, set the Key as "app" and set the Value as ^(test1|test2)$, and then the containers in Pods whose annotation contains app:test1 or app:test2 will match. tip
|
IncludePodAnnotationRegex | JSON Map | No | { "Key1": "Value1", "Key2": "Value2" } | Pod annotation allowlist is used to specify the containers to be collected. If you need to set a Pod annotation allowlist, the Key field is required and the Value field is optional.
By default, Value uses string matching. Containers only match if the value is exactly the same as the value of the Pod annotation. If the value starts with^and ends with$, it uses regular expression matching. For example, set the Key as "app" and set the Value as ^(test1|test2)$, and then the containers in Pods whose annotation contains app:test1 or app:test2 will match. tip
|
HTTP status code | Error Code | Error message | Description |
HTTP status code | Error Code | Error message | Description |
400 | InvalidArgument | Invalid argument key %s, value %s, please check argument | Invalid parameter. |
400 | PathQuotaExceed | Path Quota Exceeded | Exceeded the quota for collection paths. |
400 | ExcludePathQuotaExceed | Exclude Path Quota Exceeded | Exceeded collect blacklist quota. |
400 | ExtractRuleKeyQuotaExceed | ExtractRule Key Quota Exceeded | Exceeded the limit for custom fields defined by the extraction rule |
400 | ExtractRuleFilterKeyRegexQuotaExceed | ExtractRule FilterKeyRegex Quota Exceeded | Exceeded the limit for extraction rule's filter rules. |
404 | RuleNotExist | Rule %s does not exist | The collection configuration does not exist. |
409 | RuleAlreadyExist | Rule %s already exist | The collection configuration already exists. |
500 | InternalServerError | We encountered an unexpected server error, please try again later. | Internal server error. |