Docs

Torch Log Service

general reference

Collect multiple log files or standard output simultaneously

This topic introduces how log files or container standard output match multiple collection rules and be collected.

Background information

By default, a log file or container standard output can only match one collection rule at a time and be collected once. If multiple collection rules match the same log file or container standard output, only the most recently created rule takes effect; the rest rules are ineffective.

For example, if collecting the following files using both collection rule 1 and collection rule 2:

The collection paths for rule 1 are /var/logs/a.log and /var/logs/b.log.

The collection paths for rule 2 are /var/logs/b.log and /var/logs/c.log.

Collection rule 1 and collection rule 2 both match the log file /var/logs/b.log. If collection rule 1 is the most recently created. LogCollector will collect logs from /var/logs/a.log and /var/logs/b.log using collection rule 1, and only collect logs from /var/logs/c.log using collection rule 2.

Starting from LogCollector v2.1.0, LogCollector supports setting additional file identifiers. This allows log files or container standard output to be matched and collected by multiple collection rules.

File identifier

By default, LogCollector uses the log file ID—including file inode, device, and checksum of the first N bytes—as file identifiers. Starting with LogCollector v2.1.0, LogCollector supports adding the following additional file identifiers.

Collection rule ID (RuleID): Use the collection rule ID as an additional file identifier. That is to say, LogCollector uses collection rule ID and log file ID as file identifiers.

Log topic ID + Collection rule name (TopicIDRuleName): Use the log topic ID and collection rule name as additional file identifiers. That is to say, LogCollector will use the log topic ID, collection rule name, and log file ID as file identifiers.

Description

When using collection rule ID, if collection rule is recreated (i.e., previous collection rules are deleted and the identical collection rules are newly created), new collection rule will recollect the matched log files or container standard output since the collection rule ID changed.

When using log topic ID + collection rule name, if the collection rule is rebuilt (i.e., previous collection rules are deleted and the identical collection rules are newly created), the new rule will not recollect the matched log files or container standard output, since neither the log topic ID nor the collection rule name has changed.

Steps

Currently, you can set additional file identifiers through the TLS console or via LogCollector configuration files.

Description

If you add extra file identification settings to an existing collection rule, or add them to the configuration file of a running LogCollector, LogCollector will collect the same data multiple times. In this case, we recommend that you enable incremental collection in the collection rule.

The file identification set in the LogCollector configuration file is effective at a global scope and takes highest priority. That is to say, if you set file identification in the configuration file, then LogCollector always uses the file identification from the configuration file, regardless of whether you set file identification in the TLS console or set a different one in the TLS console compared to the configuration file.

LogCollector configuration file

Log in to the server where LogCollector is located and add the following configuration to the LogCollector configuration file to set an additional identifier. File path:

Logcollector versions prior to v2.2.0: /usr/local/filebeat-7.12.0/etc/filebeat.yml

Logcollector v2.2.0 and later: /usr/local/logcollector/etc/logcollector.yml

To view LogCollector version, see 2. View software version.

TLS console

On the Collection rule page, in the Advanced settings section, enable the Allow multiple file collections option and configure the file identifier.

Last updated: 2026.04.15 18:43:10