Docs

Torch Log Service

general reference

Aggregation rules for multi-line full text logs

This topic describes the aggregation rules and provides relevant examples for collecting multi-line logs using LogCollector in multi-line full text mode.

Aggregation rules

The aggregation rules for LogCollector multi-line full text mode are as follows:

When a log line that is read matches the first line regular expression exactly, then:

If there is an existing multi-line log block, LogCollector will send the current block first, and then create a new multiline log block. Whereafter, LogCollector uses the current line as the beginning of the new block.

If there is no existing multiline log block, LogCollector will create a new block and use the current line as its beginning.

When a log line that is read does not match the first line regular expression exactly, then:

If there is an existing multiline log block, LogCollector will append the current line to the end of the existing block.

If there is no existing multiline log block, LogCollector will send this line directly.

EOF (end-of-file) handling:

If EOF is read for 5 consecutive seconds, LogCollector will send existing multi-line log blocks.

Example of typical scenario

First-line regular expression

xxx.*

User-side raw logs

xxx1
yyy11
yyy12
xxx2
yyy21
yyy22

Logs that are sent

xxx1\nyyy11\nyyy12
xxx2\nyyy21\nyyy22

Log aggregation steps are as follows:

Read log line 1 xxx1. Line 1 fully matches the first-line regular expression xxx.*, so LogCollector creates a new multi-line log block and sets xxx1 as its beginning.

Read log line 2 yyy11. Line 2 does not fully match the first-line regular expression xxx.*, so LogCollector appends log yyy11 to the end of the current multi-line log block.

Read log line 3 yyy12. Line 3 does not fully match the first-line regular expression xxx.*, so LogCollector appends log yyy12 to the end of the current multi-line log block.

LogCollector reads line 4 xxx2. Line 4 fully matches the first-line regular expression xxx.*, so LogCollector sends the current multiline log block xxx1\nyyy11\nyyy12, and then creates a new multiline log block and uses xxx2 as its beginning.

LogCollector reads line 5 yyy21. Line 5 does not fully match the first-line regular expression xxx.*, so LogCollector appends log yyy21 to the end of the current multiline log block.

LogCollector reads line 6 yyy22. Line 6 does not fully match the first-line regular expression xxx.*, so LogCollector appends log yyy22 to the end of the current multiline log block.

EOF is read for 5 consecutive seconds. Therefore, LogCollector sends the current multiline log block xxx2\nyyy21\nyyy22.

Example of special scenario

Single-line and multi-line logs in one file

First-line regular expression

xxx.*

User-side raw logs

yyy11
yyy12
yyy13
xxx2
yyy21
yyy22

Logs that are sent

yyy11
yyy12
yyy13
xxx2\nyyy21\nyyy22

Aggregation rules for multi-line logs are as follows:

Read the first log line yyy11. Line 1 does not fully match the first-line regular expression xxx.*. Since no multiline log block exists, LogCollector sends the log yyy11 directly.

Read the second log line yyy12. Line 2 does not fully match the first-line regular expression xxx.*. Since no multiline log block exists, LogCollector sends the log yyy12 directly.

Read the third log line yyy13. Line 3 does not fully match the first-line regular expression xxx.*. Since no multiline log block exists, LogCollector sends the log yyy13 directly.

Read the fourth log line xxx2. Line 4 fully matches the first-line regular expression xxx.*, so LogCollector creates a new multiline log block and uses log xxx2 as the beginning of this block.

Read the fifth log line yyy21. Line 5 does not fully match the first-line regular expression xxx.*, so LogCollector appends log yyy21 to the end of the current multiline log block.

Read the sixth log line yyy22. Line 6 does not fully match the first-line regular expression xxx.*, so LogCollector will append log yyy22 to the end of the current multiline log block.

EOF is read for 5 consecutive seconds. Therefore, LogCollector sends the current multiline log block xxx2\nyyy21\nyyy22.

The multi-line log size exceeds 512 KB

First-line regular expression

xxx.*

User-side raw logs

xxx1
yyy11
…… // Log lines omitted.
yyy12 // Where, 1 represents the byte in the 512KB place of multi-line log. That is, the characters after 1 will be truncated during collection.
yyy13
yyy14

Logs that are sent

xxx1\nyyy11\n……\nyyy1
2
yyy13
yyy14

Aggregation rules for multi-line logs are as follows:

Read log line 1 xxx1. Line 1 fully matches the first-line regular expression xxx.*, so LogCollector creates a new multi-line log block and sets xxx1 as its beginning.

Read log line 2 yyy11. Line 2 does not fully match the first-line regular expression xxx.*, so LogCollector appends log yyy11 to the end of the current multi-line log block.

The collection process for the log lines omitted in between is the same as above.

Read log line yyy12. The line does not fully match the first-line regular expression xxx.*, so LogCollector appends log yyy12 to the end of the current multi-line log block. At this point, the multi-line log block size exceeds 512KB, so it is truncated starting at position 1 of yyy12. Whereafter, the log block xxx1\nyyy11\n……\nyyy1 is sent, followed by log block 2.

Read the third log line yyy13. This line does not fully match the first-line regular expression xxx.*. Since no multiline log block exists, LogCollector sends the log yyy13 directly.

Read log yyy14. This line does not fully match the first-line regular expression xxx.*. Since no multi-line log block exists currently, LogCollector sends log yyy14 directly.

The flush duration for multi-line logs exceeds 5 seconds

First-line regular expression

xxx.*

User-side raw logs

xxx1
yyy11
// after 5s
yyy12
yyy13

Logs that are sent

xxx1\nyyy11
yyy12
yyy13

Aggregation rules for multi-line logs are as follows:

Read log line xxx1. The line fully matches the first-line regular expression xxx.*, so LogCollector creates a new multi-line log block and sets xxx1 as its beginning.

Read log line yyy11. The line does not fully match the first-line regular expression xxx.*, so LogCollector appends log yyy11 to the end of the current multi-line log block.

EOF is read for 5 consecutive seconds. Therefore, LogCollector sends the current multiline log block xxx1\nyyy11.

Read log yyy12. This line does not fully match the first-line regular expression xxx.*. Since no multi-line log block exists currently, LogCollector sends log yyy12 directly.

Read the third log line yyy13. This line does not fully match the first-line regular expression xxx.*. Since no multiline log block exists, LogCollector sends the log yyy13 directly.

Last updated: 2025.12.12 14:35:05