When collecting container text logs or container standard output using DaemonSet, you can create collection configurations using LogCollector CRD method. This topic demonstrates how to create log collection configurations using the DaemonSet - CRD method.

Background information

TLS supports collecting container logs via DaemonSet, and creating collection configurations using the LogCollector CRD method.

When collecting container logs using DaemonSet, you can collect both container standard output and text logs. This method offers lower resource usage and flexible configuration.

Creating collection configurations via LogCollector CRD is simple and convenient, but requires certain tool proficiency. Therefore, it is suitable for advanced users familiar with Kubernetes and CRD. For details about the collection principles of the CRD method, see LogCollector CRD collection mechanism.

In addition, when collecting container logs, LogCollector supports various collection configurations, such as single-line and multi-line. Aside from the reserved fields that are added by default in various configurations, TLS also adds the following reserved fields to logs collected from containers by LogCollector and automatically creates indexes for these fields.

Reserved fields for container text logs:

Reserved field	Description
__image_name__	Image name.
__container_name__	Container name.
__container_ip__	IP address of the container or Pod.
__pod_name__	Pod name.
__pod_uid__	Unique identifier of the Pod.
__namespace__	The Namespace to which the Pod belongs.

Reserved field for container standard output:

Reserved field	Description
__container_source__	Data source type: either stdout or stderr.
__image_name__	Image name.
__container_name__	Container name.
__container_ip__	IP address of the container or Pod.
__pod_name__	Pod name.
__pod_uid__	Unique identifier of the Pod.
__namespace__	The Namespace to which the Pod belongs.

Prerequisites

LogCollector has been installed in the Kubernetes cluster. For detailed steps, see Install LogCollector (DaemonSet deployment).

LogCollector CRD is installed in the Kubernetes cluster. For detailed steps, see Install LogCollector CRD.

Limits

When collecting container standard output:

Docker container engine only supports JSON type log drivers.

If a container matches multiple collection configurations, only the latest configuration takes effect on that container.

When collecting log files inside containers:

Currently, log file collection paths inside containers support only three methods: No mount, emptyDir mount, and hostPath mount.

LogCollector does not support setting the collection path to a symbolic link, as LogCollector cannot access symbolic links to log files inside containers. Therefore, configure the collection path using the actual path of the log files in the container.

After the container stops, LogCollector receives the stop event for that container and immediately stops collecting its logs. If delay exists when LogCollector collects logs from the container, some logs generated before the container stops may not be collected.

LogCollector supports container engines including Docker and Containerd. Restrictions for accessing different types of container engines are as follows:

Docker: LogCollector accesses the Docker container engine on the host via /run/docker.sock. Ensure that this path exists and LogCollector has access permissions.

Containerd: LogCollector accesses the Containerd container engine on the host via /run/containerd/containerd.sock. Ensure that this path exists and LogCollector has access permissions.

Create LogCollector collection configuration

LogCollector CRD defines and delivers LogCollector collection configurations using yaml files. After you create and apply the file, TLS automatically creates a new LogCollector CR based on the configuration. This CR corresponds to a LogCollector collection configuration and shows in the TLS console.

Follow these steps to create a LogCollector collection configuration:

Log in to a Kubernetes cluster where LogCollector CRD is installed.

Create a YAML file to specify the collection configuration.

You can define the YAML file name based on your business needs. In this topic, access.yaml is used as an example.

vim access.yaml

Edit the YAML file and enter the following content.

apiVersion: logging.vke.volcengine.com/v1alpha1 # Keep the default settings; no modification required.
kind: CollectRule # Keep the default settings; no modification required.
metadata:
name: collectrule-sample # LogCollector CR name. It is unique within the current Kubernetes cluster.
spec:
projectName: project-name-sample # Project name. If no project name exists, TLS will automatically create one.
topicName: topic-name-sample # Log topic name. If no topic name exists, TLS will automatically create one.
lifeCycle: 30 # (Optional) Log storage period; effective only when creating a new log topic.
shardCount: 2 # (Optional) Number of shards; effective only when creating a new log topic.
hostGroupNames: # List of host groups bound to the collection configuration.
- host-group-name-sample1 # Host group name 1
- host-group-name-sample2 # Host group name 2
rule: # Detailed definition of the LogCollector collection configuration; for details, see configuration examples in this document.
RuleName: test
Paths:
- "/data/nginx/log/*/*/*.log"
......

You need to set the parameters in the metadata and spec sections based on your business requirements. Parameter descriptions are as follows:

Parameters	Data type	Required	Description
name	String	Yes	LogCollector CR name, unique within the current Kubernetes cluster. It is recommended to set this value to the RuleName specified in the rule parameter for easier identification.
projectName	String	Yes	Project name. If the specified project does not exist, it will be created automatically.
topicName	String	Yes	Name of the log topic. If no specified topic exists, TLS will automatically create one.
hostGroupNames	Array of String	Yes	Name list of host groups bound to the collection configuration. If the specified host group does not exist, a host group of host identification type will be created automatically. The corresponding host identification is set as the host group name.
rule	Object	Yes	Detailed definitions of LogCollector collection configurations. For supported parameters and explanations, see CreateRule. TopicID does not need to be configured. The topic bound to the collection configuration is determined by the topicName parameter. For rule configuration in common scenarios, see the Configuration example section of this documentation. If the specified collection configuration name (RuleName) does not exist, TLS will automatically create the configuration. If it exists, the configuration will be updated. The specified collection configuration will be automatically bound to the specified host group.
lifeCycle	Integer	No	When creating a log topic, specify the retention period of logs in this topic. Logs that exceed the retention period will be automatically deleted. The unit is days. The default value is 30 days. The value ranges from 1 to 3650. If it is set to 3650, the storage time becomes permanent storage.
shardCount	Integer	No	When creating a log topic, specify the number of shards. You can create a minimum of 1 and a maximum of 10 shards, with 1 shard created by default. Each shard provides a write capacity of 5 MiB/s and 500 times/s, and a read capacity of 20 MiB/s and 100 times/s. Plan your shards appropriately when creating a log topic. After creation, the number of shards can not be changed.
autoSplit	Boolean	No	When creating a log topic, specify whether to enable the shard auto split feature. This feature is enabled by default. True: If the amount of data written exceeds the existing shard capacity for 5 consecutive minutes, TLS will automatically split shards based on data volume to meet business needs. However, the number of shards after splitting can not exceed the maximum split count. New partitions created within the last 15 minutes will not split automatically. false (default): Auto split for shards is disabled.
maxSplitShard	Integer	No	Maximum number of shard splits when creating topic, i.e., the maximum number of shards after splitting. The value ranges from 1 to 50. The default value is 50.
topic	Object	No	It is the specific definition of topic when creating a log topic. For supported parameters and detailed descriptions, see CreateTopic. You do not need to set TopicID or ProjectId. The project name is determined by the projectName parameter. If the log topic specified by the topicName parameter does not exist, TLS will automatically create it. If it already exists, it will not be updated. Notice If you set both lifeCycle/shardCount/autoSplit/maxSplitShard and Ttl/ShardCount/AutoSplit/MaxSplitShard, the configuration of lifeCycle/shardCount/autoSplit/maxSplitShard will take precedence.
index	Object	No	When creating a log topic, for the detailed definition of index configuration, supported parameters, and descriptions, see CreateIndex. TopicID does not need to be configured. The index bound to the topic is determined by the topicName parameter. If the specified log topic does not have an index configuration, TLS automatically creates one. If it already exists, it will not be updated.

Run the following command to apply the access.yaml file configuration.

In the configuration, access.yaml refers to the file name. You can replace it based on your actual needs.

kubectl apply -f access.yaml

Related operations

Modify collection configuration

If you need to adjust the collection configuration, we recommend that you modify it by updating the LogCollector CR. After successful modification, TLS will automatically update the corresponding LogCollector CR and refresh the collection configuration that is displayed in the console.

The recommended modify method is as follows. Here, YAML file name access.yaml is used as an example.

Log in to a Kubernetes cluster where LogCollector CRD is installed.

Run the following command to edit the specified YAML file and modify the collection configuration.

vim access.yaml

Run the following command to apply the file so the new configuration can apply.

kubectl apply -f access.yaml

Delete collection configuration

To delete a LogCollector collection configuration, run the command below to remove the corresponding LogCollector CR resource. Here, logcollector_cr_name refers to the name of the LogCollector CR, such as collectrule-sample. You can replace it based on your actual situation.

View collection configuration information

After you successfully create and apply the LogCollector collection configuration, you can view its details using the console or CRD method.

Console method: In the TLS console, check the corresponding collection configuration by project and configuration name. For the operating procedure, see View Acquisition Configuration Information .

CRD method: View the LogCollector CR list in the Kubernetes cluster and check information for the specified CR by its name. How to do:

Log in to the Kubernetes cluster.

Run the following command to view the list of LogCollector CRs in your Kubernetes cluster. From the list, obtain the name of the CR you want to review.

kubectl get collectrule

The following is an example of echo message:

Run the following command to view the detailed definition of the specified LogCollector CR in the spec field of the echo message. Here, logcollector_cr_name refers to the name of the LogCollector CR, such as collectrule-sample. You can replace it based on your actual situation.

kubectl get collectrule logcollector_cr_name -o yaml

View LogCollector CR status

After you create a LogCollector collection configuration using CRD, if the console does not display the corresponding configuration, or if changes to a CR are not synchronized, the configuration creation or update has failed. You can check the status and description of the LogCollector CR in the Kubernetes cluster to figure out the cause of the failure.

Run the following command to check the status of the specified LogCollector CR. In this case, logcollector_cr_name is the name of the LogCollector CR, such as collectrule1. You can replace it as needed.

The status field in the output indicates the status of the LogCollector CR. Note:

Field			Description
projectId			Project ID.
topicId			Log topic ID.
hostGroupIds			List of host group IDs.
ruleId			Collection rule ID.
status	code	LogCollector CR operation status code. 200: LogCollector configuration is created or modified successfully. Non-200: Failed to create or modify LogCollector configuration.
status		message	If the code is not 200, this field returns the specific error message. You can troubleshoot the issue based on the message field.

Example of echo message:

LogCollector collection configuration is created or updated successfully.

Failed to create or update LogCollector collection configuration:

Configuration examples

The following are sample collection configurations for common scenarios. For parameter descriptions of the metadata and spec sections, see Create LogCollector collection configuration.

Collect container standard output for specified workload

Collect the standard output and standard error of all containers that match the following configuration: workload type is Deployment; the name is test-nginx; the Pod Label has key label-test-key1 with value label-test-key1 or key label-test-key2 with value label-test-key2.

Here is a LogCollector collection configuration example:

Collect container standard output for specified namespace

Collect the standard output and standard error from all containers that match the following configuration: Namespace is name namespace-test-name; Pod name is pod-test-name; container name is container-test-name.

Here is a LogCollector collection configuration example:

Collect container text logs using separator mode

Collect logs from access.log file under /var/logs/app directory in all containers that match the following configuration: Namespace name is namespace-test-name; Pod name is pod-test-name; container name is container-test-name. The collection mode is separator mode.

Here is a LogCollector collection configuration example:

Collect container text logs in JSON mode

Collect logs from access.log file under /var/logs/app directory in all containers that match the following configuration: Namespace name is namespace-test-name; Pod name is pod-test-name; container name is container-test-name. The collection mode is JSON mode.

Here is a LogCollector collection configuration example:

Next steps

After creating the collection configuration, LogCollector will monitor and collect container log files or container standard output according to the specified rules. The log data will be stored in the specified log topic.

Log indexing is disabled by default. To query and analyze collected log data in the console, you need to enable the index feature and configure indexing for the log topic.

Torch Log Service