Docs

Torch Log Service

Data collection FAQ

How to troubleshoot host group heartbeat exceptions (container)

When collecting container text logs or standard output using LogCollector, if the LogCollector heartbeat status in the host group is Exception, it may cause log collection failures and other issues. When the LogCollector heartbeat is Exception, you can refer to this document to troubleshoot the issue step by step and restore the LogCollector heartbeat status.

Background information

LogCollector is a log collection client developed in-house by TLS, used for quickly reporting container log data. After installing LogCollector on the container, LogCollector will periodically send heartbeat packets to the server. If LogCollector shows no heartbeat on the host group configuration page in the TLS console, it indicates a connection failure between the client and server.

Issue symptoms

On the host group details page in the TLS console, if the status bar corresponding to certain IP addresses in the heartbeat status information area displays Exception, it means that the LogCollector heartbeat status for this container is Exception, and the LogCollector client on the container failed to connect to the server.

When the heartbeat is Exception, the LogCollector client on the container cannot properly report the collected log data to the server, resulting in logs not being collected into the log topic, and the console cannot query and retrieve new log data in real time.

Notice

LogCollector reads the IP address of the first network interface card of the Pod it resides in and reports it as the Pod's IP address to the host group via heartbeat.

When Daemonset is rolling upgraded or restarted, existing Pods will be deleted, and existing LogCollector instances will stop running, causing the status bar corresponding to these existing Pod IP addresses to display Exception. After Daemonset is rolling upgraded or restarted, new Pods will be created, and new LogCollector instances will start running. The IP addresses of these new Pods will be reported to the host group via heartbeat.

Troubleshooting plan

This document uses the installation of LogCollector via Daemonset in a Kubernetes cluster as an example for explanation.

1. Check LogCollector status

After logging in to the Kubernetes cluster, run the following command to check the running status of LogCollector.

In the following example, the name of the Daemonset is log-collector, and the namespace is tls-log.

Description of the output information:

If the NAME field in the returned information does not contain a Pod named "log-collector-xxxxx", it indicates that LogCollector has not been installed in this Kubernetes cluster. Please refer to Common solutions for container LogCollector for processing.

If the STATUS field in the returned information contains values other than Running, it indicates that LogCollector has already been installed in the current container, but the LogCollector running status on this Pod is abnormal. Please refer to Common solutions for container LogCollector for processing.

If the returned information shows that all STATUS field values are Running, it indicates that LogCollector has been successfully installed in this Kubernetes cluster and is running normally.

If the LogCollector heartbeat status still shows Exception in the console at this time, please follow the steps below to continue troubleshooting.

2 Check authentication parameter configuration

When installing LogCollector as a Daemonset, you must first create a ConfigMap and add the TLS configuration information in the logcollector_config.yaml file's output.tls field for authentication and connecting to the TLS service. If the parameter configuration is incorrect, it may cause LogCollector heartbeat is abnormal.

Run the following command to open the filebeat.yml file and check the authentication parameters configured when creating the ConfigMap. In the following example, the Daemonset name is log-collector and the namespace is tls-log.

Details about the authentication parameters are as follows.

Parameters	Example value	Description	How to obtain
endpoint	https://tls-cn-beijing.ivolces.com	TLS connection domain, i.e., the service address.	After logging in to the Volcano Engine console, view the connection domain name on the details page of the project.
secret_id	AKLY*********	The Access Key ID of the primary Volcano Engine account.	After logging in to the Volcano Engine console with the primary account, click Key Management in the drop-down list at the top right of the page, and follow the prompts to view and copy the Access Key ID.
secret_key	TUdZ********	The Secret Access Key of the primary Volcano Engine account.	After logging in to the Volcano Engine console with the primary account, click Key Management in the drop-down list at the top right of the page, then follow the instructions on the page to view and copy the Secret Access Key.
region	cn-beijing	The region where the project is located (Region).	You can view the current project's region in the top navigation bar of the TLS console.

Tip

In output.tls, do not configure both the ip and label parameters at the same time, otherwise heartbeat failures may occur.

An example configuration file is as follows.

If the authentication parameter configuration is incorrect, modify it and restart the Daemonset, and then check the host group status. If the LogCollector heartbeat status still shows Exception in the console, please follow the steps below to continue troubleshooting.

3 Check whether LogCollector is the latest version

LogCollector supports host groups starting from version 1.0.1. If the LogCollector version installed in the container is not 1.0.1 or later, LogCollector heartbeat abnormality may occur, and the server cannot recognize this LogCollector. Therefore, when LogCollector heartbeat abnormality occurs, check whether the LogCollector version meets the requirements.

Run the following command to check the LogCollector version. The "Image" field in the returned information displays the image address of the currently installed LogCollector. If it is not version 1.0.1 or above, or is not the latest version, please refer to Upgrade LogCollector (Kubernetes cluster) to upgrade LogCollector to the latest version.

An example of the returned information is as follows.

After upgrading the LogCollector version, if the LogCollector heartbeat status still shows Exception in the console, please follow the steps below to continue troubleshooting.

4 Check the Host identification configuration

Tip

For host groups of the Host identification address type, refer to this step for troubleshooting.

When collecting container text logs or standard output logs using LogCollector, it is recommended to create host groups of the Host identification type.

In host groups of the Host identification address type, the configured Host identification must match the machine label configured in the LogCollector configuration file. Otherwise, the host group will not display the heartbeat status of LogCollector.

Run the following command to view the LogCollector configuration file.

The system returns the following information. The value of label is the machine label configured in LogCollector. If this value does not match the Host identification configured in the host group, modify the host group configuration and check the machine status again later.

5 Check host group IP address configuration

Tip

For host groups of the IP address type, refer to this step for troubleshooting.

By default, LogCollector reads the IP address of the first network interface card of the Pod and records it as the Pod IP address in the agent_info.json file. When creating an IP-type host group in the console, you need to enter the IP address recorded in agent_info.json as the Pod IP address in the host group, thereby adding the LogCollector of the specified Pod to the host group.

If the IP address configured in the host group does not match the IP address in agent_info.json, the console will display Exception for the container LogCollector heartbeat status and will not be able to collect logs.

You can use the following command to view agent_info.json and check whether the IP address in the ip field matches the IP address displayed in the console.

View agent_info.json.

kubectl -n tls-log exec -it log-collector-9ph8j -- bash

cat agent_info.json

In the returned information, the ip field indicates the current Pod IP address obtained by LogCollector.

{
hostname : C02G64T1MD6R ,
ip : 10.*.*.* ,
logcollect_version : 1.0.2 ,
update_time : 2022.01.07 18:38:12
}

View the IP address configured in the console.

On the details page of the host group, you can view the IP addresses of all Pods in the host group. If the IP address configured here does not match the IP address shown in agent_info.json, modify the host group configuration and check the machine status again later.

Last updated: 2026.07.01 15:03:37