After you deploy the VM Agent collector, you can configure alerts related to the collector and the corresponding exporter to monitor their running status. This topic describes the common metrics and alert configurations of the VM Agent collector.

Container resource saturation

Container resource saturation is primarily used to monitor the CPU usage and the memory usage of the VM Agent collector and the exporter. Alerts are triggered if container resources are insufficient.

Generally, a globally unified policy is adopted for monitoring container resource saturation. No additional configuration is required for monitoring the collection add-ons.

Recommended alert configurations

Container CPU monitoring

It is recommended to use the following PromQL statement to configure container CPU monitoring: The following table shows the relationship between alerting thresholds and alert levels.

Alert Level	Trigger Threshold	Duration
P1	> 0.8	1 minute
P2	> 0.6	1 minute

Container memory monitoring

It is recommended to use the following PromQL statement to configure container memory monitoring: The following table shows the relationship between alerting thresholds and alert levels.

Alert Level	Trigger Threshold	Duration
P1	> 0.8	Trigger immediately
P2	> 0.6	Trigger immediately

O&M suggestions

The kube-state-metrics agent and the VM Agent collector support sharding. You can increase the number of shards without the need to adjust the quota when resources are insufficient. For more details, refer to Scaling out agents.

After an alert is triggered, you can increase the requests and limits of container resources for other agents. It is recommended to increase both requests and limits by 100% during each adjustment.

Container operational status

Container operational status is mainly used to monitor the operational status and memory usage of the VM Agent collector and the exporter. Alerts are triggered when containers are in an abnormal status or the memory is insufficient.

Generally, a globally unified policy is adopted. No additional configuration is required for monitoring the collection add-ons.

Recommended alert configurations

Container status monitoring

It is recommended to use the following PromQL statement to configure container status monitoring. The following table shows the relationship between alerting thresholds and alert levels.

Alert Level	Trigger Threshold	Duration
P0	= 1	5 minutes
P1	= 1	Trigger immediately

O&M suggestions

The kube-state-metrics agent and the VM Agent collector support sharding. You can increase the number of shards without the need to adjust the quota when resources are insufficient. For more details, refer to Scaling out agents.

Node resource saturation

Node resource saturation is primarily used to monitor the CPU usage and the memory usage of nodes. Even if container resource usage has not reached the quota, insufficient virtual machine resources may still cause resource-related issues.

Recommended alert configurations

Node CPU monitoring

It is recommended to use the following PromQL statement to configure node CPU monitoring: The following table shows the relationship between alert thresholds and alert levels.

Alert Level	Trigger Threshold	Duration
P1	> 2	1 minute
P2	> 1	1 minute

Node memory monitoring

It is recommended to use the following PromQL statements to configure node memory monitoring: The following table shows the relationship between alerting thresholds and alert levels.

Alert Level	Trigger Threshold	Duration
P1	< 0.2	Trigger immediately
P2	< 0.4	Trigger immediately

O&M suggestions

Add node resources to the cluster.

Run the kubectl delete pod command to delete pods that can be restarted, forcing the pods to be rescheduled to nodes with more available resources.

Write requests from collectors to VMP

The success rate of write requests from the collector to VMP is primarily used to monitor the success rate of data writing to a VMP workspace by the collector. If the collector fails to write data to the VMP workspace, your monitoring data may be lost.

Recommended alert configurations

Request failure rate

It is recommended to use the following PromQL statement to configure the failure rate of data write requests for the collector: The following table shows the relationship between alerting thresholds and alert levels.

Alert Level	Trigger Threshold	Duration
P0	> 0.5	Trigger immediately
P1	> 0.1	Trigger immediately
P2	> 0	Trigger immediately

Whether request metrics exist

We recommend using the following PromQL statement to check whether the collector's request metrics are present. The following table shows the relationship between alerting thresholds and alert levels.

Alert Level	Trigger Threshold	Duration
P0	= 1	Trigger immediately

O&M suggestions

It is recommended to configure this monitoring item after the collector runs as expected. During the installation phase, first check the connectivity between your environment and VMP and check whether the authentication information is valid.

If related alerts are triggered during operation, submit a ticket for assistance.

Collection target status

The collection target status is mainly used to monitor whether the connectivity between the collector and the collection target is healthy, and whether the target itself is in a normal status. If the connection between the collector and the collection target is abnormal, or the collection target status is abnormal, the collector will fail to collect valid data.

Recommended alert configurations

Request success rate

We recommend using the following PromQL statement to monitor the collector's data scrape request status. The following table shows the relationship between alerting thresholds and alert levels.

Alert Level	Trigger Threshold	Duration
P1	= 1	5 minutes
P2	= 1	Trigger immediately

O&M suggestions

Check the network connectivity between the collector and the monitored service. Focus on the security group configurations and whether the collector and the business nodes can ping each other.

Check whether the monitored service is healthy and whether the metrics interface is running as expected.

Volcengine Managed Service for Prometheus