Volcengine Kubernetes Engine
Volcengine Kubernetes Engine
Volcengine Kubernetes Engine
Document
Volcengine Kubernetes Engine
What's New
History feature release notes
Kubernetes version release notes
Containerd runtime release notes
Add-on release notes
Monitoring add-ons
Scheduling add-ons
DNS add-ons
Security add-ons
GPU add-ons
Sandbox add-on
Security announcements
Product Announcement
Historical announcements
Overview
Getting started
User guide
Clusters
Creating clusters
Connecting to clusters
Upgrading clusters
Node pool
Creating node pools
Scaling out/in node pools
Nodes
Authorization management
Image
Namespaces
Networking
Cluster network
Container Network
Service
LoadBalancer
CLB
Ingress
APIG Ingress
ALB Ingress
Customize ALB Ingress forwarding rules
Certificate and security
CLB Ingress
Nginx Ingress
Gateway API
Workloads
Creating workloads
Configure workloads
Pods
Configuration management
Image acceleration
Storage
FSX client
EBS Volumes
TOS Volumes
NAS volumes
FSX access methods
NFS access methods
EFS volumes
vePFS volumes
FSX access methods
vePFS Client access methods
NFS access methods
CloudFS volumes
Local volumes
Autoscale
Node autoscaler
Workload autoscaler
Horizontal Pod Autoscaler (HPA)
Cron Horizontal Pod Autoscaler (CronHPA)
Intelligent Horizontal Pod Autoscaler (IHPA)
Horizontal Pod Autoscaler based on event-driven (KEDA)
Add-on management
Network add-ons
Storage add-ons
Monitoring add-ons
Scheduling add-ons
DNS add-ons
GPU add-ons
Core add-ons
Sandbox add-on
Upgrading add-ons
Heterogeneous computing
NVIDIA GPU
mGPU scheduling
RDMA high-performance computing
Scheduling management
Topology-aware scheduling
QoS-aware scheduling
Observability
Log management
Basic observation
Setup & Configuration
Cluster observation
Storage observation
Other observation
Using metrics
Full stack observation
Alert management
Diagnosis and inspection
Node check and self-healing
Application center
Backup center
Stability
Large-scale cluster management
Best practices
Authorization
Clusters
Nodes and node pools
Networking
Storage
Scheduling
Observability
Solution
Security and compliance
API reference
Cluster management
NodePool management
Node management
Virtual nodes management
Add-on management
Tag management
Access policy management
Instance images management
Cloud native observability
Inspection and diagnosis
Task management
Volcengine Container Instance
Appendix
SDK reference
Development guide
Terraform
FAQs
General FAQs
Cluster FAQ
cluster management
cluster network
cluster upgrade
Node and Node pool FAQs
Authorization FAQs
Workload FAQs
Service and Ingress FAQs
Storage FAQs
Heterogeneous computing FAQs
Observability FAQs
Quota FAQs
Add-on FAQs
Cloud-native AI
ServingKit
LLM applications
ComfyUI application
Deploying the Al inference application through Helm
Deepseek practices
Qwen practices
Kimi practice
Seed practices
Best practices
DeepSeek model deployment
GLM model deployment
MiniMax model deployment
Qwen model deployment
Kimi model deployment
Volcengine Container Instance
What's new
Historical feature release notes
Product Announcement
Overview
Billing
User guide
Integrating with VCI
Virtual node
VKE cluster uses virtual nodes
Self-managed/third-party cluster uses virtual node
Creating an instance
General purpose instance
GPU-accelerated computing instance
VCI Agent Sandbox
Startup acceleration
E2B-based management
Upgrade guide
Networking
Monitoring
Scheduling
Container configuration
VCI Image Cache
Best practices
Security and compliance
FAQs
- Documentation
Volcengine Kubernetes EngineVolcengine Kubernetes EngineUser guideDiagnosis and inspectionCluster inspectionInspection item descriptions and fix suggestions
Copy page
Download PDFCluster inspection
Inspection item descriptions and fix suggestions
Copy pageDownload PDFInspection item descriptions and fix suggestions
The cluster inspection feature of the AIOps suite for VKE is built on rich cluster inspection experience and solutions from Volcano Engine internal teams. This topic describes cluster inspection items, check content, and methods for fixing issues.
tip
In this topic, the reference documentation for modifying the number of workload replicas and ResourceQuota for Deployments is used as an example. The configuration method is the same for other types of workloads.
ResourceQuota
Inspection item | Check content | Fix suggestion | References |
Inspection item | Check content | Fix suggestion | References |
VCI network quota | Check whether the VCI network quota of the cluster is used up. If the quota is used up, VCIs will fail to be created. | None | |
VCI quota check | Check whether the VCI quota of the cluster is used up. If the quota is used up, VCIs will be unable to be created for the cluster. | ||
Cluster CLB resource quota | Check whether the usage of the Internet-facing CLB quota and the internal-facing CLB quota for the cluster is greater than 80%. If the quota is used up, services and Ingresses will fail to be created. | ||
Cluster node quota | Check whether the cluster node quota usage is greater than 80%. If the quota is used up, nodes will be unable to be added to the cluster. | ||
Node pool quota | Check the node pool quota of the cluster. If the quota is used up, node pools will be unable to be added to the cluster. |
Resource utilization
Inspection item | Check content | Fix suggestion | References |
Inspection item | Check content | Fix suggestion | References |
High node CPU utilization | Check whether the node CPU utilization is greater than 80%. High utilization may cause CPU resource preemption, which affects normal business operations. |
| |
High node memory utilization | Check whether the node memory utilization is greater than 80%. High utilization may cause OOM errors, which affect normal business operations. | | |
Insufficient allocatable memory in the cluster | Check whether the allocated memory of the node exceeds 80%. If the allocatable memory is less than the pod request value, the pod cannot be created. | | |
Insufficient allocatable CPU in the cluster | Check whether the allocated CPU of the node exceeds 80%. If the allocatable CPU is less than the pod request value, the pod cannot be created. | | |
Node MaxPod | Check whether the number of new pods that the node can accommodate is less than two. After the upper limit is reached, new pods cannot be scheduled. |
| |
High pod memory utilization | Check whether the workload memory load is greater than 95%. Excessive load may cause Out of memory (OOM) errors, which affect normal business operations. |
| |
High pod CPU utilization | Check whether the CPU load of the workload exceeds 95%. Excessive load may cause CPU resource contention and affect normal business operations. | | |
Node diagnosis
Inspection item | Check content | Fix suggestion | References |
Inspection item | Check content | Fix suggestion | References |
NotReady node | Check whether NotReady nodes exist in the cluster. If the node is in an abnormal status, pods cannot be scheduled to the node. | View the node status. Add or delete nodes when necessary. | |
VCI virtual node status | Check the status of VCI virtual nodes in the cluster. If the node status is abnormal, VCI will fail to be created. | Contact technical support for assistance. | None |
Cluster risk
Inspection item | Check content | Fix suggestion | References |
Inspection item | Check content | Fix suggestion | References |
Node security group inconsistency | Check whether the security groups of all nodes in the cluster are consistent. If the security groups are inconsistent, some nodes may be unable to access the network, which will result in network faults. | Make sure that the security groups configured for all nodes are consistent. | |
CoreDNS add-on status | Check whether the CoreDNS add-on is not in the Running state. If the add-on is in an abnormal status, DNS resolution in the cluster will fail, which will lead to access failures through the service name. | Check the status of the CoreDNS add-on and identify and resolve exception causes. | |
Cluster kubeconfig certificate expiration | Check whether the kubeconfig certificate in the cluster is about to expire or has expired. After the kubeconfig certificate expires, you will not be able to use the certificate to access the cluster. The alert status varies based on different expiration or near-expiration times. Valid values:
| Apply for a new certificate and remove expired certificates. | |
Workload replicas | Check whether the expected and actual number of replicas for the workload are consistent. If the number is inconsistent, high reliability requirements are not met. | Identify workloads with an abnormal number of replicas to resolve exception causes and update the number of replicas. | |
DaemonSets status | Check whether the number of DaemonSets matches the number of nodes. If the expected number of replicas is not reached, related features may not function as expected. | Identify and resolve the cause of workloads with an abnormal number of replicas, and update the number of replicas. | |
Availability of the CR service in the VPC | If the VPC where the cluster is located cannot access CR, the pods that use the specified image will fail to start. | Check whether the VPC where the cluster is located is connected to the CR instance. | |
CoreDNS high availability | Check whether the number of replicas for the CoreDNS add-on is greater than 2, and whether different replicas are deployed on different nodes. If the requirements are not met, CoreDNS does not feature high availability and is at risk of a single point of failure. When a node fails or restarts, CoreDNS will be unable to provide services, which affects normal business operations. | Check the status of CoreDNS replicas. Maintain two or more replicas, and distribute replicas across different nodes. | |
CoreDNS add-on status | Check whether the CoreDNS add-on is not in the Running state. If the add-on is in an abnormal status, DNS resolution in the cluster will fail, which will lead to access failures through the service name. | Check the status of the CoreDNS add-on and identify and resolve exception causes. | |
Node pool configuration | Check the node pool feature of the cluster. If the node pool is missing, it may result in high node load and pod scale-out failure. | Configure the node pool. | |
Number of worker nodes in the cluster | Check whether the number of worker nodes in the cluster is less than 2. A cluster with a single node has a single point of failure risk. | Scale out nodes or create a node pool. | |
Cluster deletion protection | Check whether the deletion protection feature has been enabled for the cluster. If the deletion protection is not enabled, the cluster may be mistakenly deleted by the console or API, which results in business failures. | Enable the deletion protection feature. | |
Load Balancer service and corresponding CLB instance configuration | Check whether the CLB instance configuration in the annotation when you create the Load Balancer service is consistent with that in the CLB console. tip Only the instance billing type and instance type of the CLB instance are checked. The corresponding annotations are volcengine-loadbalancer-billing-type and volcengine-loadbalancer-spec. | Modify the annotation in the CLB console to ensure that the configuration in VKE is consistent with that in the CLB console. | |
Cluster CLB instance status | Check whether the CLB instances in the cluster are in a normal status. If the CLB instances are in an abnormal status, the kube-apiserver service of the cluster will become unavailable. | ||
Audit log enabling status | Check whether the cluster auditing feature has been enabled for the cluster. Enabling the auditing feature facilitates troubleshooting and analysis of daily cluster issues. | Enable the cluster auditing feature. |
Last updated: 2026.07.24 14:51:36
All rights reserved © Beijing Volcano Engine Technology Co., Ltd2024
