Docs

Volcengine Kubernetes Engine

Product Announcement

[Product announcement] VKE add-on optimization - addressing resource reporting exceptions in the Kubernetes Device Plugin framework

Background

In the native Kubernetes Device Plugin framework, kubelet calls the ListAndWatch API of Device Plugin via a long-lived gRPC connection to continuously obtain device resource information on the node (such as GPUs and RDMA NICs). When this gRPC connection is unexpectedly broken due to network issues, process crashes, or other reasons, the kubelet reports the corresponding device resources as 0, making GPUs or RDMA resources on the node unavailable for scheduling.

This issue is a known design flaw in the Kubernetes community's Device Plugin framework. If the ListAndWatch stream is broken, the kubelet does not automatically re-establish the connection or retry. Recovery depends completely on the Device Plugin side proactively re-registering. The community currently does not provide any built-in automatic recovery mechanism for the Device Plugin framework. The long-term direction is to replace Device Plugin with the DRA (Dynamic Resource Allocation) architecture, but a full migration will take considerable time.

Scope of impact

Affected add-ons

The following add-ons are involved in this resource reporting issue:

nvidia-device-plugin

rdma-device-plugin

vpc-cni

mgpu

Issue description

When the ListAndWatch connection between the above add-ons and kubelet is unexpectedly disconnected, the following symptoms occur:

GPU or RDMA resources on the node are reported as 0.

Pods that depend on these resources cannot be scheduled to the node.

Existing workloads are unaffected, but new scheduling requests will fail.

Solution

To address the above community framework flaw, VKE has released new versions for all affected add-ons, adding automatic detection and recovery capabilities for resource reporting exceptions. When a resource reporting anomaly is detected, the add-on will automatically initiate the re-registration process to restore device resource reporting, requiring no manual intervention.

Considerations

During add-on upgrades, already allocated device resources and running workloads are unaffected.

We recommend performing add-on upgrades during off-peak hours.

If you still encounter resource reporting anomalies after the upgrade, submit a ticket to contact us.

Operational recommendations

If you have installed and used any of the above add-ons in your cluster, we recommend upgrading them to the latest versions promptly to benefit from automatic detection and recovery capabilities and other related optimizations.

Add-on name	Minimum version requirements	Upgrade notes
nvidia-device-plugin	v2.2.1 or later	After upgrading to this version, automatic detection and recovery are enabled. cgroupv2 is supported starting with this version. To use cgroupv2 on your nodes, upgrade to v2.2.1 or later.
rdma-device-plugin	v3.6.2-vke.1 or later	After upgrading to this version, automatic detection and recovery are enabled.
vpc-cni	v1.8.11 or later	After upgrading to this version, automatic detection and recovery are enabled.
mgpu	v0.6.5 or later	After upgrading to this version, enable Health check in the mgpu add-on configuration (for configuration details, see mgpu). If enabled, the add-on will automatically trigger re-registration and restore resource reporting upon detection of a resource reporting anomaly.

tip

For detailed steps and instructions on upgrading add-ons, see Upgrading add-ons.