Docs

Volcengine Kubernetes Engine

Product Announcement

[Product announcement] The XID error isolation feature is disabled for the nvidia-device-plugin add-on by default

Background

The Nvidia Device Plugin natively provides the GPU isolation capability. When certain XID errors occur on a GPU in a node, the add-on automatically isolates the GPU to prevent workloads from being scheduled onto the abnormal GPU. This ensures business security. However, after automatic isolation is performed, the Nvidia Device Plugin does not automatically restore the isolated GPU. You need to manually restart the Nvidia Device Plugin to resume scheduling. Therefore, if you do not promptly handle the isolated GPU, or if the remaining number of GPUs after isolation does not meet the requirements of scheduling tasks, GPU resources may become idle.

tip

For more information about the native GPU isolation capability of Nvidia Device Plugin, see the official documentation. For more methods of handling common XID errors, see Methods of handling common NVIDIA GPU XID errors.

Impacted version

The nvidia-device-plugin add-on of v2.2.0 and later. For the release notes of the add-on, see nvidia-device-plugin.

Change description

Starting from v2.2.0, the nvidia-device-plugin add-on makes the following changes:

By default, the XID health check feature is disabled, and GPUs are no longer automatically isolated based on certain XID errors. To automatically handle XID errors, you can enable GPU self-healing in node check and self-healing. For details, see Configuring node check and self-healing.

The DCGM_CUSTOM_XID_ERRORS_TOTAL_COUNTER and DCGM_CUSTOM_XID_ERRORS_TOTAL_COUNTER metrics are deprecated and replaced with the DCGM_EXP_XID_ERRORS_COUNT metric.

Configuration method

To use the GPU isolation capability based on XID errors, you can manually modify the default configuration of the nvidia-device-plugin add-on.

In the left navigation bar, click Cluster. On the page that appears, click the name of the target cluster to go to the management page of the cluster.

In the left navigation bar of the cluster management page, click Addons. On the page that appears, click the GPU tab.

Move the pointer over the nvidia-device-plugin add-on and click Configuration. Manually unselect the Disable XID health check parameter to enable the feature of automatically isolating abnormal GPUs based on XID errors.

[Recommended] Enable cloud-native observability and configure GPU alert rules to promptly identify and handle idle GPU resources. For details, see AI resource observability.

Last updated: 2026.07.15 15:51:31