Docs

GPU Compute Service

Cluster diagnosis

Creating diagnosis tasks

This topic describes how to diagnose the health status of HPC-optimized instances. This helps you locate related exceptions or performance metrics that do not meet your expectations.

Limits

Currently, this feature is only available for Linux instances in the Running state.

Instances to be diagnosed need to have the cloud assistant Agent installed.

If the target instance is in the diagnosis analysis, wait for the diagnosis to complete before you select the instance for a new diagnosis.

Usage notes

If you are using the cluster diagnosis service for the first time, you need to click the "Create now" button. This grants role-related permissions to your account (CustomRoleForDiagnose and CustomRoleForHpcDiagnose) that have read and write permissions for ECS resources. This allows the HPC cluster to assume the corresponding role to access the cloud resources involved in your diagnosis task.

tip

After you click the "Create now" button, the system automatically completes the role-related authorization.

If only one role (CustomRoleForDiagnose or CustomRoleForHpcDiagnose) is displayed, this indicates that the other role has been granted the corresponding permissions.

Cross-service access is performed by using IAM roles. An IAM role is granted permissions to access specific cloud resources, and the current service can assume this role to access resources of other cloud services. For more information, see IAM.

Procedure

warning

This diagnosis process may affect the businesses that run on the instance and stop your businesses. Before you start the diagnosis, confirm the business status and select an appropriate time. For example, you can diagnose the instance during business idle periods.

Before the diagnosis, related packages and dependencies are automatically installed on the instance. If the instance is based on a custom image, you may fail to install required packages and dependencies and thereby fail to detect some diagnostic items.

In the left-side navigation pane, select "Instances & Images > HPC Cluster".

In the top navigation bar, select the project and region of the target instance.

Click the "Cluster diagnosis" tab.

Click the "Create diagnosis" button and configure the following information.

Parameter	Description	Sample value
Diagnosis selected	Select the item to be diagnosed. Stand-alone diagnosis and cluster diagnosis are supported. tip Both stand-alone diagnosis and cluster diagnosis options are applicable to all HPC-optimized instances. Stand-alone diagnosis Instance-level diagnosis, mainly used for health checks of GPU devices. Diagnosis items include devices within one GPU instance (such as driver configuration, GPU card, and network interface configuration), NCCL Test, intra-machine bandwidth, and other items. For more details, refer to Diagnosis overview. You can select the following methods for diagnosis: GPU device Health Check: The diagnostic items checked are exactly the same as those of the "GPU device health check" option. This option offers a faster diagnostic process, which is applicable for quickly troubleshooting small faults. tip If you select "GPU device health check", the diagnosis is expected to take 5 minutes. In-depth health check of GPU device: The diagnostic items checked are exactly the same as those of the "GPU device Health Check" option. If this option is configured, the diagnosis is slower, with more content covered. This causes a longer diagnosis process and more occupied disk space, which is applicable for troubleshooting complex faults. tip If you select "In-depth health check of GPU device", the diagnosis is expected to take 40 minutes. Cluster diagnosis Cluster-level detection, mainly used to diagnose connectivity and performance in HPC clusters. Diagnosis items include the connectivity of RDMA network interfaces and switches within a cluster, as well as nccl-tests connections and nccl-tests performance between instances in the cluster. For more details, refer to Diagnosis item overview.	Stand-alone diagnosis/GPU device Health Check
HPC cluster	Select the HPC cluster where the instance to be diagnosed is deployed. Learn more about creating HPC clusters.	-
Instance selected	Select the target instance to be diagnosed. tip If the target instance is undergoing diagnosis analysis, wait for the diagnosis to complete before you reselect the instance for another diagnosis.	-
Cloud assistant installation detection	Check if the instance has the cloud assistant Agent installed. If not, we recommend installing the cloud assistant Agent first. tip If cloud assistant is not installed, you can not perform instance diagnosis.	-

After you complete the configuration, click the "Create" button to begin the diagnosis.

Once the diagnosis is complete, you can view the results.

Subsequent steps

If there are exceptions that you are suggested to fix, you can follow the repair suggestion or the diagnosis item repair suggestions. Then click the "Re-diagnosis" button to confirm whether the exceptions have been resolved.

Last updated: 2025.12.03 16:28:04