Docs

Elastic Compute Service

Monitoring metrics

Viewing GPU/RDMA monitoring data of instances

For GPU Compute Service (GPU), the Volcano Engine provides unique GPU monitoring and RDMA monitoring features to help you quickly view the graphics card and RDMA network information of your instances.

Prerequisites

Check whether the instance has been installed with the O11yAgent collection plugin based on the ECS monitoring guide. If not, refer to White-screen installation guide or Manual plugin installation guide to complete the installation.

Usage notes

Currently, only GPU supports this feature. For specification details, refer to Heterogeneous Computing.

You can also configure data exception alerts of the metric for GPU and RDMA by following the guidance of Creating alarm policies.

tip

For "Alarm object", select "Elastic Compute > ECS".

For "Subnamespace", select "GPU" or "RDMA".

The frequency for collecting instance GPU/RDMA monitoring metrics is 60s, which means that GPU/RDMA monitoring metrics in an instance are collected every 60s.

Procedure

In the top navigation bar, select the project and region of the target instance.

In the left-side navigation pane, select "Instances & Images > Instances".

Click the name of the target instance to go to the instance details page.

Select the "Monitoring" tab to view the instance data in the "GPU monitor" and "RDMA monitor" tabs.

GPU monitor

On this page, you can view memory usage, GPU usage, GPU temperature, GPU Encoder Usage, GPU decoder usage, GPU memory usage, and other metrics.

tip

Only for instances with A100/A800 graphics cards and the number of graphics cards ≥ 2, you can view the metrics of "NVLink Inbound Total Bandwidth" and "Total bandwidth of NVLink outgoing direction".

RDMA monitor

You can view "RDMA network inflow/outflow rate", "Number of packets sent/received by RDMA network", "RDMA network outgoing direction pause packet number", "Number of RDMA network incoming pause packets", and other metrics on this page.

tip

Only for HPC-optimized instances with the A100/A800 GPU graphics cards, you can view metrics of "Number of packets sent/received by RDMA network", "RDMA network outgoing direction pause packet number", "Number of RDMA network incoming pause packets", and "RDMA network outbound/incoming traffic pause time".

Related operation

You can perform as follows:

You can also create alert policies for GPU and RDMA monitoring to quickly detect instances with exceptions and ensure their normal operations.

Connect HPC-optimized instances to VMP to achieve millisecond-level monitoring of GPU/RDMA metrics. This provides technical assurance for stable operation and performance improvement of HPC workloads across multiple dimensions, including troubleshooting, resource optimization, and cost management. Learn more.

FAQ

How to resolve the issue of RDMA monitoring data being empty when I run container training tasks using the dedicated mode of the RDMA network interface?

Last updated: 2026.06.18 17:49:51