Docs

GPU Compute Service

Best Practices

Using second-level monitoring feature

This topic describes how to configure and use the second-level monitoring feature provided by VolcanoEngine.

tip

This feature is in Beta. If you need a trial, please contact your account manager.

Scenario

In Large Language Model (LLM) training scenarios, a hybrid approach integrating Tensor Parallelism (TP), Pipeline Parallelism (PP), and Data Parallelism (DP) is widely adopted to minimize communication overhead. TP primarily involves intra-instance computation and communication, with relatively fixed implementations provided by NVIDIA. PP is a high-frequency activity (operating at 40 ms intervals) with millisecond-level single-execution latency. DP is a low-frequency activity (operating at 5s intervals) with 100 ms-level single-execution latency.

To help you resolve communication-related issues more precisely and efficiently, VolcanoEngine offers a second-level monitoring feature.

Prerequisites

Create a Volcengine Managed Service for Prometheus (VMP) in the same region as your cluster. For more information, see Volcengine Managed Service for Prometheus.

Select an appropriate workspace type based on your actual needs.

Execute the following command to check the version of the Cloud Monitor plugin and make sure that a version no lower than 3.16.0 is installed on the instance:

LD_LIBRARY_PATH=/usr/local/cloud-monitor-agent/lib /usr/local/cloud-monitor-agent/cloud-monitor-agent version --config-file /usr/local/cloud-monitor-agent/config.yaml

tip

If the plugin is not installed or the version is too low, reinstall it. For more information, see Installing plugin.

If your instance cluster is large in scale, use the cloud assistant feature to quickly install the plugin. For more information, see Batch installing Cloud Monitor plugin.

Make sure that Grafana is installed. For more information, see Install Grafana.

Procedure

Step 1: Enable the second-level monitoring feature

Cloud assistant-based configuration

When you enable the second-level monitoring feature on large-scale instances, you can use the cloud assistant feature to automatically complete the monitoring configuration in a batch.

tip

To enable the second-level monitoring feature for non-RDMA metrics, refer to the "Manual configuration" section below.

In the top menu bar, select the region where your business is located.

In the left-side navigation pane, select "Maintenance & Monitoring > Cloud Assistant" to enter the job management page.

Select the "Custom commands" tab and click the "Create a custom command" button.

On the "Create a custom command" page, configure the following information to create a custom command.

This topic only describes the differentiated parameters. For the complete parameter description, see Cloning to custom commands.

Argument	Description	Sample value
Command type	Select "Python" for "Command type". VolcanoEngine provides a Python script to help you complete the configuration quickly.	Python
Command contents	Copy and paste the following content. #!/usr/bin/python # -- coding: utf-8 -- import json import subprocess file_path = "/usr/local/cloud-monitor-agent/conf_plugin.json" # Specify the file path # Enter the new value. enable={{enable}} infoList='''{{user_password_url_list}}''' flag = False if enable == 1: flag = True # Add a backslash before each double quote. infoList = infoList.replace('"', '\"') # Read the file content and process it. with open(file_path, "r") as file: data = json.load(file) # Modify the value. data["PrometheusRdmaCounterCollectorEnabled"] = flag data["PrometheusUserPasswordUrlList"] = infoList # Save the modified content to the file. with open(file_path, "w") as file: json.dump(data, file, indent=4) subprocess.call(["systemctl", "daemon-reload"]) subprocess.call(["systemctl", "restart", "cloud-monitor-agent.service"])	-
Use argument	Select this option to use custom parameters. When this option is selected, custom parameters in the command content will be displayed automatically.	-

Example:

If your instance does not have the cloud assistant Agent installed, perform the following steps to install it.

Batch installation in the console

Click the "Agent management" tab, select the instances that require second-level monitoring, and install the cloud assistant Agent.

tip

You can batch install the cloud assistant Agent on multiple instances in the console. After successful installation, you need to restart the instances for the configuration to take effect.

Install cloud assistant Agent by executing the scripts manually

You can install the cloud assistant Agent in an instance by using the script provided by VolcanoEngine.

tip

After script-based installation is complete, you do not need to restart the instance.

Installation through intranet

Execute the following command in the instance to install the cloud assistant Agent through the VolcanoEngine private network.

# Beijing Region
sudo bash -c "$(curl http://assist-client-beijing.tos-cn-beijing.ivolces.com/install.sh)"

# Shanghai Region
sudo bash -c "$(curl http://assist-client-shanghai.tos-cn-shanghai.ivolces.com/install.sh)"

# Guangzhou Region
sudo bash -c "$(curl http://assist-client-guangzhou.tos-cn-guangzhou.ivolces.com/install.sh)"

# Beijing Region
sudo bash -c "$(curl http://assist-client-beijing.tos-s3-cn-beijing.ivolces.com/install.sh)"

# Shanghai Region
sudo bash -c "$(curl http://assist-client-shanghai.tos-s3-cn-shanghai.ivolces.com/install.sh)"

# Guangzhou Region
sudo bash -c "$(curl http://assist-client-guangzhou.tos-s3-cn-guangzhou.ivolces.com/install.sh)"

Installation through internet

If an EIP is associated with your ECS instance, you can execute the following command to install the cloud assistant Agent through the Internet.

sudo bash -c "$(curl http://assist-client.tos-cn-beijing.volces.com/install.sh)"

sudo bash -c "$(curl http://assist-client.tos-s3-cn-beijing.volces.com/install.sh)"

Create a cloud assistant task to configure the second-level monitoring feature on the target instance.

Select the "Job management" tab and click the "Create a job" button.

On the "Create a job" page, fill in the information in the following table.

This topic only describes the differentiated parameters. For the complete parameter description, see Creating jobs.

Parameter	Description	Sample value
Job name	Set the job name.	Configure the second-level monitoring feature
Command type	Select "Custom commands".	Custom commands
Command	Select the custom command you created in Step 4.	Configure second-level monitoring
Parameter	The parameters for second-level monitoring. enable: Required. Whether to enable the second-level monitoring feature for RDMA metrics. Valid values: 0: Disable. 1: Enable. user_password_url_list: Required. Format: [{"user":"xxx","passwd":"xxx","url":"http://write.prometheus-cn-beijing.ivolces.com/workspaces/xxx"}]. Where: user: Enter the VMP username. passwd: Enter the VMP login password for the corresponding user. url: Enter the Remote Write URL. For more information, see Obtaining Remote Write and Query addresses.	-
Execute object	Select the instance for which you want to configure second-level monitoring based on information such as the instance name, project, or tag.	-

Click the "Confirm" button to complete the job creation.

After the job is created, the system automatically redirects you to the Job management page. The new job then executes automatically based on its execution method.

You can view the execution status and results of the job on each instance by following the instructions in Viewing job execution results.

Manual configuration

If you only need to use the second-level monitoring feature on a few instances or need to enable second-level monitoring for non-RDMA metrics, you can manually configure the second-level monitoring feature according to the instructions in this section.

Execute the following command to open the /usr/local/cloud-monitor-agent/conf_plugin.json configuration file.

vi /usr/local/cloud-monitor-agent/conf_plugin.json

Press the i key to enter edit mode and modify the following parameters.

Parameter descriptions:

Parameter	Description	Sample value
PrometheusRdmaCounterCollectorEnabled	Whether to enable second-level monitoring for RDMA metrics. Valid values: true: Enable false: Disable.	true
PrometheusNormalCollectorEnabled	Whether to enable second-level monitoring for non-RDMA metrics. Valid values: true: Enable false: Disable tip You cannot configure the collection interval. The default collection interval is 30s.	true
PrometheusUserPasswordUrlList	The Basic Auth settings for the VMP workspace. Multiple workspaces are supported. Format: [{"user":"xxx","passwd":"xxx","url":"http://write.prometheus-cn-beijing.ivolces.com/workspaces/xxx"}]. user: Enter the VMP username. passwd: Enter the VMP login password for the corresponding user. url: Enter the Remote Write URL. For more information, see Obtaining Remote Write and Query addresses.	-

Execute the following command to restart the Cloud Monitor service.

systemctl restart cloud-monitor-agent.service

Step 2: Configure Grafana

Click the "Toggle menu" button in the upper left corner, and then select "Connections > Data Sources" from the directory tree on the left.

Click the "Add data source" button, and search for and add Prometheus.

On the "Settings" tab, configure the VMP data source.

Parameter descriptions:

Parameter	Description
Prometheus server URL	Enter the query URL. For more information, see Obtaining Remote Write and Query addresses.
Basic auth	Enable this feature to use VMP authentication information.
User	Enter the VMP username.
Password	Enter the VMP login password for the corresponding user.

After the configuration is complete, click the "Save & test" button at the bottom of the page.

Select "Dashboards" from the directory tree on the left, and choose "Import" from the "New" drop-down list on the right.

Select the VolcanoEngine second-level monitoring JSON file, and click the "import" button to import the file.

tip

If you do not have the VolcanoEngine second-level monitoring JSON file, contact your account manager to obtain it.

After the file is imported, you can view VolcanoEngine instance second-level monitoring data directly in Grafana.

Last updated: 2025.12.03 15:45:51