Docs

Volcengine Managed Service for Prometheus

AI add-on integration

Ray integration

Volcengine Managed Service for Prometheus (VMP) allows you to monitor Ray instances that are deployed in Volcengine Kubernetes Engine (VKE) clusters.

Background

Ray is a distributed computing framework developed by RISELab at the University of California, Berkeley. It provides a parallel-processing computing layer for machine learning and big data programs, reducing the difficulty of developing large-scale end-to-end machine learning workflows.

You can use VMP to monitor the metrics of Ray instances deployed in VKE clusters. The metrics include the usage of cluster node resources (such as CPU, GPU, and memory), task execution latency and success rate, actor status, distributed storage performance, and other metrics. This way, you can improve the running efficiency and reliability of tasks such as distributed training and reinforcement learning, and achieve key metric alerting and optimization.

Prerequisites

A VMP workspace has been created. For details, see Creating workspaces.

VKE has been activated.

A cluster has been created. For details, see Creating clusters.

A Ray instance is deployed in the cluster.

Cloud native observability has been enabled for the cluster, and prometheus-agent has been installed in the cluster. For details, see Enabling observability.

tip

The VMP workspace and the VKE cluster must be deployed in the same region.

Procedure

Configure an integration task

In the top navigation bar, select the target region.

In the left navigation bar, click Integration Center > Component. On the page that appears, find and click Ray to configure the integration task.

Configure the deployment environment of the collector. Click VKE environment and select an associated cluster.

Click Create Integration Task to specify the details of the integration task.

Parameter	Description
Basic information
Task name	Specify the name of the integration task.
Acquisition configuration
Metric scraping interval	Configure the collection interval of the collector.
JobLabel	You can select labels from Kubernetes Pod objects, and the label value will be used as the job label for all metrics. For example, if you set the JobLabel parameter to foo and the Pod label is foo: bar, the system will add a job="bar" label to all collected metrics. If this parameter is left empty, the default job label for the metrics consists of the namespace and name of the PodMonitor object, which is in the format <namespace>/<name>.
Namespace	Specify the namespace where the collection target is deployed in the cluster. Values: arbitrary: indicates that metrics of resources in any namespace in the cluster can be collected. custom: indicates that you can specify custom namespaces in the cluster, and only metrics of resources in the specified namespaces can be collected. For example, if you set the value to default, only metrics of resources in the default namespace can be collected.
LabelSelector	Select collection targets by using the labels of Kubernetes Pod objects.

Click Confirm to complete the configuration.

Obtain monitoring dashboards

Click the Effect preview tab to view sample monitoring dashboards. Click Get the Grafana template to download the Grafana monitoring dashboard template in JSON format.

Query metrics

Click the Acquisition metric tab to view the list of metrics for the collection targets.

View preset alerts

Click the Alert template tab to view the preset alert templates for Ray. Click Turn on alert. You are redirected to the alert group creation page, where you can swiftly create and enable alerts based on the current alert template.

Validate results

After the configuration is complete, you can view the status and details of the integration task to check whether the task is running as expected.

In the left navigation bar, click Integration Center > Integration task.

Click VKE environment. You can view the status of the integration task in the task list.

After the integration task is created, you can perform the following O&M operations:

Click the name of the task to go to the task details page.

Click Edit in the Actions column to edit the task.

Click Suspend task in the Actions column to suspend the task.

Click Delete in the Actions column to delete the integration task.

Monitoring information

Query metrics

You can quickly query and display metric data by using the Explore feature of VMP. For more details, refer to Querying metrics.

Create dashboards

You can install Grafana in a VKE cluster or an ECS instance and then use Grafana to view metrics and create dashboards.

Deploying Grafana in a VKE cluster and accessing workspaces

Deploying Grafana on an ECS instance and integrating Grafana with a workspace

Configure alerts

You can configure alerts in the alerting center of VMP. For more details, see Creating alert rules.

Last updated: 2026.10.10 10:16:50