Docs

Volcengine Managed Service for Prometheus

Resource and application monitoring

Monitoring custom Kubernetes clusters

Multi-cloud cluster monitoring is a common requirement of users. This topic describes how to use Volcengine Managed Service for Prometheus (VMP) to monitor custom Kubernetes clusters.

Background

Challenges in multi-cloud cluster monitoring

The usage of multi-cloud clusters (such as multi-vendor clusters and hybrid cloud clusters consisting of public cloud clusters and self-built clusters) provides the following benefits for enterprises: diversified risks, reduced costs, abundant resources, and flexible selections. However, the cross-vendor and cross-region cluster architecture also leads to the following issues, which are outstanding in multi-cloud monitoring:

Challenges in data aggregation and visualization: Monitoring data is distributed across different geographic locations, which cannot be aggregated and displayed in a centralized manner. As a result, a monitoring dashboard that provides a global view cannot be created, and administrators cannot know the status and performance of all clusters at a glance.

Resource and cost management: To monitor multi-cloud clusters, you must build multiple monitoring systems, which leads to higher resource consumption and monitoring costs.

Complexity of configuration and management: Repeated configuration and management policies may be required to manage multi-cloud clusters, which increases management complexity. In addition, monitoring consistency cannot be ensured caused by monitoring system fragmentation. In this way, potential faults and issues cannot be identified in a timely manner.

Fault diagnosis and response: In a multi-cloud environment, locating and diagnosing issues may be more difficult because cross-cloud resources and services may be involved. Therefore, the processes that consist of monitoring, fault response, and upgrade must be consistent. This ensures that issues are resolved in a timely manner to reduce impacts on businesses.

To ensure the monitoring consistency for multi-cloud clusters, VMP provides multi-cloud cluster monitoring solutions and collection agents. This allows users to monitor cloud services and integrate and monitor third-party clusters and self-managed clusters in a centralized manner.

tip

This topic describes how to quickly configure multi-cloud cluster monitoring by using one of the easiest ways. The related agents adopt the default specifications and are not mounted with a PersistentVolumeClaim (PVC) for persistent storage. This helps users quickly know how to integrate custom clusters with VMP.

Monitoring solution

VMP provides you with two ready-to-use metric collection solutions. VMP allows you to directly install collection agents in standard Kubernetes clusters and integrate the clusters with VMP workspaces to collect and aggregate metrics related to multi-cloud clusters. This achieves the monitoring of multi-cloud clusters in a centralized manner. For more details, see Integrating self-managed Kubernetes clusters.

Prerequisites

A Volcano Engine root user has been created, and VKE has been activated.

A cluster has been created. For details, see Creating clusters.

The ingress-nginx agent is installed in the cluster and internal network load balancing is configured. For more details, refer to Installing an add-on.

A VMP workspace has been created. For details, see Creating workspaces.

Cloud Enterprise Network (CEN) is activated, and the internal network of Volcano Engine and the network where the custom cluster is located are connected. For more details, refer to the documentation of CEN.

A self-built or custom cluster is created.

The Kubernetes version of the custom cluster is v1.20 to v1.24. Otherwise, compatibility issues may occur.

Ensure that the custom cluster can access the public network. If not, refer to How do I download and deploy a collector in an environment without a public network?

A namespace is created. This topic takes volcano-metrics as an example.

warning

Do not install the collection agents of the open source Prometheus or other monitoring systems in a custom cluster. These agents may conflict with the collector agent provided by VMP, leading to agent exceptions.

kubectl can be used to connect to clusters in the local environment (Linux/Mac). The clusters include VKE clusters and custom Kubernetes clusters.

Configuration procedure

Step 1: Configure a workspace proxy in the VKE cluster

VMP workspaces provide Remote Write URLs. After you connect the VPC of your custom cluster to the Volcano Engine network using CEN, you can use a proxy to write large-scale metric data persistently to the VMP workspace.

In the left navigation pane, click Services and Ingresses > Service. Click Create with Yaml to create a service. To access the VMP workspace via a Service of type ExternalName, you need to provide the workspace address. For information about obtaining a workspace address, see Obtaining workspace addresses.

Click Workload > Resource Explorer in the left navigation pane. Click Create with Yaml to create an Ingress. Specify the mapping relationship between the custom domain name and the service.

Select Custom from the Type drop-down menu.

Enter the YAML configuration in the Yamlfield.

After the configuration is complete, click Services and Ingresses > Ingress in the left navigation pane to view the private IP address that corresponds to the Ingress.

Step 2: Configure DNS in the custom cluster

To ensure that the collector deployed in the custom cluster can identify the custom domain name, you need to configure CoreDNS in the custom cluster to specify the mapping relationship between the custom domain name and the internal IP address of the Ingress.

Use kubectl to connect to the custom cluster.

Run the following command to modify the CoreDNS configuration file:

Add custom hosts in the configuration file, such as 192.168.48.3 vmp-test.com.

Step 3: Deploy the collector in the custom cluster

warning

Volcano Engine provides two out-of-the-box collection solutions: VMP Agent and VM Agent. This topic uses VM Agent deployment as an example. For details, see Collector solution overview.

Download the collector deployment script to your local environment. For details, see Obtaining scripts.

Use kubectl to connect to the custom cluster.

Run the following command to deploy the VM Agent collector:

Check results

View agent status

After the agent is installed, you can go to the custom cluster page to view the working status of the collector agent. The following table describes the name, type, and feature of each agent.

Agent Name	Workload Type	Namespace	Description
vmagent	StatefulSet	volcano-metrics	Collects metrics exposed by kubelet, kubelet-cadvisor, kube-state-metrics, and node-exporter in the cluster.
vm-operator	Deployment	volcano-metrics	The controlling agent of VM Agent.
kube-state-metrics	StatefulSet	volcano-metrics	Provides the metric data of Kubernetes resources.
node-exporter	DaemonSet	volcano-metrics	Provides the metric data of node resources.

View ingress-nginx monitoring data

The proxy method is used to write monitoring data to the VMP workspace. The write performance is closely related to the performance of the Ingress instance in the VKE cluster. It is recommended that you use the observability feature of the VKE cluster to monitor and perform O&M on the Ingress instance. You can refer to the following topics:

ingress-nginx monitoring

Nginx Ingress best practices

Monitoring custom Nginx Ingress controllers

Monitoring and alerting

Metric

You can quickly query and display metric data by using the Explore feature of VMP. For more details, refer to Querying metrics.

Dashboard

You can build your own Grafana and view metrics and create dashboards through Grafana. It includes:

Deploying Grafana in a VKE cluster and accessing workspaces

Deploying Grafana on an ECS instance and integrating Grafana with a workspace

Alert

You can configure alerts in the alerting center of VMP. For more details, see Creating alert rules.

FAQ

How do I download and deploy a collector in an environment without a public network?

The collectors provided by VMP are stored in Container Registry (CR) by default. If your custom cluster is deployed in an environment without public networks and cannot access the Internet, you can perform the following steps to download and deploy the collector:

Download the related collector image in an environment where you can access the Internet.

Upload the image to a private cloud registry.

When you deploy the collector, add the fields of the private registry to allow the collector image to be pulled from the private registry.

Last updated: 2026.10.10 11:04:51