Docs

Volcengine Kubernetes Engine

Storage observation

Container storage observability

Container storage is the base storage layer used by the container runtime in a Kubernetes cluster for writing data, logs, and temporary files. Its usage status directly affects application stability. This topic describes how to configure container storage observability in the VMP Integration Center. This helps you monitor the usage of container file systems and temporary Pod storage in real time.

Background

In a Kubernetes cluster, each container has a writable layer (rootfs) for storing files, logs, and other data generated during runtime by default. When the usage of the writable layer of a container exceeds the limit, the container will be evicted, which will cause business interruption. In addition, Pod-level ephemeral storage covers resources such as logs and temporary data volumes (emptyDir) from all containers within the Pod. If the upper limit is exceeded, the Pod will be evicted.

Metric description

Traditional CPU and memory monitoring cannot cover the preceding storage scenarios. VKE provides container storage observability. By deploying a collection add-on on each node, it periodically obtains usage data for container file systems and Pod ephemeral storage from Kubelet, converts it into standard Prometheus metrics, and then reports it to a VMP workspace. The following table describes the main metrics collected.

Metric name	Metric type	Description
container_fs_usage_bytes	Gauge	The number of bytes used in the container's writable layer. Files, logs, and other data generated by the container runtime are stored in this layer by default. If usage continues to increase, the container may be evicted.
container_fs_limit_bytes	Gauge	The maximum capacity of the container writable layer.
ephemeral_storage_pod_usage_bytes	Gauge	The total usage of Pod-level ephemeral storage, including the sum of the logs for all containers, emptyDir volumes, and other data within the Pod.
ephemeral_storage_pod_limit_bytes	Gauge	The maximum capacity of Pod-level ephemeral storage.

tip

The preceding metrics are collected by the vke-pod-storage-exporter add-on. This add-on is deployed on each node as a DaemonSet and obtains data by calling the local Kubelet API.

Core resources

The following table describes core resources involved in the container storage observability feature.

Resource name	Resource type	Description
vke-pod-storage-exporter	DaemonSet	The metric collection add-on deployed on each node is used to obtain container file system and Pod ephemeral storage usage from Kubelet, and exposes Prometheus-format metric data via the /metrics endpoint.
vke-pod-storage-exporter	ServiceAccount	Used for add-on authentication. It holds the token required to access the Kubelet API.
vke-pod-storage-exporter	ClusterRole / ClusterRoleBinding	Grant the get and list permissions to the add-on on resources to ensure the add-on can call the Kubelet API as expected.
vke-pod-storage-exporter	Service	Provide a unified service endpoint (port 9103) for the collection add-on to enable VMP to pull metrics through service discovery.

Prerequisites

A VKE cluster has been created. For details, see Creating managed clusters.

Cloud-native observability has been enabled. For details, see Enabling observability.

The VKE observability feature has been enabled. For details, see VKE observability.

Procedure

Select the target region in the top navigation bar.

In the left navigation bar, click Integration Center > Component.

In the Storage section, click Container storage metric to configure container storage observability.

Select an environment for deploying the collector. Select VKE environment, and select the target VKE cluster from the drop-down list.

Click Create access task, and configure the task details.

Parameter	Description
Basic information
Task name	Specify a name for the integration task. The name can only contain Chinese characters, letters, digits, and underscores (_). It must be 1 to 40 characters in length.
Project	The project to which the instance belongs. Project management is a resource management method that allows you to manage cloud resources used by different businesses or projects in groups. IAM authorization based on projects helps maintain resource independence and data security, and enables you to view resource bills from the project perspective, which helps you efficiently calculate cloud resource usage costs. For details, see Project management.
Tag	The resource label added to the instance. A resource label consists of a key-value pair. You can use resource labels to classify and aggregate cloud resources from different dimensions. This applies to scenarios such as label-based authorization and resource bill splitting. You can add up to 50 labels. For more details, see Managing labels.
Configuring collection
JobLabel	You can select a label from Kubernetes Pod objects, and the value of the label will be used as the job label for all metrics. For example, if jobLabel is set to foo and the Pod label is foo: bar, the system will add the job="bar" label to all collected metrics. If this field is left empty, the job label of the metric will be set to the namespace and name of the PodMonitor object by default, which is in the format of <namespace>/<name>.
Collection interval	Specify the collection interval of the collector. It is the interval at which the collection add-on requests monitoring data from the node Kubelet, such as 15 seconds. A smaller value indicates higher monitoring accuracy, but will increase the storage and query overhead of the VMP workspaces.
Advanced settings
Metric Relabel configuration	Configure the collected metric relabeling. For details, see Metric relabeling.

Click Confirm to create the integration task.

tip

When you create an integration task, the system will automatically install the collection add-on vke-pod-storage-exporter in the vmp-system namespace of the target VKE cluster.

Validating results

After configuration is complete, you can view the task status and details to check whether the integration task is running as expected.

In the left navigation bar, click Integration Center > Access task.

Click the VKE environment tab. In the task list, you can view the status of the integration task.

Click the task name to go to the details page of the task, where you can view the detailed information, target list, and monitoring dashboard.

Monitoring information

View metrics

You can use the Explore feature of VMP to quickly query and display metric data. For details, see Querying metrics.

The following table lists the common query examples.

Query scenario	PromQL statement
Query container filesystem usage	container_fs_usage_bytes
Query Pod ephemeral storage usage	ephemeral_storage_pod_usage_bytes
Query the top 10 container filesystem usage	topk(10, container_fs_usage_bytes)
Query the top 10 container filesystem utilization	topk(10, container_fs_usage_bytes / container_fs_limit_bytes * 100)
Query Pods whose upper limit is about to be exceeded (ephemeral storage utilization > 80%)	ephemeral_storage_pod_usage_bytes / ephemeral_storage_pod_limit_bytes * 100 > 80

Create dashboards

You can deploy self-managed Grafana in a VKE cluster or on an ECS instance, and use Grafana to view metrics and create dashboards.

Deploying Grafana in a VKE cluster and integrating Grafana with a workspace

Deploying Grafana in an ECS instance and integrating Grafana with a workspace

Configuring alerts

You can configure container storage-related alerts in the Alert Center of VMP. For details, see Creating alert rules.

Last updated: 2026.07.24 10:53:35