You need to enable JavaScript to run this app.
Volcengine Kubernetes Engine

Volcengine Kubernetes Engine

Copy page
Download PDF
Other observation
Image acceleration observability
Copy page
Download PDF
Image acceleration observability
VKE supports P2P acceleration and image lazy loading (Nydus) features. These acceleration features significantly improve image pulling speed and reduce application deployment time. VKE also allows you to monitor these image acceleration features. This topic describes how to configure image acceleration observability.​
Background information​
The image acceleration features in VKE mainly include:​
  • P2P acceleration. It utilizes internal network bandwidth resources of nodes to distribute images across nodes. This type of acceleration can reduce pressure on container registries, greatly improve image pull speed, and shorten application deployment time. For details, see Overview of P2P image acceleration.​
  • Image lazy loading (Nydus). The cluster implements image lazy loading through Nydus, improving image pull speed during pod creation. For details, see Lazy loading of container images.​
Prerequisites​
  • The VKE observability feature has been enabled. For details, see VKE observability.​
  • To monitor the P2P acceleration feature, ensure that:​
  • To monitor the image lazy loading (Nydus) feature, ensure that:​
  • The image-accelerator add-on has been installed. For details, see image-accelerator.​
  • To observe the webhook and CPU/memory usage of the image-accelerator add-on, upgrade the image-accelerator add-on to v1.0.6 or later.​
Enabling observability​
Step 1: Configure observability items​
  1. Log in to the VKE console.​
  1. In the left navigation bar, click Cluster. Find the target cluster and click its name.​
  1. In the left navigation pane of the cluster management page, click Observability settings, and select the Basic observability tab.​
  1. Find the Image speedup card and click Enable observation to go to the observability item configuration page.​
  1. The system automatically checks the necessary conditions for enabling observability, such as the workspace configuration and add-on status.​
  1. For the Observation item parameter, select observability items for which you want to collect metrics. For items for which metric collection is not enabled, the system will not collect the corresponding metrics, and the system will prompt you to install the related add-ons. You can decide whether to install the related add-ons and select the corresponding observability items based on your business requirements.​
  1. After the configuration is complete, click Next to go to the metric configuration page.​
Step 2: Configure metric collection​
  1. On the metric configuration page, view the collection interval and metric quantity corresponding to the enabled observability items.​
  1. Click Edit metrics. This enables you to select or discard specific metrics of the add-on, modify the collection interval, and configure metric relabeling.​
  • For Collection interval, select a collection interval for the metrics of the add-on. The supported collection intervals vary for different add-ons.​
  • In the metric list, select a metric to collect. To stop collecting a metric, uncheck it. Click the All, Basic metrics, or Other metrics tab to filter metrics by type.​
tip​
  • The impact of adjusting the metric collection interval is as follows. Adjust the interval as needed:​
  • Reducing the collection interval: This increases the number of metrics reported per unit time and improves monitoring accuracy. However, the cost of standard VMP workspaces increases.​
  • Increasing the collection interval: This decreases the number of metrics reported per unit time and lowers the cost of standard VMP workspaces. However, this decreases monitoring accuracy.​
  • Cloud service metrics are categorized into Basic metrics and Other metrics. The billing method varies based on different types of metrics. For details, see VMP billing.​
  • In the metricRelabelings section, click Add rule to configure metric relabeling rules. This allows you to rewrite the labels of each metric after collection and before writing to storage. Common use cases include modifying label values, adding new labels, and renaming labels to standardize label naming for queries. For details, see Metric relabeling.​
  • ​
    No.​
    Feature​
    Replacement value​
    Description​
    ①​
    Action type​
    action​
    The action to apply to the metrics. Currently, only replace is supported. The logic is as follows:​
    • If the values of targetLabel and sourceLabels are the same, sourceLabels is overwritten.​
    • If the key specified by targetLabel does not exist, a new label is created by default.​
    ②​
    Source labels​
    sourceLabels​
    Specify one or more source labels, separated by | if multiple labels are provided. The system reads the values of these labels for matching or extraction.​
    ③​
    Regular expression​
    regex​
    The regular expression pattern used to match source label values.​
    • The default is (.*), which matches any value.​
    • You can use capture groups such as (.+) to extract partial content.​
    ④​
    Replacement value​
    replacement​
    The value written to the target label after a successful match.​
    • The default is $1, which refers to the content of the first capture group in the regular expression.​
    • A fixed string is also supported.​
    ⑤​
    Target label​
    targetLabel​
    The name of the target label to which the processing result is written.​
    • If the target label already exists, its existing value is overwritten.​
    • If the target label does not exist, a new label is created.​
    ​
  1. After the configuration is complete, click Next to go to the alert and notification configuration page.​
Step 3: Configure alerts and notifications​
  1. In the Alert and notification step, select the platform to receive alerts. Currently, Unified Alert Center and The original VMP alarm strategy are supported.​
  1. For the Notification strategy name parameter, select the created alert notification policy and specify the notification object and Notification method parameters based on different alert levels.​
tip​
  • If you do not need to configure alerts for a certain level, you can unselect the corresponding alert level.​
  • If the existing notification policies do not meet your needs, you can select vke_default_notify_policy (automatically created) for the Notification strategy name parameter, and specify the notification object and Notification method parameters for the required alert levels. After you save the configuration, the system will automatically create a custom notification policy named vke_default_notify_policy in the unified Alert Center or VMP.​
  1. In the Alerting aggregation policy section, select the created alert aggregation policy.​
  1. Configure preset alert rules.​
  • Enable or disable the switch next to a preset alert rule. After it is enabled, the system detects metrics based on the configured trigger conditions and generates alerts. When it is disabled, the rule will no longer trigger new alerts.​
  • Click View PromQL to view the description and corresponding PromQL statement of the current preset alert rule.​
  • Click the name of the preset rule that you want to configure to expand the preset alert rule configuration page, and configure the trigger conditions for alerts of different levels (P0, P1, or P2) in the Triggering condition section.​
  • ​
    Parameter​
    Description​
    Alert level​
    Select the check box in the Enable column that corresponds to each alert level (P0, P1, or P2) to activate its trigger condition. If you do not select an option, alerts at this level are not configured.​
    Comparison condition​
    Specify the comparison condition between the PromQL statement and the threshold. You can configure multiple comparison conditions, including Greater than, Greater than or equal to, Less than, Less than or equal to, Equal to, and Not equal to.​
    Threshold​
    Configure the metric threshold.​
    The duration of the condition​
    Configure the duration for which the metric continuously meets the trigger condition. The system triggers the alert at the corresponding level only if the metric continuously meets the comparison condition within this duration.​
    Label​
    Configure the label of the alert rule. It is mainly used for alert identification, grouping, deduplication, and routing matching. A label consists of a key and a value. Where:​
    • Key: A string. It can start with a letter or an underscore (_), but cannot start with a double underscore (__). It can contain letters, digits, and underscores (_) and can be 1 to 256 characters in length.​
    • Value: A string. It can contain Chinese characters, English letters, digits, and special characters, and can be 1 to 512 characters in length.​
    Common configuration example:​
    ​
    labels:​
    severity: critical # The alert level.​
    business: pay # The business line.​
    alert_type: resource # The alert type.​
    ​
    ​
    Annotation​
    Configure the annotation of the alert rule. It is mainly used to display key information. An annotation consists of a key and a value. Where:​
    • Key: A string. It can start with a letter or an underscore (_), but cannot start with a double underscore (__). It can contain letters, digits, and underscores (_) and can be 1 to 256 characters in length.​
    • Value: A string. It can contain Chinese characters, English letters, digits, and special characters, and can be 1 to 512 characters in length.​
    Common configuration example:​
    ​
    annotations:​
    summary: "High CPU usage on instance {{ $labels.instance }}"​
    description: "Current CPU usage is {{ $value | humanizePercentage }}, threshold is 80%"​
    runbook_url: "https://xxx/docs/cpu-alert"​
    dashboard_url: "https://grafana/xxx"​
    ​
    ​
    ​
  1. Click Complete the configuration.​
Description​
  • In addition to using preset alert templates to configure alert rules directly, you can customize the template configurations or directly use PromQL statements to configure alert rules. For details, see Alert configuration overview.​
  • Details of the preset alert rule templates, including rule descriptions, PromQL configurations, and common alert handling procedures. See Preset alert templates.​
Observability dashboards​
tip​
The built-in dashboards provide features such as period-over-period comparisons of monitoring data and the ability to copy dashboard queries. For details, see Usage of dashboards.​
Image acceleration monitoring​
You can view the built-in monitoring dashboard in the VKE console.​
  1. In the left navigation pane of the cluster management page, click Monitoring center > Dashboard.​
  1. In the left dashboard list, select Image acceleration service monitoring > Image acceleration monitoring to view the monitoring dashboard.​
  • The following table describes the charts on the image acceleration monitoring dashboard.​
Description​
For the method of obtaining the PromQL statements that correspond to the charts, see Using dashboards.​
  • ​
    Chart name​
    Unit​
    Chart description​
    Dragonfly Seed Peer​
    ​
    ​
    UP​
    Count​
    Shows the number of Seed Peer instances summarized by version at the current time, reflecting the availability and deployment scale of instances for each version.​
    • Under normal circumstances, the instance count remains stable and consistent with the expected deployment scale and version distribution.​
    • If this count drops abnormally or falls to 0, some or all instances may be unavailable, or monitoring information collection may be abnormal. Check the pod status, add-on replica count, version changes, and metric collection targets.​
    Traffic distribution (1 hour)​
    Bytes​
    Shows the cumulative increase in traffic aggregated by download type for the Seed Peer over the past 1 hour, reflecting the traffic distribution across different sources or transmission types.​
    • Under normal conditions, the cumulative traffic changes smoothly with actual download load, and the proportion of each type matches the application's access pattern.​
    • If the cumulative traffic or type distribution spikes, drops, or skews abnormally, it typically indicates a sudden change in download load, a shift in P2P hit paths, or abnormal back-to-origin traffic. Check the application release cadence, download type labels, P2P links, and origin server access status.​
    Disk usage space​
    Bytes​
    Shows the total disk space and available disk space (total minus used) aggregated by Pod for the Seed Peer at the current moment, reflecting local cache storage capacity and usage.​
    • Under normal conditions, the total disk space stays stable, and available space is sufficient, changing smoothly with cache writes and reclaims.​
    • If available space drops abnormally or total space changes unexpectedly, it usually indicates continuous cache growth, delayed reclaim, or mount capacity changes. Check the cache directory usage, cleanup policy, disk mounts, and Pod logs.​
    Download bandwidth​
    bit/s​
    Shows the instantaneous download rate (converted to bit/s) aggregated by Pod and download type for the Seed Peer over the last 5 minutes, reflecting the bandwidth load on the download link.​
    • Under normal conditions, the download bandwidth changes smoothly with the business load, and the changes across different download types are consistent with the actual task volume.​
    • If the download bandwidth rises or drops abnormally or fluctuates sharply, it typically indicates a sudden change in download load, link congestion, P2P transmission anomalies, or discontinuous metric collection. Check the download task count, network bandwidth, peer connections, and add-on logs.​
    Upload bandwidth​
    bit/s​
    Shows the instantaneous upload rate (converted to bit/s) aggregated by Pod for the Seed Peer over the last 5 minutes, reflecting the bandwidth load for outbound data serving.​
    • Under normal conditions, the upload bandwidth changes smoothly with downstream request volume and is consistent with the distribution tasks handled by the node.​
    • If this value spikes, drops, or fluctuates sharply, it usually indicates sudden changes in downstream requests, link congestion, data distribution anomalies, or incomplete metric collection. Check the distribution task volume, network bandwidth, peer connections, and add-on logs.​
    Total number of parallel download tasks​
    Count​
    Shows the number of Seed Peers' concurrent download tasks aggregated by Pod at the current moment, reflecting real-time download concurrency pressure.​
    • Under normal conditions, the number of concurrent tasks changes smoothly with the business load and does not accumulate continuously.​
    • An abnormal spike in the number of concurrent tasks, or a prolonged failure to drop back, typically indicates a sudden surge in requests, slow downloads, or task blocking. Check the download queue, network and disk performance, upstream services, and add-on logs.​
    Small file download time (0-64MiB)​
    ms​
    Shows the P50, P95, and P99 download latency percentiles for the Seed Peer in the last 5 minutes, broken down by task size tier for files in the 0–64 MiB range. This information reflects the distribution of latencies for downloading small files.​
    • Under normal conditions, latency at each percentile stays stable, with no continuous widening between higher percentiles and the median.​
    • If download latency increases abnormally or higher percentiles degrade continuously, it usually indicates network congestion, slow peer responses, disk I/O bottlenecks, or task queuing. Check the percentile curves, task size tiers, network and disk performance, and add-on logs.​
    Return to source small file download time (0-64MiB)​
    ms​
    Shows the P50, P95, and P99 latency percentiles of back-to-origin requests for the Seed Peer in the last 5 minutes, broken down by task size tier for files in the 0–64 MiB range. This information reflects the distribution of latencies for downloading data from the origin server.​
    • Under normal conditions, the back-to-origin latency across all percentiles remains stable, with no sustained widening between the higher percentiles and the median.​
    • If the back-to-origin latency rises abnormally or the higher percentiles deteriorate continuously, it typically indicates slow origin server responses, back-to-origin network congestion, disk I/O bottlenecks, or request queuing. Check the origin server's health, back-to-origin link, task size tiers, and add-on logs.​
    Total number of downloads​
    Count​
    Shows the cumulative increase in download tasks aggregated by Pod over the past 1 minute in the Seed Peer, reflecting short-cycle download task load.​
    • Under normal conditions, the task count changes smoothly with download volume and matches actual image pull activity.​
    • If this count spikes, drops, or stops, it usually indicates sudden workload changes, download entry anomalies, or metric collection anomalies. Check image pull events, pod request distribution, add-on state, and the collection pipeline.​
    Total number of failed download tasks​
    Count​
    Shows the cumulative increase in download task failures aggregated by Pod over the past 1 minute in the Seed Peer, reflecting short-cycle download failures.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • If the failure increment rises abnormally, it typically indicates failures in the network, origin server, peer nodes, disk, or task processing pipeline. Check the failed task logs, network connectivity, origin server and peer health status, and disk space.​
    Total number of proxy request tasks​
    Count​
    Shows the cumulative increase in proxy request tasks aggregated by Pod over the past 1 minute in the Seed Peer, reflecting short-cycle proxy request load.​
    • Under normal conditions, the request increment changes steadily in line with business traffic volume and is consistent with actual proxy traffic.​
    • If the request increment rises, drops, or is interrupted abnormally, it usually indicates a sudden change in business traffic, proxy entry anomalies, or metric collection anomalies. Check proxy request logs, client access, Pod status, and the collection pipeline.​
    Total number of proxy request task failures​
    Count​
    Shows the cumulative increase in proxy request task failures aggregated by Pod over the past 1 minute in the Seed Peer, reflecting short-cycle proxy request failures.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • If the failure increment rises abnormally, it typically indicates failures in the proxy path, upstream services, authentication configuration, or network connections. Check the failed request logs, upstream availability, proxy configuration, and network connectivity.​
    Total number of prefetch tasks​
    Count​
    Shows the cumulative increase in prefetch tasks aggregated by Pod over the past 1 minute in the Seed Peer, reflecting short-cycle prefetch load.​
    • Under normal conditions, the task count changes smoothly with the prefetch plan and matches actual prefetch activity.​
    • If this count spikes, drops, or stops, it usually indicates sudden changes in the prefetch plan, task dispatch anomalies, or metric collection anomalies. Check the prefetch configuration, task dispatch records, pod state, and the collection pipeline.​
    Total number of return download requests​
    Count​
    Shows the cumulative increase in back-to-origin download requests aggregated by Pod over the past 1 minute in the Seed Peer, reflecting short-cycle access load on the origin server.​
    • Under normal conditions, the request increment changes smoothly with the business load and is consistent with P2P cache misses and the back-to-origin policy.​
    • An abnormal spike, drop, or interruption in the request increment typically indicates a change in the P2P hit rate, a sudden change in business requests, a back-to-origin policy anomaly, or a metric collection anomaly. Check the P2P hit status, back-to-origin configuration, origin server access logs, and metric collection pipeline.​
    Total number of failed download requests​
    Count​
    Shows the cumulative increase in failed back-to-origin download requests aggregated by Pod over the past 1 minute in the Seed Peer, reflecting short-cycle failures in accessing the origin server.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • An abnormal spike in the failure increment typically indicates origin server unavailability, back-to-origin network failures, authentication failures, or request timeouts. Check the origin server's health status, network connectivity, authentication configuration, and failed request logs.​
    Threads​
    Count​
    Shows the current thread count per Seed Peer process, reflecting process and thread resource usage.​
    • Under normal conditions, the thread count stays stable and changes within a reasonable range with concurrency load.​
    • If this count spikes, drops, or keeps growing, it usually indicates a concurrency surge, thread leaks, task blocking, or process anomalies. Check the process state, thread stacks, task queues, and add-on logs.​
    CPU​
    %​
    Shows the average CPU usage per Pod for the Seed Peer in the last 1 minute, reflecting the compute resource load on the add-on.​
    • Under normal conditions, the CPU usage changes steadily in line with application load, with no sustained high levels or unexplained spikes.​
    • If the CPU usage rises, drops, or fluctuates sharply, it typically indicates sudden request surges, compute hotspots, process blocking, or instance anomalies. Check the Pod load, CPU limits and throttling, hotspot operations, and add-on logs.​
    Memory​
    Bytes​
    Shows the resident memory usage of each Seed Peer process at the current moment, reflecting the physical memory size of the add-on.​
    • Under normal conditions, memory usage remains stable and varies within a controllable range with the business load.​
    • If memory usage rises abnormally or grows continuously, it typically indicates cache accumulation, memory leaks, task backlogs, or a sudden load spike. Check the Pod memory limits, process memory profiles, cache and task queues, and OOM events.​
    Open File Descriptors​
    Count​
    Shows the current number of open file descriptors per Seed Peer process, reflecting file and network connection resource usage.​
    • Under normal conditions, the file descriptor count stays stable and changes smoothly with connection and file access volume.​
    • If this count increases abnormally or keeps growing, it usually indicates connection leaks, unreleased file handles, or request backlogs. Check the process FD limit, network connections, file access, and add-on logs.​
    Dragonfly Peer​
    ​
    ​
    UP​
    Count​
    Shows the number of Peer instances summarized by version at the current time, reflecting the availability and deployment scale of instances for each version.​
    • Under normal circumstances, the instance count remains stable and consistent with the expected deployment scale and version distribution.​
    • If this count drops abnormally or falls to 0, some or all instances may be unavailable, or monitoring information collection may be abnormal. Check the pod status, add-on replica count, version changes, and metric collection targets.​
    Traffic distribution (1 hour)​
    Bytes​
    Shows the cumulative increase in traffic aggregated by download type for the Peers over the past 1 hour, reflecting the traffic distribution across different sources or transfer types.​
    • Under normal conditions, the cumulative traffic changes smoothly with actual download load, and the proportion of each type matches the application's access pattern.​
    • If the cumulative traffic or type distribution spikes, drops, or skews abnormally, it typically indicates a sudden change in download load, a shift in P2P hit paths, or abnormal back-to-origin traffic. Check the application release cadence, download type labels, P2P links, and origin server access status.​
    Disk usage space​
    Bytes​
    Shows the total disk space and available disk space (total minus used) aggregated by Pod for the Peer at the current moment, reflecting local cache storage capacity and usage.​
    • Under normal conditions, the total disk space stays stable, and available space is sufficient, changing smoothly with cache writes and reclaims.​
    • If available space drops abnormally or total space changes unexpectedly, it usually indicates continuous cache growth, delayed reclaim, or mount capacity changes. Check the cache directory usage, cleanup policy, disk mounts, and Pod logs.​
    Download bandwidth​
    bit/s​
    Shows the instantaneous download rate (converted to bit/s) aggregated by Pod and download type for the Peers over the last 5 minutes, reflecting the bandwidth load on the download link.​
    • Under normal conditions, the download bandwidth changes smoothly with the business load, and the changes across different download types are consistent with the actual task volume.​
    • If the download bandwidth rises or drops abnormally or fluctuates sharply, it typically indicates a sudden change in download load, link congestion, P2P transmission anomalies, or discontinuous metric collection. Check the download task count, network bandwidth, peer connections, and add-on logs.​
    Upload bandwidth​
    bit/s​
    Shows the instantaneous upload rate (converted to bit/s) aggregated by Pod for the Peer over the last 5 minutes, reflecting the bandwidth load for outbound data serving.​
    • Under normal conditions, the upload bandwidth changes smoothly with downstream request volume and is consistent with the distribution tasks handled by the node.​
    • If this value spikes, drops, or fluctuates sharply, it usually indicates sudden changes in downstream requests, link congestion, data distribution anomalies, or incomplete metric collection. Check the distribution task volume, network bandwidth, peer connections, and add-on logs.​
    Total number of parallel download tasks​
    Count​
    Shows the number of Peers' concurrent download tasks aggregated by Pod at the current moment, reflecting real-time download concurrency pressure.​
    • Under normal conditions, the number of concurrent tasks changes smoothly with the business load and does not accumulate continuously.​
    • An abnormal spike in the number of concurrent tasks, or a prolonged failure to drop back, typically indicates a sudden surge in requests, slow downloads, or task blocking. Check the download queue, network and disk performance, upstream services, and add-on logs.​
    Small file download time (0-64MiB)​
    ms​
    Shows the P50, P95, and P99 download latency percentiles for the Peers in the last 5 minutes, broken down by task size tier for files in the 0–64 MiB range. This information reflects the distribution of latencies for downloading small files.​
    • Under normal conditions, latency at each percentile stays stable, with no continuous widening between higher percentiles and the median.​
    • If download latency increases abnormally or higher percentiles degrade continuously, it usually indicates network congestion, slow peer responses, disk I/O bottlenecks, or task queuing. Check the percentile curves, task size tiers, network and disk performance, and add-on logs.​
    Return to source small file download time (0-64MiB)​
    ms​
    Shows the P50, P95, and P99 latency percentiles of back-to-origin requests for the Peers in the last 5 minutes, broken down by task size tier for files in the 0–64 MiB range. This information reflects the distribution of latencies for downloading data from the origin server.​
    • Under normal conditions, the back-to-origin latency across all percentiles remains stable, with no sustained widening between the higher percentiles and the median.​
    • If the back-to-origin latency rises abnormally or the higher percentiles deteriorate continuously, it typically indicates slow origin server responses, back-to-origin network congestion, disk I/O bottlenecks, or request queuing. Check the origin server's health, back-to-origin link, task size tiers, and add-on logs.​
    Total number of downloads​
    Count​
    Shows the cumulative increase in download tasks aggregated by Pod over the past 1 minute in the Peer, reflecting short-cycle download task load.​
    • Under normal conditions, the task count changes smoothly with download volume and matches actual image pull activity.​
    • If this count spikes, drops, or stops, it usually indicates sudden workload changes, download entry anomalies, or metric collection anomalies. Check image pull events, pod request distribution, add-on state, and the collection pipeline.​
    Total number of failed download tasks​
    Count​
    Shows the cumulative increase in download task failures aggregated by Pod over the past 1 minute in the Peers, reflecting short-cycle download failures.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • If the failure increment rises abnormally, it typically indicates failures in the network, origin server, peer nodes, disk, or task processing pipeline. Check the failed task logs, network connectivity, origin server and peer health status, and disk space.​
    Total number of proxy request tasks​
    Count​
    Shows the cumulative increase in proxy request tasks aggregated by Pod over the past 1 minute in the Peers, reflecting short-cycle proxy request load.​
    • Under normal conditions, the request increment changes steadily in line with business traffic volume and is consistent with actual proxy traffic.​
    • If the request increment rises, drops, or is interrupted abnormally, it usually indicates a sudden change in business traffic, proxy entry anomalies, or metric collection anomalies. Check proxy request logs, client access, Pod status, and the collection pipeline.​
    Total number of proxy request task failures​
    Count​
    Shows the cumulative increase in proxy request task failures aggregated by Pod over the past 1 minute in the Peers, reflecting short-cycle proxy request failures.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • If the failure increment rises abnormally, it typically indicates failures in the proxy path, upstream services, authentication configuration, or network connections. Check the failed request logs, upstream availability, proxy configuration, and network connectivity.​
    Total number of prefetch tasks​
    Count​
    Shows the cumulative increase in prefetch tasks aggregated by Pod over the past 1 minute in the Peers, reflecting short-cycle prefetch load.​
    • Under normal conditions, the task count changes smoothly with the prefetch plan and matches actual prefetch activity.​
    • If this count spikes, drops, or stops, it usually indicates sudden changes in the prefetch plan, task dispatch anomalies, or metric collection anomalies. Check the prefetch configuration, task dispatch records, pod state, and the collection pipeline.​
    Total number of prefetch task failures​
    Count​
    Shows the cumulative increase in prefetch task failures aggregated by Pod over the past 1 minute in the Peers, reflecting short-cycle prefetch failures.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • If the failure increment rises abnormally, it typically indicates prefetch source unavailability, network or disk anomalies, invalid task parameters, or processing pipeline failures. Check the prefetch task logs, origin server availability and network connectivity, disk space, and prefetch configuration.​
    Total number of return download requests​
    Count​
    Shows the cumulative increase in back-to-origin download requests aggregated by Pod over the past 1 minute in the Peers, reflecting short-cycle access load on the origin server.​
    • Under normal conditions, the request increment changes smoothly with the business load and is consistent with P2P cache misses and the back-to-origin policy.​
    • An abnormal spike, drop, or interruption in the request increment typically indicates a change in the P2P hit rate, a sudden change in business requests, a back-to-origin policy anomaly, or a metric collection anomaly. Check the P2P hit status, back-to-origin configuration, origin server access logs, and metric collection pipeline.​
    Total number of failed download requests​
    Count​
    Shows the cumulative increase in failed back-to-origin download requests aggregated by Pod over the past 1 minute in the Peers, reflecting short-cycle failures in accessing the origin server.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • An abnormal spike in the failure increment typically indicates origin server unavailability, back-to-origin network failures, authentication failures, or request timeouts. Check the origin server's health status, network connectivity, authentication configuration, and failed request logs.​
    Threads​
    Count​
    Shows the current thread count per Peer process, reflecting process and thread resource usage.​
    • Under normal conditions, the thread count stays stable and changes within a reasonable range with concurrency load.​
    • If this count spikes, drops, or keeps growing, it usually indicates a concurrency surge, thread leaks, task blocking, or process anomalies. Check the process state, thread stacks, task queues, and add-on logs.​
    CPU​
    %​
    Shows the average CPU usage per Pod for the Peers in the last 1 minute, reflecting the compute resource load on the add-on.​
    • Under normal conditions, the CPU usage changes steadily in line with application load, with no sustained high levels or unexplained spikes.​
    • If the CPU usage rises, drops, or fluctuates sharply, it typically indicates sudden request surges, compute hotspots, process blocking, or instance anomalies. Check the Pod load, CPU limits and throttling, hotspot operations, and add-on logs.​
    Memory​
    Bytes​
    Shows the resident memory usage of each Peer process at the current moment, reflecting the physical memory size of the add-on.​
    • Under normal conditions, memory usage remains stable and varies within a controllable range with the business load.​
    • If memory usage rises abnormally or grows continuously, it typically indicates cache accumulation, memory leaks, task backlogs, or a sudden load spike. Check the Pod memory limits, process memory profiles, cache and task queues, and OOM events.​
    Open File Descriptors​
    Count​
    Shows the current number of open file descriptors per Peer process, reflecting file and network connection resource usage.​
    • Under normal conditions, the file descriptor count stays stable and changes smoothly with connection and file access volume.​
    • If this count increases abnormally or keeps growing, it usually indicates connection leaks, unreleased file handles, or request backlogs. Check the process FD limit, network connections, file access, and add-on logs.​
    Dragonfly Scheduler​
    ​
    ​
    UP​
    Count​
    Shows the number of Scheduler instances summarized by version at the current time, reflecting the availability and deployment scale of instances for each version.​
    • Under normal circumstances, the instance count remains stable and consistent with the expected deployment scale and version distribution.​
    • If this count drops abnormally or falls to 0, some or all instances may be unavailable, or monitoring information collection may be abnormal. Check the pod status, add-on replica count, version changes, and metric collection targets.​
    Total number of gRPC requests​
    Count​
    Shows the cumulative increase in processed requests aggregated by Pod and gRPC status code over the past 1 minute in the Scheduler, reflecting short-cycle gRPC request load and result distribution.​
    • Under normal conditions, the request increment changes smoothly with the scheduling load, with success status codes accounting for the majority.​
    • An abnormal spike in the request increment or the number of error status codes typically indicates a sudden surge in call volume, service processing failures, timeouts, or dependency anomalies. Check the gRPC status code distribution, caller requests, Scheduler logs, and dependent services.​
    Total number of parallel scheduling tasks​
    Count​
    Shows the number of concurrent scheduling tasks aggregated by Pod in the Scheduler at the current moment, reflecting real-time scheduling concurrency pressure.​
    • Under normal conditions, the number of concurrent tasks changes smoothly with the business load and does not accumulate continuously.​
    • If this count spikes or stays high for an extended period, it usually indicates scheduling request surges, slow processing, or task blocking. Check the scheduling queue, CPU and memory resources, downstream interactions, and Scheduler logs.​
    Total number of scheduled registered tasks​
    Count​
    Shows the cumulative increase in scheduling registration tasks aggregated by Pod over the past 1 minute in the Scheduler, reflecting short-cycle node registration load.​
    • Under normal conditions, the task increment changes steadily in line with node registration activity and is consistent with actual node joining or registration behavior.​
    • If the task increment rises, drops, or is interrupted abnormally, it typically indicates a sudden change in node registration activity, registration path anomalies, or metrics collection pipeline issues. Check node joining events, registration request logs, Scheduler status, and the metric collection pipeline.​
    Total number of failed scheduling registration tasks​
    Count​
    Shows the cumulative increase in scheduling registration task failures aggregated by Pod over the past 1 minute in the Scheduler, reflecting short-cycle node registration failures.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • If the failure increment rises abnormally, it typically indicates invalid node information, network anomalies, service processing failures, or dependency faults. Check the registration failure logs, node parameters, network connectivity, and dependent services.​
    Total number of scheduled tasks​
    Count​
    Shows the cumulative increase in scheduling tasks aggregated by Pod over the past 1 minute in the Scheduler, reflecting short-cycle scheduling load.​
    • Under normal conditions, the task count changes smoothly in line with download requests and is consistent with actual scheduling activity.​
    • If this count spikes, drops, or stops, it usually indicates sudden changes in scheduling requests, service processing anomalies, or metric collection anomalies. Check scheduling request logs, client activity, Scheduler status, and the collection pipeline.​
    Total number of scheduled task failures​
    Count​
    Shows the cumulative increase in scheduling task failures aggregated by Pod over the past 1 minute in the Scheduler, reflecting short-cycle task scheduling failures.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • If the failure increment rises abnormally, it typically indicates insufficient available peers, scheduling policy anomalies, dependency failures, or invalid request parameters. Check the scheduling failure logs, peer node status, scheduling configuration, and dependent services.​
    Total number of information reported by nodes​
    Count​
    Shows the cumulative increase in node info reports aggregated by Pod over the past 1 minute in the Scheduler, reflecting short-cycle node status reporting load.​
    • Under normal conditions, the report increment changes steadily in line with node scale and reporting intervals.​
    • If the report increment rises, drops, or is interrupted abnormally, it usually indicates sudden changes in node count, reporting retries, network issues, or collection anomalies. Check node status, reporting interval, network connectivity, and Scheduler logs.​
    Total number of failures reported by nodes​
    Count​
    Shows the cumulative increase in node info report failures aggregated by Pod over the past 1 minute in the Scheduler, reflecting short-cycle node status report failures.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • If this count increases abnormally, it usually indicates invalid node information, network issues, service processing failures, or dependency faults. Check reporting failure logs, node state, network connectivity, and dependent services.​
    Total number of nodes releasing information​
    Count​
    Shows the cumulative increase in node release info events aggregated by Pod over the past 1 minute in the Scheduler, reflecting short-cycle node exits or release event load.​
    • Under normal conditions, the event increment matches the actual node removal or release activity, with no unexplained spikes.​
    • If this count spikes, drops, or stops, it typically indicates frequent node joining or removal, release process anomalies, or metric collection anomalies. Check node lifecycle events, release request logs, Scheduler state, and the collection pipeline.​
    Total number of failures to release information from nodes​
    Count​
    Shows the cumulative increase in node release info failures aggregated by Pod over the past 1 minute in the Scheduler, reflecting short-cycle node exit or release failures.​
    • Under normal conditions, the failure increment remains at 0 or stays low and stable.​
    • If this count increases abnormally, it usually indicates inconsistent node states, network issues, service processing failures, or dependency faults. Check release failure logs, node state, network connectivity, and dependent services.​
    Goroutines​
    Count​
    Shows the current Goroutine count per Scheduler process, reflecting the Go runtime concurrency scale.​
    • Under normal conditions, the Goroutine count stays stable and changes within a reasonable range with scheduling load.​
    • If this count spikes, drops, or keeps growing, it usually indicates a concurrency surge, Goroutine leaks, task blocking, or process anomalies. Check Goroutine stacks, task queues, dependency calls, and Scheduler logs.​
    CPU​
    %​
    Shows the average CPU usage per Pod for the Scheduler in the last 1 minute, reflecting the compute resource load on the add-on.​
    • Under normal conditions, the CPU usage changes steadily in line with scheduling load, with no sustained high levels or unexplained spikes.​
    • If the CPU usage rises, drops, or fluctuates sharply, it typically indicates a sudden surge in scheduling requests, compute hotspots, process blocking, or instance anomalies. Check the Pod load, CPU limits and throttling, hotspot operations, and Scheduler logs.​
    Memory​
    Bytes​
    Shows the resident memory usage of each Scheduler process at the current moment, reflecting the physical memory size of the add-on.​
    • Under normal conditions, memory usage remains stable and varies within a controllable range with the scheduling load.​
    • If memory usage rises abnormally or grows continuously, it typically indicates cache accumulation, memory leaks, task backlogs, or a sudden load spike. Check the Pod memory limits, process memory profiles, cache and task queues, and OOM events.​
    Open File Descriptors​
    Count​
    Shows the current number of open file descriptors per Scheduler process, reflecting file and network connection resource usage.​
    • Under normal conditions, the file descriptor count stays stable and changes smoothly with connection and file access volume.​
    • If this count increases abnormally or keeps growing, it usually indicates connection leaks, unreleased file handles, or request backlogs. Check the process FD limit, network connections, file access, and Scheduler logs.​
    ​
P2P acceleration monitoring​
You can view the built-in monitoring dashboard in the VKE console. The dashboard presents information such as the total number of proxy requests, total number of current proxy requests, total number of tasks, total number of failed tasks, and total number of shards.​
  1. In the left navigation pane of the cluster management page, click Monitoring center > Dashboard.​
  1. In the left dashboard list, click Image acceleration service monitoring > P2P image acceleration monitoring to view the monitoring dashboard.​
tip​
You can also deploy self-managed Grafana in the cluster and use it to view metrics and create dashboards. For more details, see Deploying Grafana in a VKE cluster and integrating Grafana with a workspace.​
Image lazy loading (Nydus) monitoring​
After you configure Nydus-based image lazy loading, you can view the built-in monitoring dashboard in the VKE console. The dashboard presents information such as Nydus Daemon quantity and Nydus Daemon memory usage.​
  1. In the left navigation pane of the cluster management page, click Monitoring center > Dashboard.​
  1. In the left dashboard list, click Image acceleration service monitoring > Image lazy loading (Nydus) monitoring to view the monitoring dashboard.​
tip​
You can also deploy self-managed Grafana in the cluster and use it to view metrics and create dashboards. For details, see Integrating data from VMP into self-managed Grafana.​
View metrics​
  • To query integrated or non-integrated metrics in the cluster, or to understand metric meanings, see Querying metrics.​
  • You can use the Explore feature of VMP to quickly query and display metric data. For details, see Querying metrics.​
​
Last updated: 2026.10.22 17:24:38