Probe management provides full lifecycle management for data collection probes in the observability system, including installing probes, viewing probe details, modifying configuration, upgrading, monitoring status, and more. These management operations ensure the stability, reliability, and trustworthiness of observability data in large-scale distributed systems. This topic describes how to install and manage probes.

Background

Glossary

Term	Description
Add-on	A component that manages probes and acts as a controller. It is primarily used to deliver configurations to probes and is typically deployed as a Deployment in container environments. For example, the apmplus-opentelemetry-collector component in a VKE cluster is primarily used to deliver configurations to and manage the o11yagent-apm-otel-collector probe.
Probe	An independent component embedded in a target system to collect data. It continuously collects metrics, logs, traces, and other data without changing business logic and reports data to a monitoring storage backend. For example, the o11yagent-apm-otel-collector probe in server-side applications collects metrics, logs, traces, and other data from the corresponding application and reports the data in batches to Server-side Monitoring.

Metric descriptions

The following table describes the metrics displayed on the monitoring dashboards for probes and probe instances:

Metric	Description
CPU	The CPU usage of a probe or probe instance. This metric can be used to evaluate collector load, identify traffic spikes, and prevent computing bottlenecks caused by overload.
Mem	The memory usage of a probe or probe instance, which reflects the memory overhead of observability data buffers and parsing and processing logic.
OTEL Receiver Bytes	The number of bytes of raw data received per second by a probe or probe instance. This metric can be used to measure the total volume of data pushed from upstream services to the probe and detect sudden spikes in business instrumentation reporting.
OTEL Receiver Request Count	The number of reporting requests received per second by a probe or probe instance. This metric primarily reflects the request frequency of business reporting and can be used to determine whether the reporting frequency is abnormal on the business side.
Flush Bytes	The total outbound traffic (in bytes per second) that a probe or probe instance pushes in batches to the observability storage backend after aggregation and compression.
Flush Count	The number of batch reports per second by a probe or probe instance. This metric indicates the frequency of network calls that the probe makes to push data to the backend. A higher Flush Count indicates that the probe divides data into smaller batches, causing greater network request overhead.

Prerequisites

Before installing a third-party cloud container probe or host probe, ensure that you have obtained the Volcano Engine account AK/SK information with APMPlusServerDataExportRolePolicy permission, which is used to authorize the third-party cloud for probe installation and data reporting. For information about how to obtain these credentials, see How do I obtain API access keys?

Before installing a third-party cloud container probe, obtain the cluster name information for the container cluster in the third-party cloud.

Currently, host probes can be installed only on Linux hosts running CentOS 7 series or later.

When you use the VPC access method during probe installation, ensure network connectivity between the resources to be collected and the Observability Platform.

Install probes

Volcano Engine VKE probes

Third-party cloud container probes

Host probes

Log in to the Observability Platform console.

In the left navigation pane, select Integration center -> Probe Management.

Click Install probe in the upper-right corner, and select Volcano Engine VKE on the Container probe tab.

Select the target region and complete cross-service authorization to ensure that the Observability Platform can properly obtain VKE cluster and add-on installation information.

tip

Cross-service authorization needs to be configured only once. If cross-service authorization has not been completed, follow the on-screen instructions for configuration.

Select the target cluster to install the probe.

If the cluster does not have a probe installed, click Install with one-click to install the latest version of the add-on and container probe.

If the cluster already has a probe installed, reinstallation is not required. Additionally, click Configuration to navigate to the container cluster add-on management page. On this page, you can configure deployment information for the probe instance. Click Uninstall to quickly uninstall the add-on and container probe.

Log in to the Observability Platform console.

In the left navigation pane, select Integration center -> Probe Management.

Click Install probe in the upper-right corner, and select Third-party cloud on the Container probe tab.

Select the target region and configure parameters as needed. The installation command for the third-party cloud container probe will be generated.

Parameter	Description
Integration methods	The method for a third-party cloud application to access Server-side Monitoring. You can select VPC access or public network access based on network connectivity and security requirements. tip To use VPC access, ensure the resources you want to collect are network-connected to the Observability Platform.
Cloud provider type	The cloud provider to which the application belongs. Currently, Tencent Cloud, Huawei Cloud, Alibaba Cloud, AWS, and self-built Kubernetes are supported.
Cluster name	The name of the container cluster in the third-party cloud where the probe will be installed.
Workload type	The workload type used when you deploy the probe as a container. You can select Deployment, DaemonSet, or StatefulSet as needed. Deployment: Deploys probe instances via a stateless workload and supports elastic scaling of multiple replicas. This type is suitable for stateless, centralized collection scenarios, for example, APM, trace, and cluster-level metric aggregation. DaemonSet: Deploys probe instances via a DaemonSet. This type is suitable for node logs, host monitoring, container metrics, and profiling scenarios. StatefulSet: Deploys probe instances via a stateful workload. This type is suitable for Prometheus collection scenarios.
Access Key	The AK information of the Volcano Engine account, together with the SK information, is used to authorize the third-party cloud for probe installation and data reporting.
Secret Key	The SK information corresponding to the Volcano Engine account AK.

Connect to the third-party cloud container cluster via kubectl and execute the installation command to install the probe. After the probe is successfully installed, wait 3 to 5 minutes before monitoring data can be collected normally.

Execute the kubectl get pods -n o11yagent-monitor command to view the installed probe processes. If the probe STATUS is ONLINE, the installation succeeds.

To uninstall an installed probe, execute the helm uninstall o11yagent-operator -n o11yagent-monitor command.

Log in to the Observability Platform console.

In the left navigation pane, select Integration center -> Probe Management.

Click Install probe in the upper-right corner.

Select the target region and configure parameters as needed. The installation command for the host probe will be generated.

Parameter	Description
Operating system	Select the operating system of the host where the probe will be installed. Currently, Linux is supported.
Cloud vendor	Select the cloud provider to which the host belongs. Currently, Volcano Engine, Tencent Cloud, Huawei Cloud, and self-built hosts are supported.
Integration methods	The method for the host to access Server-side Monitoring. You can select VPC access or public network access based on network connectivity and security requirements. tip To use VPC access, ensure the resources you want to collect are network-connected to the Observability Platform.
workspace	The workspace for host access to Server-side Monitoring. Workspaces are a data management and isolation method provided by Server-side Monitoring. Each workspace has independent data reporting credentials and can centrally manage traces, logs, and metrics for different application ownership scenarios.
Access Key	The AK information of the Volcano Engine account, together with the SK information, is used to authorize the third-party cloud for probe installation and data reporting.
Secret Key	The SK information corresponding to the Volcano Engine account AK.

Log in to the host and execute the installation command to install the probe. After the probe is successfully installed, wait 3 to 5 minutes before monitoring data can be collected normally.

Execute the o11yagentctl ps command to view the installed probe processes. If the probe STATUS is ONLINE, the installation succeeds.

To uninstall an installed probe, execute the wget -qO- https://o11yagent-registry-cn.tos-cn.ivolces.com/scripts/o11yagentd/latest/o11yagentd.sh | bash -s -- uninstall --service o11yagent_cloud command.

Manage probes

Volcano Engine VKE probes

Third-party cloud container probes

Host probes

Log in to the Observability Platform console.

In the left navigation pane, select Integration center -> Probe management, and then go to the Volcano Engine VKE tab under Container probe. Then, view the list of add-ons associated with integration tasks.

tip

Add-ons in the same environment type have the same name and can be distinguished by the cluster they belong to.

Upgrade an add-on: When a new version of an add-on is released, a reminder icon will appear in the target add-on row. Click Upgradable to upgrade to the latest version with one click.

tip

The platform releases new versions of add-ons periodically. We recommend that you upgrade regularly to access the latest features.

Deployment configuration: Find the target add-on and click Configuration in the "Actions" column to modify its deployment settings, including deployment method, number of instances, CPU request/limit, and memory request/limit. For configuration recommendations, see Recommended apmplus-opentelemetry-collector specifications.

Click the name of the target add-on. You can then view the probe(s) managed by the add-on in the expanded section.

Click the name of the target probe to go to the details page of the corresponding workload and view more workload details.

Click Details in the "Actions" column to the right of the target probe to view probe details and the probe instance.

Section	Description
Cluster	The container cluster to which the current probe belongs. Click the cluster ID in the "Basic information" section to go to the cluster management page.
Resource Configuration	The Request/Limit configuration of the current probe. Click Edit to modify the configuration. Request: The requested amounts of CPU and memory resources. Limit: The upper limits of CPU and memory resource usage.
Probe Instance	Click the name of the target probe instance to go to the Pod details page corresponding to the instance and view more information.
		Click Metric in the "Actions" column to the right of the target probe instance to view monitoring dashboards for the probe instance metrics. You can use this feature for performance analysis. For more information about metrics, see Metric descriptions.
		Click Runtime Logs in the "Actions" column to the right of the target probe instance to go to the log management page and view logs generated while the current probe instance is running. You can use this feature for fault analysis.
Integration Task	The integration task associated with the current probe. Click the integration task name to go to the integration task configuration details page. For details, see Integration management.
Monitoring dashboard	The monitoring dashboards for metrics related to the current probe, which can be used for performance analysis. For more information about metrics, see Metric descriptions.
Runtime Logs	Go to the log page of the workload corresponding to the probe and view logs generated while the current probe is running. You can use this feature for fault analysis.

Observability Platform