Python applications have different integration methods in various scenarios. This document describes the detailed actions for integrating Python applications with OpenTelemetry. After integration, you can automatically collect trace information, automatically generate metrics, and use features such as trace search and log search.

Integrate via OpenTelemetry

OpenTelemetry is an observability project under CNCF, designed to provide standardized solutions in the observability domain. It addresses standardization issues for data models, collection, processing, and export of observability data, and offers services independent of third-party vendors.

If your application has already integrated the OpenTelemetry SDK, the APM server side can receive data reported by OpenTelemetry. After integrating with APM, you will gain capabilities such as RED metric generation, multidimensional trace search, single trace search, topology generation, and more.

Limits

APMPlus uses Prometheus as the storage for OpenTelemetry metrics and supports querying metrics with promql. For specific applications, see custom dashboard and alert management.

There are certain limitations when converting OpenTelemetry metrics to Prometheus metrics:

As with open source, ExponentialHistogram is not supported.

Data with aggregation method delta is not supported. Do not modify the default aggregation configuration on the SDK side. Ensure that the aggregation method for SDK side data is cumulative.

Each workspace has its own data reporting credentials. To manage and isolate data using a workspace, ensure that the apmplus-opentelemetry-collector component has been upgraded to version v3.1.0 or above.

Considerations

Caution for gevent applications

To use gevent coroutines, you need to set the environment variable GEVENT_ENABLE=true. For example, if your program uses the following coroutine, you need the environment variable GEVENT_ENABLE=true.

Caution for Django applications

If your application uses the Django framework, pay attention to the following before integration:

To automatically integrate via OpenTelemetry in a K8s environment, ensure that the Python version is 3.8 or above;

It is recommended to deploy the service using uWSGI. For deployment methods, see Host Django applications with uWSGI. Starting directly with the Python command may result in reporting failures.

The introduction of OpenTelemetry may cause Django applications to stop using the default configuration file, requiring you to re-specify the configuration file via environment variables:

You can add this setting to the project's startup script, or add environment variables through a YAML file:

Automatic integration

APM OpenTelemetry Collector is a data collector developed by APM based on OpenTelemetry Collector, compliant with the Apache 2.0 license. It supports the collection of OpenTelemetry Monitoring data, as well as the collection of Prometheus metrics, application Monitoring data, and Kubernetes resource metrics.

Install APM OpenTelemetry Collector.

Install the apmplus-opentelemetry-collector component via the VKE cluster component center. Go to the VKE cluster interface, select Component Management -> Monitoring, and click Install in the upper right corner of the apmplus-opentelemetry-collector component under ... to complete the deployment and management of the Apmplus OpenTelemetry Collector.

Add the instrumentation.apmplus.volcengine.com/inject-python and instrumentation.apmplus.volcengine.com/workspace annotations to the Kubernetes Deployment configuration of the service. The following is a code example:

apiVersion: apps/v1
kind: Deployment
metadata:
name: python-flask-demo
labels:
app: python-flask-demo
spec:
replicas: 1
selector:
matchLabels:
app: python-flask-demo
template:
metadata:
labels:
app: python-flask-demo
annotations:
instrumentation.apmplus.volcengine.com/inject-python: 'true' # Enable APMPlus observation injection
instrumentation.apmplus.volcengine.com/workspace: default # Specify the workspace to which the service belongs
spec:
containers:
- name: python-flask-demo
image: python-flask:latest
imagePullPolicy: Always
env:
- name: CONTAINER_RUNTIME
value: "k8s"

Restart the application.

Opentelemetry SDK integration

To report data by manually instrumenting with the community-provided Opentelemetry Python SDK, refer to the community's integration documentation and python demo to complete the Python project setup.

Complete instrumentation and reporting using opentelemetry-instrument

Install the dependency packages.

The opentelemetry-distro package installs the API, SDK, as well as the opentelemetry-bootstrap and opentelemetry-instrument tools.

Specify the data reporting backend and authentication parameters in the startup command.

Parameters	Descriptions
<your-service_name>	Application service name.
<apmplus_endpoint>	The server address for the corresponding environment. You can go to apmplus_endpoint/collector_endpoint Access Center to view and obtain it. For details, see: How to use the Access Center?.
<collector_endpoint>
<app_key>	The app_key provided by the service access console. For the method to obtain it, see: How to obtain the authentication information AppKey for application performance monitor?.

To report data directly to the ApmPlus backend, see the following example:

To report data to the Collector, there is no need to specify exporter_otlp_headers. See the following example:

You can also configure these parameters via environment variables, as shown below:

Manual instrumentation using opentelemetry-sdk

Download the API and SDK packages.

In the code, create TracerProvider, MeterProvider, and LoggerProvider and use them to report data.

For related information in the official Opentelemetry documentation, refer to Instrumentation. The demo in the official documentation prints metrics and traces to the console. To report data to the collector, in addition to using the grpc/http exporter, you also need to set the endpoint, headers, and other information.

It is recommended to use the collector endpoint as the reporting address. See the demo example below:

TraceProvider example:

MeterProvider example:

LoggerProvider example:

For log reporting, in addition to initializing LoggerProvider, it is recommended to use opentelemetry LoggingInstrumentor to instrument the logging library, thereby associating logs and traces.

Install LoggingInstrumentor

Initialize LoggerProvider and add LoggingInstrumentor

Frameworks supported for automatic injection

Supported Packages	Version
volcenginesdkarkruntime(Volcano Engine Ark agent SDK)	>= 0.0.1
openai	>= 0.27.0
langchain	>= 0.0.346
mcp	>=1.6.0, <=1.8.0
aio_pika	>= 7.2.0, < 10.0.0
aiohttp	~= 3.0
aiopg	>= 0.13.0, < 2.0.0
asgiref	~= 3.0
asyncio	all
asyncpg	>= 0.12.0
aws_lambda	all
boto	~=2.0
boto3	~= 1.0
botocore	~= 1.0
cassandra-driver	~= 3.25
scylla-driver	~= 3.25
celery	>= 4.0, < 6.0
click	>= 8.1.3, < 9.0.0
confluent-kafka	>= 1.8.2, <= 2.4.0
dbapi	all
django	>= 1.10
elasticsearch	>= 6.0
falcon	>= 1.4.1, < 4.0.0
fastapi	~= 0.58
flask	>= 1.0
grpcio	>= 1.42.0
httpx	>= 0.18.0
jinja2	>= 2.7, < 4.0
kafka-python	>= 2.0, < 3.0
kafka-python-ng	>= 2.0, < 3.0
logging	all
mysql-connector-python	>= 8.0, < 10.0
mysqlclient	< 3
pika	>= 0.12.0
psycopg	>= 3.1.0
psycopg2	>= 2.7.3.1
pymemcache	>= 1.3.5, < 5
pymongo	>= 3.1, < 5.0
PyMySQL	< 2
pyramid	>= 1.7
redis	>= 2.6
remoulade	>= 0.50
requests	~= 2.0
sqlalchemy	>= 1.0.0, < 2.1.0
sqlite3	all
starlette	~= 0.13.0
psutil	>= 5
threading	all
tornado	>= 5.1.1
tortoise-orm	>= 0.17.0
urllib	all
urllib3	>= 1.0.0, < 3.0.0
wsgi	all

Python log integration and considerations

Log collection for Python applications relies on OpenTelemetry. Once the entire process of integrating OpenTelemetry into Python applications is completed, log information can be automatically collected and reported.

Operating principle

OpenTelemetry Python logs are implemented by instrumenting the logging library. By enhancing log information and adding the corresponding reporting handler, logs are reported to the APMPlus backend and associated with Trace and Span information. Therefore, only logs printed through the logging library can be collected and sent to APMPlus. Logs generated by other methods, such as the console, cannot be collected.

The complete process for a log in a Python application, from its creation to being sent via OpenTelemetry and reported to the APMPlus backend, is as follows:

Considerations

When collecting logs from Python applications, note the following considerations.

Log level configuration

The Python logging library has a log level configuration, which defaults to WARNING. If a log's level is lower than the configured level, the log will not be printed, nor will it be reported to the APMPlus server side. Therefore, to collect logs such as INFO and DEBUG, the log level must be manually set using logging.basicConfig. Example code is as follows:

Propagate parameter configuration

Typically, logging is not used directly for log recording; instead, logging.getLogger(name) is used to create a logger, and logs are recorded through the logger. OpenTelemetry logging is instrumented based on the root logger. Therefore, when logs are recorded using a logger and propagate = False is set, logs recorded by the logger will not propagate to the root logger and cannot be collected or reported by OpenTelemetry.

Each logger instance created via logging.getLogger(name) has a propagate property, which defaults to True. The operating principle is as follows.

propagate = True (default): After a logger processes a log, it checks the propagate flag. If it is True, the log record will continue to be passed up to the parent logger. This process continues until the root logger.

propagate = False: After a logger processes a log, propagation will stop. The parent logger and higher-level ancestor loggers will be completely unaware of the existence of this log.

Asynchronous log recording

Log recording needs to associate logs with the request trace to which they belong. Therefore, in OpenTelemetry LoggingInstrumentor, log information is enhanced by obtaining the context information at the time of log printing and storing it in the log information. However, when an asynchronous method is used to record logs in actual business scenarios, context information may be lost. In this case, logs cannot be associated with a Trace, resulting in the TraceId information being displayed as empty on the APMPlus log page.

Troubleshooting

When log collection does not yield any log data, after ruling out the situations mentioned in the Note section, you can troubleshoot the issue by proactively adding the corresponding logs. Example:

After setting logging basicConfig and before printing logs, add the following code to print the handler and level information of the logging library.

At this point, if LoggingInstrumentor is enabled, the logging.root.handlers in the output information will include the following handler.

If you use logger to print logs, you can use the following code to print logs.

At this point, if LoggingInstrumentor is enabled, the logging.root.handlers in the output information will include the following handler.

Application Performance Management Plus