You need to enable JavaScript to run this app.
E-MapReduce

E-MapReduce

  • Documentation
    • E-MapReduceBest practices for EMR Serverless queuesIntegrate EMR Serverless with log service TLS
Copy page
Download PDF
Best practices for EMR Serverless queues
Integrate EMR Serverless with log service TLS
Copy page
Download PDF
Integrate EMR Serverless with log service TLS

EMR Serverless supports integration with the VolcanoEngine log service (TLS) product, delivering job-related operation log data to TLS for unified processing and management. This document introduces the key steps for integrating EMR Serverless with TLS to deliver log data.

Limits

Currently, RayJob only supports delivery of Driver logs. Spark supports delivery of both Driver and Executor logs. Logs for other tasks can be viewed in the job instance details.

Prerequisite configuration
  1. Enable TLS , and create a project and log topic.
  2. After completing the TLS-side configuration, you can submit a ticket to contact VolcanoEngine technical support personnel and provide the TopicID, so they can help you complete the EMR-side configuration. Once technical support personnel complete the EMR-side configuration, you can subsequently deliver supported EMR Serverless job log data to your TLS log topic.

View log details

After completing the prerequisite steps, you can then view detailed log details in the corresponding project and topic in the TLS console.

tip

When going to the TLS console to view log details, ensure that your account has at least permissions to view log projects and log details in the TLS console.

The supported log types and example content are as follows:

Log types

Examples

Spark Driver logs

{
    "__container_name__": "spark-kubernetes-driver",
    "__content__": "26/04/29 16:31:19 WARN [main] SparkConf: The configuration key 'spark.scheduler.listenerbus.eventqueue.size' has been deprecated as of Spark 2.3 and may be removed in the future. Please use the new key 'spark.scheduler.listenerbus.eventqueue.capacity' instead.",
    "__context_flow__": "1777451480893#c969921b8ec7b233-11690e6******1-0",
    "__image_name__": "emr-serverless-online-cn-beijing.cr.volces.com/emr-serverless-engine/spark:3.5.1-py3.12-ubuntu20.04-742",
    "__namespace__": "ns-2******-144000******",
    "__package_offset__": "1696903452055484",
    "__path__": "/var/data/spark-bda2526c-2e42-4c1b-a052-af8a7adb57b5/logs/syslog",
    "__pod_name__": "bq-3******-driver",
    "__pod_uid__": "48941247-2e26-45fe-bcd0-fd******",
    "__tag____receive_time__": "1777451481889",
    "__tag__account_id__": "emr_serverless_2******",
    "__time__": 1777451480893,
    "filename": "syslog"
}

Spark Executor logs

{
    "__content__": "I20260429 16:59:44.233684 140043121325824 ShuffleWriterNode.cc:144] init: forceShuffleWriterType = 0, preAllocSize 30 <= 2000, partitions = 1000, partitioning = 2, use VeloxShuffleWriterV2",
    "__context_flow__": "1777453185223#afb80dc061713c6b-af60ee******c-239",
    "__package_offset__": "30977949650294357",
    "__path__": "/data/ns-2******-1341100243457933312/bq-332652190-exec-7/var/data/spark-25d32af9-9d66-4ef8-97d6-484528d59798/logs/stderr",
    "__tag____receive_time__": "1777453185338",
    "__time__": 1777453185223,
    "account_id": "2******",
    "cluster_id": "ccvm01a30mbf393fuur0g",
    "filename": "stderr",
    "pod_name": "bq-332652190-exec-7"
}

RayJob logs

{
    "__container_name__": "ray-head",
    "__content__": "\u001b[36m(UDFActor pid=5071, ip=9.127.161.106)\u001b[0m 2026-04-29 16:33:20 - PdfParseUDF-140438855291600 - WARNING - Got HTTP 429 (Too Many Requests), retrying (attempt 1/4) after 5.0s delay\u001b[32m [repeated 34x across cluster]\u001b[0m",
    "__context_flow__": "1777451603065#0a48b3a926d4a57b-7eef4c45df7d94f5-13856",
    "__image_name__": "las-ai-online-cn-beijing.cr.volces.com/las/ve-daft:0.7.2.post4-py3.11-ubuntu24.04",
    "__namespace__": "ns-2******-1441487549972348928",
    "__package_offset__": "2822841226614695",
    "__path__": "/tmp/ray/session_2026-04-28_22-34-19_670047_1/logs/job-driver-rayjob-3******-trntt.log",
    "__pod_name__": "rayjob-3******-nfp44-head-7lzth",
    "__pod_uid__": "73c3d104-5db1-4be3-99f3-76139c58c82a",
    "__tag____receive_time__": "1777451604567",
    "__tag__ray_job_name__": "rayjob-3******",
    "__time__": 1777451603065
}

Common query statements

Log delivery only delivers logs to your topic. You need to configure indexes yourself to query. You can extract index fields from the raw data according to TLS prompts.
The common query statements are as follows:

  • Query by job ID
    • spark driver, for example: pod_name : "bq-328853200-driver"
    • spark executor, for example: pod_name: "bq-328853200-exec-7"
    • rayjob, for example: tag__ray_job_name : "rayjob-328853201"
Last updated: 2026.04.30 11:24:28