This article introduces the background pain points, core value, and usage guide of the EMR Agent queue diagnostics capability, helping operations teams quickly assess the availability, resource pressure, job status, and recent change impact of EMR Serverless queues.

Current situation and pain points #

In daily operations of EMR Serverless, queues are the core objects for job submission, resource quotas, and compute group scheduling. Queue issues typically directly affect job startup, operational efficiency, and business stability, but manual troubleshooting presents the following pain points:

1. Fragmented resource perspectives and high troubleshooting cost

Queue issues often involve queue status, compute group status, CPU/memory/GPU quotas, job distribution, shuffle pressure, and recent configuration changes simultaneously. Operations personnel need to repeatedly switch between the console, monitoring, job lists, and operation records to piece together the complete context.

2. Complex semantics of Serverless resources

The meanings of fields such as public queues, exclusive queues, fixed resources, elastic resources, compute group MinCU/MaxCU, AssignedCU, GPU/MGPU, and others differ. Relying on a single configuration item or a single monitoring curve can easily lead to misjudging normal resource states as abnormal.

3. Job symptoms and queue root causes do not always correspond one-to-one

Job failures, slow submission, slow execution, and queue waiting may result from insufficient queue resources, unavailable target compute groups, unreasonable job parameters, SQL/code issues, or recent change failures. Manual troubleshooting requires combining multiple types of evidence for judgment, with a high experience threshold.

4. Difficulty in standardizing reporting criteria

Different personnel may describe the risks of the same type of queue with varying granularity, conclusion priorities, and handling suggestions, causing troubleshooting closure to depend on individual experience and making it difficult to establish a unified operations process.

Introduction to queue diagnostics #

EMR Agent queue diagnostics is designed for EMR Serverless queue scenarios. After the user provides the queue ID, the Agent automatically collects basic queue information, compute group configuration, resource monitoring, recent jobs, and evidence of abnormal operations, and generates a structured diagnostic report.
The queue diagnostics primarily address the following questions:

• Is the queue currently available, and are there any risks such as exceptions, overdue payments, expiration, release, or ongoing changes?
• Are the compute groups in an operable state, and could capacity configuration affect job submission or execution?
• Are resources such as CPU, memory, GPU, and MGPU approaching their limits, and is there sustained high or peak pressure?
• Have recent jobs exhibited obvious failures, queuing, prolonged execution, or concentration of resource consumers?
• Have recent queue or compute group operations failed, are they being processed, or are there changes related to current symptoms?
• What is the most appropriate action to take now, and what are the directions for subsequent optimization?

Use cases #

Queue diagnostics are suitable for the following use cases:

1. Slow job submission or prolonged queuing

For example, jobs are stuck in submission or resource initialization, or the number of running jobs is abnormal, leading to the suspicion of insufficient queue resources or unavailable target compute groups.

2. Queue resource pressure investigations

For example, sustained high allocation or utilization rates of CPU, memory, GPU, or MGPU, requiring assessment of whether to scale out, adjust quotas, or optimize job resource parameters.

3. Recent job failure concentration analysis

For example, multiple job failures occur within a short period in the same queue, requiring differentiation between queue-side resource or scheduling issues and job-specific SQL, code, or parameter issues.

4. Compute group configuration and status checks

For example, a Spark, Presto, Ray, or general compute group is unavailable, has insufficient configuration, or has an unreasonable waiting strategy, affecting jobs of the corresponding type.

5. Queue health checks after changes

For example, after scaling, creating or deleting compute groups, suspending, or releasing operations, it is necessary to confirm whether the changes affect queue availability and job execution.

6. Routine inspections and capacity assessments

For example, to quickly understand the resource usage, main resource consumers, and potential optimization points of a queue over a period of time.

Prerequisites #

Before using queue diagnostics, confirm the following:

• The EMR Agent service has been enabled. For reference, see the EMR Agent Activation Guide.
• The current account has permission to view the target EMR Serverless queue, jobs, monitoring, and operation records.

Diagnostic entry points #

EMR Agent queue diagnostics provides three common entry points.

Entry point 1: Initiate from the queue resource list

Go to the Serverless → Queue Resources page, and click AI Diagnostics in the row of the target queue. This entry point will directly redirect to EMR Agent and automatically carry over the current queue context. It is suitable for quickly initiating diagnostics from the queue inspection, resource list, or routine operations pages.

Entry point 2: Initiate using natural language in EMR Agent Assistant

After entering the EMR Agent Assistant, you can directly describe your diagnostic request in the input box and provide the target queue ID.

Common question formats:

帮我看一下 1547635987612434432 这个 Serverless 队列最近是否有资源瓶颈
这个队列最近作业提交很慢，帮我判断是不是资源不足：队列ID 1547635987612434432
帮我诊断一下队列 1547635987612434432

Entry point 3: Initiate via the resource diagnostics form

After entering the EMR Agent Assistant, click Resource Diagnostics on the homepage, and select Serverless Queue Diagnostics from the dropdown menu. The system will display the queue diagnostics form. Fill in the queue information as required and submit the form.

Diagnostic process #

Step 1: Select a diagnostic entry point
You can initiate diagnostics from the AI Diagnostics button in the queue resource list, or enter EMR Agent Assistant to initiate via natural language or the resource diagnostics form.
Step 2: Wait for the Agent to automatically collect and analyze data
The Agent will automatically retrieve basic queue information, compute group information, monitoring summaries, recent jobs, and evidence of abnormal operations, then generate a queue diagnostics report. There is no need to manually switch between multiple systems to collect context during the diagnostic process.
Step 3: View the diagnostic results
After diagnostics are complete, you can view the report summary in the current session or go to the report details to see the full diagnostic content. The report is organized into modules such as anomaly insights, resource analysis, and health assessment, making it easy to quickly locate issues and take action.

Report interpretation #

The queue diagnostics report typically includes the following modules:

When reading the report, it is recommended to focus on the following first:

• Is there a clear exception conclusion: If "Exception insight" is empty or only indicates that no clear exception was found, it means that the current evidence does not show a queue-level failure.
• Resource bottleneck type: Distinguish between CPU, memory, GPU, MGPU, Shuffle, or compute group quota issues, and avoid focusing only on a single metric.
• Impact scope: Confirm whether the impact is on the entire queue, a specific compute group, a specific engine type, or a small number of jobs.
• Recommendation trigger conditions: Recommendations such as scaling out, adjusting MinCU/MaxCU, modifying job parameters, or retrying failed jobs usually need to be executed in conjunction with the trigger conditions described in the report.