Configuring clustersMonitoring and alertsenables users to monitor the operational status and resource status of clusters in real time, promptly detect and handle anomalies, and ensure business stability. This article provides several types of basic Monitoring and alertsconfigurations and recommends applying these basic configurations.
tip
After disabling the alert template, all alert policies generated by the preset alert template will be deleted accordingly.
On the Alert center>Alert Template>Preset Alert Template page, in Resource Type , select Data Platform>E-MapReduce to filter the templates. Refer to Configure cloud server resource alertsconfiguration steps to enable the E-MapReduce preset alert template.
tip
Data Middle Platform>E-MapReduce also provides other preset alert templates, which can be enabled as needed.
Core service metrics directly affect business stability. It is recommended to configure alert policies for these metrics to promptly detect and address anomalies.
Log in to the EMR console, go to the EMR cluster details page>Alert center>Subscribe to event alerts, and navigate to the Cloud Monitoring alert policy page.
On the Cloud Monitoring alert policy page, click Create alert policy to enter the creation page.
Set Resource type to Data Middle Platform>E-MapReduce.
Configure alert Dimension, metric types Trigger condition, and other alert policy details. For detailed parameter descriptions, see Create Alert Policy .
The table below lists key service metrics and recommended alert policies. Please refer to these policies for configuration.
Alert dimension | Alert metric and Note | Metric level | Trigger condition |
|---|---|---|---|
HDFS | DFS space used by HDFS | Critical | Alert when usage exceeds 90% of total capacity tip Recommendation: Increase the JVM heap size of the NameNode and clean up directories containing a large number of small files |
HDFS storage utilization | Critical | Alert when above 80% | |
Number of lost HDFS blocks | Critical | Configure as needed. If lost blocks are not allowed, set alert for Missing block > 0. | |
NameNode heap space usage ratio | Critical | Alert when above 80% | |
DataNode old generation space utilization | Critical | Alert when above 95% tip Alert sending interval recommendation: The alert should persist for a certain period within a specified time frame. | |
Yarn | ResourceManager old generation space | Critical | Alert when above 95% |
NodeManager old generation space | Critical | Alert when above 95% | |
Hive | Number of open hiveserver2 connections | Critical | Set as needed |
HiveMetaStore heap memory utilization | Critical | Alert when above 95% | |
HiveServer2 heap memory utilization | Alert when above 95% | ||
Number of HiveServer2 query failures | Set as needed | ||
Hbase | HRegion full gc | P2 | Monitor and troubleshoot if it occurs frequently. |
HRegion read rate | P2 | Set as needed | |
HRegion write rate | P2 | Set as needed | |
Presto/Trino Coordinator | Current amount of HeapMemory used by JVM | P1 | Alert for exceeding 95% of heap memory |
Presto/Trino Worker | Current amount of HeapMemory used by JVM | P1 | Alert for exceeding 95% of heap memory |
ZooKeeper | Number of ZooKeeper znodes | P2 | Set as needed |
ZookeeperOverview | Total number of connections in ZooKeeper cluster | P2 | Set as needed |
Kafka(KafkaConsumerGroup) | Kafka Consumer Group Lag | P1 | Set as needed |