E-MapReduce
E-MapReduce
- Documentation
E-MapReduceGuide to operating componentsYARNYARN enhancementsYARN resource pool management
Copy page
Download PDFCopy pageDownload PDFEMR enables you to activate Node Label for YARN clusters, associate node resources and queues, divide them into resource pools, and visually manage resource pools.
tip
- This feature is an enhanced capability of the YARN component in EMR. All configurations can be implemented by manually modifying the YARN configuration file.
- After creating a resource pool, only the resource pool-associated queues can use the computing resources of the resource pool nodes; other queues cannot use them.
- Only supports creating a single hierarchy with YARN label, for example:
pool1. - Only supports the YARN
Capacity Scheduler. - Node resources that are not bound belong to the default resource pool. Configure the association between the
defaultresource pool and queues in Service List> YARN >Queue Management. For details, see YARN queue management.
- Enable
NodeLabels.
Method 1 | On the Service list > YARN >Service parameters page, search for |
|---|---|
Method 2 | On the **Service list > YARN ** page, click Service Operations> EnableNodeLabels, in the upper right corner. |
- Restart the service to apply the configuration.
After restarting the service, click Service list > YARN > Resource Pool to enter the resource pool creation page.
Parameters | Note |
|---|---|
Resource pool label | The unique identifier of the resource pool; the resource pool label is equivalent to the Node-Label name. Can be associated with node groups and queues. |
Associated queues | Select the YARN queues that need to use the resources of this resource pool. After selecting, you can configure the resource percentage and maximum resource percentage for this queue in the resource pool. Supports multiple selection and allows independent configuration for sub-queues. |
Resource percentage | The minimum guaranteed resource percentage for this queue on nodes in this resource pool. warning The sum of the resource percentages at each level must be 100%. |
Maximum resource percentage | The upper limit of the resource percentage that this queue can use on nodes in this resource pool. warning The maximum resource percentage must be greater than the resource percentage. |
Node group | Select nodes to add to this resource pool. The selected nodes will be labeled with the resource pool tag, and their resources will be exclusively allocated to the associated queues according to the configuration above. |
tip
- For example, in a cluster with 10
Corenodes, each with 32 cores, all 10 nodes are associated with resource poolpool1.pool1is associated with queuestest1andtest2. If the resource percentage fortest1is set to 25% and fortest2is set to 75%, and the maximum resource percentage for both is 100%, then when there are tasks competing in queuestest1andtest2,test1can use 80 cores andtest2can use 240 cores. When only a single queue submits tasks, the maximum available resources are 320 cores. - In rare cases, the same queue may have its resource percentage set separately in multiple resource pools. In this case, the total guaranteed resources for the queue are the sum of the allocations from each resource pool.
Custom resource pools
Associated resource pools will display resource summary, instance details, associated queues, and other information in the resource pool list.
- Supports editing resource pools to modify queue parameters or instance associations.
- Supports deleting resource pools; the resource pool and its corresponding relationships will be deleted.
Default resource pool
After enabling resource pool management, resources not assigned to a resource pool belong to the default resource pool. Configure the association between the default resource pool and queues in Service List> YARN >Queue Management. For details, see YARN queue management.
