EMR Serverless queue supports mGPU, enabling resource sharing on a single GPU card, flexible allocation of compute cores and GPU memory resources, and strict isolation. This article introduces how to enable and use mGPU resources.

• mGPU can only be enabled when creating a queue. If enabled, the compute engine uses mGPU resources; if not enabled, the compute engine uses the entire GPU instance resources.
• A maximum of 16 mGPU Executors/Workers can be created on a single GPU card. Compute performance decreases as the number of mGPUs increases. It is recommended to plan the number of engine Executors/Workers according to business requirements.
• The minimum GPU memory setting for mGPU is 256 MiB, and the minimum core is 5.

• mGPU is VolcanoEnginea container-based shared GPU solution based on kernel virtualization isolation and an in-house scheduling framework. Through kernel virtualization technology, it enables resource sharing on a single GPU card, flexible allocation of compute cores and GPU memory resources, and strict isolation.
• After enabling mGPU, jobs in the queue can directly specify the use of part of the cores and GPU memory of a single GPU card. In small model scenarios, this can significantly reduce usage costs and improve resource utilization.
• On EMR Serverless GPU queue, both Ray and CustomJob engines support mGPU. You can use mGPU to share GPU resources among multiple jobs/Pods without modifying engine code.

Enable mGPU #

On the EMR Serverless Create queue interface, after selecting GPU Resource allocation, you can choose to enable mGPU. Once created queue, mGPU resources will be used.

Engine usage of mGPU resources #

Related parameters #

See the table below for mGPU-related parameters. For detailed GPU parameters, refer to EMR Serverless GPU parameter description.

CustomJob #

In the default compute group, you can additionally set the following parameters:

-- Set mgu-related parameters
emr.serverless.mgpu.core=30;
emr.serverless.mgpu.memory=1000;

-- Set historical GPU parameters. When the mgpu queue is enabled, full GPU cards are used by default.
-- Equivalent to emr.serverless.mgpu.core=100
serverless.custom.job.task.gpus=1; 

After setting, each CustomJob Task will mount a GPU card and use only 30% of its compute power and 1000 MiB of GPU memory.

RayJob #

In the default compute group, you can additionally set the following parameters:

-- mgpu-related parameters
serverless.ray.worker.mgpu.core=30;
serverless.ray.worker.mgpu.memory=4096;

-- Set historical GPU parameters. When the mgpu queue is enabled, full GPU cards are used by default.
-- Equivalent to serverless.ray.worker.mgpu.core;
serverless.ray.worker.gpu.amount=1;

After setting, each Ray Worker will mount a GPU card, but each Worker will only use 30% of the GPU card's compute power and 4096 MiB of GPU memory. You can also see that each Worker has one GPU card resource in the Ray dashboard:

RayCluster / RayServe #

In Ray Cluster/Ray Serve, you can set the required mGPU core and GPU memory for Worker/Head through the interface.

When the setting is greater than or equal to 100, it indicates the use of the full card's computing power. Only integer multiples of 100 can be set. For example, 200 means using the computing power of two full GPU cards.

In EMR Console > Queue Details > Queue Monitoring > GPU Resources , you can view the current queue's mGPU Core and Memory usage.

If the job fails to start after setting mGPU parameters, check whether the values of mGPU core and mGPU memory meet the requirements:

• When mGPU core is set to not exceed the computing power of a single GPU card: the range is 5–100.
• When mGPU core is set to greater than or equal to 100, only integer multiples of 100 can be set.
• When mGPU core is set to greater than or equal to 100, mGPU memory cannot be set.