Memcg asynchronous memory reclaim is an enhanced feature provided by the veLinux kernel on top of the standard Memory Cgroup. This topic describes the working principle, configuration, and query methods of this feature, helping you reduce the impact of memory reclaim on request latency in latency-sensitive workloads through background asynchronous reclaim.

Background

Memory Cgroup (Memcg) is the memory subsystem of Linux kernel control groups (cgroups). It isolates the memory behaviors of a group of processes from the rest of the system, and tracks and limits their memory usage. Common use cases include isolating memory-intensive applications, setting memory quotas for containers or VMs, and preventing a single workload from exhausting the memory of an entire node. Learn more

Direct memory reclaim: When Memcg usage reaches its limit, the kernel reclaims memory synchronously on the memory allocation path of the application thread. The thread is blocked during reclaim. Under high memory pressure, this manifests as tail latency jitter, increased memory allocation latency, and occasional service stalls.

Asynchronous memory reclaim: Two memory watermarks, low and high, are configured for the cgroup. When available memory drops to the low watermark, Memcg wakes up a background kworker thread to reclaim memory asynchronously until available memory recovers to the high watermark. The reclaim process does not occupy the memory allocation path of application threads, so threads continue running without being blocked. Memory is freed before usage reaches the limit, reducing the likelihood of direct memory reclaim being triggered (though it cannot be guaranteed to be completely avoided) and minimizing request latency jitter.

Applicable scenario

Memcg asynchronous memory reclaim is suitable when a workload uses Memcg for memory isolation and exhibits any of the following symptoms under high memory pressure:

Noticeable jitter in request tail latency (P99, P999).

Increased latency in memory allocation (page fault, malloc).

Occasional service stalls despite available CPU and I/O headroom on the node.

This feature is especially suitable for workloads that are latency-sensitive, experience significant memory allocation bursts, or require memory isolation between containers or process groups, such as online services, cache services, and real-time data processing.

Usage notes

Item	Description
Kernel version	veLinux kernel versions that support this feature: veLinux: 5.4.250-1-velinux1 and later 5.15.152.ve.7u1 and later veLinux CentOS Compatible: 5.4.250-1.ve1 and later 5.15.152-7.1.ve2 and later tip If the memory.watermark_scale_factor or memory.watermark interface, or the corresponding sysctl parameter does not exist in your instance operating system, the current kernel version does not support or has not enabled this feature.
Cgroup version	cgroup v1 and cgroup v2 are supported. tip If your operating system uses cgroup v2, we recommend referring to the Linux community solution first and using the memory.reclaim interface to flexibly control reclaim timing and the amount of memory reclaimed. This helps maximize memory utilization and minimize the impact of memory reclaim.
Risk warning	The asynchronous reclaim feature is triggered automatically when memory watermarks reach the configured thresholds. It does not evaluate the appropriateness of the current reclaim timing or the amount of memory reclaimed, and may still have some impact on workload latency.

Procedure

Configuration recommendations

We recommend that you first use the cgroup interface to enable asynchronous memory reclaim for the target workload on a canary basis. After verifying the gains and CPU overhead, use sysctl (configuration file) to persist the default values for subsequently created cgroups. Configure via kernel boot parameters only when the workload requires default values to be applied automatically at boot time.

Configuration item	Recommendation
Watermark ratio	Start with a small value on a canary basis, for example, 100. If direct memory reclaim remains frequent, gradually increase the ratio. If CPU overhead increases or reclaim becomes too frequent, decrease the ratio.
Enablement scope	We recommend enabling the feature individually for latency-sensitive Memcgs where memory reclaim jitter has already been observed (via the cgroup interface). Enabling it globally via sysctl or kernel boot parameters is not recommended.
Reclaim concurrency	Keep the default value of 1. If you need to increase it, verify the CPU overhead and workload latency in a test environment first.

Configuration methods

Configure via the cgroup interface

Configure global sysctl parameters

Configure kernel boot parameters

This method applies only to a single existing Memcg. Changes take effect immediately after configuration and are lost when the system restarts or the cgroup is recreated. It is suitable for enabling the feature on a canary basis for individual latency-sensitive workloads, and then expanding the scope after verifying tail latency gains and CPU overhead.

Execute the following command to query the watermark ratio and actual watermark of a single existing Memcg.

Cgroup v1
Cgroup v2

tip
Replace <memcg_path> with the actual path of the target Memcg.

cat /sys/fs/cgroup/memory/<memcg_path>/memory.watermark_scale_factor
cat /sys/fs/cgroup/memory/<memcg_path>/memory.watermark

Interface description:

Interface	Attribute	Default value	Description
memory.watermark_scale_factor	Read/write	0	Configure the watermark ratio for the current Memcg. Valid values range from 0 to 1000. A value of 0 disables asynchronous reclaim for the cgroup. A value of 1000 sets the low watermark to 10% of memory.limit_in_bytes, and the high watermark to twice the low watermark.
memory.watermark	Read-only	0	View the calculated low and high watermark values for the current cgroup to verify that the configuration has taken effect.

Execute the following command to configure the watermark ratio for the specified Memcg.

Cgroup v1
Cgroup v2

tip
Replace <memcg_path> with the actual path of the target Memcg, and replace 100 with the desired value.

echo 100 > /sys/fs/cgroup/memory/<memcg_path>/memory.watermark_scale_factor

This method sets the global default values for subsequently created Memcgs and does not affect existing cgroups. It is suitable for applying uniform default values to new workloads after canary validation has passed.

Execute the following commands to query the global default watermark ratio and the maximum concurrency of memcg kworkers for subsequently created Memcgs.

warning

The sysctl query result represents only the global default values and does not indicate that asynchronous reclaim has been enabled for existing Memcgs. To verify the actual configuration of an existing cgroup, use the cgroup interface to query.

cat /proc/sys/vm/memcg_watermark_scale_factor
cat /proc/sys/vm/memcg_mem_reclaim_wq_max_active

Select a configuration method as needed to set the global default values for subsequently created Memcgs.

Configuration by using the proc interface

Configure by using the sysctl file

Execute the following commands on the target Linux instance to configure the memcg_watermark_scale_factor and memcg_mem_reclaim_wq_max_active parameter values via the proc interface.

tip

This method allows configuration changes to be made and applied while the system is running. However, the configuration will reset after an operating system restart and must be reconfigured.

The following commands set the default watermark ratio for subsequently created Memcgs to 100 and set the maximum concurrency of memcg kworkers per NUMA node to 1. Replace the values as needed.

Interface description:

Interface	Default value	Description
/proc/sys/vm/memcg_watermark_scale_factor	0	Configure the default watermark ratio for subsequently created Memcgs. Modifying this parameter does not affect existing Memcgs.
/proc/sys/vm/memcg_mem_reclaim_wq_max_active	1	Configure the maximum number of concurrent memcg kworker threads. The default value of 1 means that at most one memcg kworker runs concurrently per NUMA node.

This method sets the global default watermark ratio during system boot, applying it to subsequently created Memcgs. It takes effect after the instance is restarted and persists across restarts. It is suitable for scenarios where default configuration should be applied automatically at system boot.

warning

Modifying the GRUB configuration file is a high-risk operation. Incorrect configuration may prevent the system from starting normally. Back up the original configuration file before proceeding, and verify the changes in a test environment before applying them to a production environment.

Execute the following command to back up the GRUB configuration file.

sudo cp /etc/default/grub /etc/default/grub.bak

Execute the following command to check whether the asynchronous memory reclaim parameters already exist in the GRUB configuration file.

grep 'memcg_watermark_scale_factor' /etc/default/grub

Parameter descriptions:

Parameter	Description
memcg_watermark_scale_factor=	Set the global Memcg watermark ratio at system boot. This is equivalent to calling the sysctl interface /proc/sys/vm/memcg_watermark_scale_factor, and applies to subsequently created Memcgs.

If the output is empty, execute the following command to append the memcg_watermark_scale_factor parameter.

tip

Replace 100 with the target watermark ratio value.

sudo sed -i '/^GRUB_CMDLINE_LINUX=/s/\"$/ memcg_watermark_scale_factor=100\"/' /etc/default/grub

If the output is not empty, execute the following command to update the memcg_watermark_scale_factor parameter value.

tip

Replace 100 with the target watermark ratio value.

sudo sed -i 's/memcg_watermark_scale_factor=[^ "]*/memcg_watermark_scale_factor=100/' /etc/default/grub

Execute the following command to persist the parameter configuration to the GRUB configuration file.

veLinux
veLinux CentOS Compatible
Legacy boot mode

grub-mkconfig -o /boot/grub2/grub.cfg

UEFI boot mode

grub-mkconfig -o /boot/efi/EFI/debian/grub.cfg
Legacy boot mode

grub2-mkconfig -o /boot/grub2/grub.cfg

UEFI boot mode

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

Elastic Compute Service