Docs

Elastic Compute Service

Linux operating system issue

Solutions for improper configuration of I/O timeout arguments for Linux instance NVMe disks

If the ECS instance of the Linux system uses the NVMe system disk and experiences unexpected slow I/O reads and writes due to improper configuration of I/O timeout arguments (such as the timeout period for I/O read and write operations on NVMe disks being set too small), leading to system and application exceptions or business interruptions, you can solve the issue by adjusting the io_timeout argument of the NVMe driver.

tip

Slow I/O read and write: Refers to the execution speed of input/output operations during disk read/write processes being lower than expected or required time.

Causes

The io_timeout argument in the NVMe driver is used to control the maximum tolerable I/O timeout period. If the latency of the I/O read/write operation is too high and exceeds the configured value of this argument, the NVMe driver will return an I/O failure. This will cause the file system on the NVMe disk to switch from the previously mounted readable and writable status to a read-only status, making subsequent write operations all fail, and thereby resulting in system and application exceptions or business interruptions.

tip

In most Linux platform versions, the argument is set to a default configuration of 30 seconds.

Procedure

Configure the io_timeout argument (temporary)

Configure the io_timeout argument (permanent)

tip

The temporary configuration of the io_timeout argument will expire after restarting the instance.

Execute the following command to check the path of the kernel module where the io_timeout argument is located.

cat /sys/module/nvme_core/parameters/io_timeout

tip

Due to differences in kernel version, the kernel module for the NVMe driver may also vary. It generally includes nvme.ko or nvme_core.ko, so the complete name of the timeout argument is nvme.io_timeout or nvme_core.io_timeout.

Execute the following command to write the desired timeout period into the kernel module. Once configured, it will take effect immediately.

sudo sh -c 'echo {Value} > /sys/module/nvme_core/parameters/io_timeout'

sudo sh -c 'echo {Value} > /sys/module/nvme/parameters/io_timeout'

tip

Please replace the Value in the command with the actual timeout period to be adjusted.

Last updated: 2026.06.18 17:49:52